My guess is they started timing out around https://chromium.googlesource.com/chromium/src/+/84d4dfb32ea85d528373295087f10d0766d556b6 . The 10 or so builds before that all passed successfully. The next build, https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Tests%20%281%29/builds/46249 , failed.

Suspiciously close to my change in r467230. There's nothing there that should affect test behavior beyond possible timing issues AFAICT. I'll start looking at this.

Looking at the failure, all tests are timing out. Not a single pass.

I'd seen this type of issue when chrome is unable to connect to the UI service.

Noticeable because ScreenMus waits on a connection here: https://cs.chromium.org/chromium/src/ui/views/mus/screen_mus.cc?rcl=03d4670df994c2b4cdc96c3f2ee8903f2d3c8ca2&l=62

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2f53aa1c96d14295e7739cae01db7d780f3dcb44

commit 2f53aa1c96d14295e7739cae01db7d780f3dcb44
Author: sky <sky@chromium.org>
Date: Mon May 01 16:28:52 2017

Change timeout of mash/mus_browser_Tests to 15 minutes

They should not come close to one hour.

BUG= 717028 
TEST=test only change
R=jonross@chromium.org
TBR=jonross@chromium.org

Review-Url: https://codereview.chromium.org/2855623002
Cr-Commit-Position: refs/heads/master@{#468324}

[modify] https://crrev.com/2f53aa1c96d14295e7739cae01db7d780f3dcb44/testing/buildbot/chromium.chromiumos.json

Finally figured this out. Simple race condition caused by mash_session starting up slightly faster now. Fix incoming.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/faf6b1e5ba15004aec259c817b75509df9639c88

commit faf6b1e5ba15004aec259c817b75509df9639c88
Author: Ken Rockot <rockot@chromium.org>
Date: Wed May 03 06:56:42 2017

Fix mash_browser_tests race

mash_session startup has been racing with the manual
content_packaged_services instance registration done by the browser
test framework. This can and almost always does cause the service
manager to launch a new browser process instance which in turn prevents
the test browser process from getting hooked up as a service instance.

This change ensures that a connection to mash_session is only initiated
after the test browser process is successfully launched and registered,
thus avoiding the race.

BUG= 717028 

Change-Id: I8240f2dab0aa9248b5065e597ae9cf8f2b140235
Reviewed-on: https://chromium-review.googlesource.com/493800
Commit-Queue: Ken Rockot <rockot@chromium.org>
Reviewed-by: Scott Violet <sky@chromium.org>
Cr-Commit-Position: refs/heads/master@{#468911}
[modify] https://crrev.com/faf6b1e5ba15004aec259c817b75509df9639c88/chrome/test/base/mash_browser_tests_main.cc
[modify] https://crrev.com/faf6b1e5ba15004aec259c817b75509df9639c88/chrome/test/base/mojo_test_connector.cc
[modify] https://crrev.com/faf6b1e5ba15004aec259c817b75509df9639c88/chrome/test/base/mojo_test_connector.h

The ongoing mus_browser_tests failures appear to be unrelated, but mash_browser_tests is now consistently green again.