In the case of terra-release, this looks like contention over DUTs, possibly between both suites from that builder. Both suites were created at ~6:20. bvt-arc ran its first test at 6:45, but bvt-inline did not run its first test until 8:00 (on the same devices). This is most clearly visible at the suite details page for bvt-inline https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/suiteDetails?suiteId=216109616 in the DUT timeline by comparing it to the test timeline.


The nautilus case looks unrelated, and looks like it was due to a single device hanging in provision

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/suiteDetails?suiteId=216106669

Additional theory on terra: that board runs on shard cros-full-0006 which is ticking really slowly (<1 tick / min). Investigating that as a possible cause at  Issue 862685

More example of recent failures:

From daisy-release:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941335069720270672
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941395393730622816
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941425838850157024
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941546963549535856
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941665577766221792


Here are other examples from various builders:
veyron_mickey-release: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941334976006637728
eve-release: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941335061735746480
kip-release: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941335036853874528
monroe-release: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941335022502945600

Would this issue be the cause of bvt-arc failures on R68-10718.50.0 beta?  Seeing a lot of failures for cheets_PlayStoreTest.

https://stainless.corp.google.com/search?view=list&first_date=2018-07-10&last_date=2018-07-11&owner=chromeos-test&waterfall=chromeos%28_release%7C%29&suite=bvt-arc&test=cheets_PlayStoreTest&build=R68-10718.50.0&status=FAIL&status=ERROR&status=ABORT&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

This bug is going in a lot of directions at once. Let's keep it on a single topic.

So far addressed on this bug: are failures on terra, daisy, and veyron_jerry -> Issue 862685

Nautilus -> Provision failure or flake.

I believe all other issues are unrelated unless there is evidence otherwise, so let's pursue them separately.

We just got another batch of SuiteTimedOut failures: 

nyan_kitty-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941215432188509408
peach_pit_paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941215429166806224
peppy-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941215428750727440
winky-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941215416122367808
link-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941215436996772000

Bumping priority up

We just got another round of SuiteTimedOut failures. Bumping priority, and assigning to on-call.

nyan_big-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941125367367690832
nyan_kitty-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941125366640528512
peach_pit-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941125363897998000
wolf-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8941125349278427936

I broke off failures from comment 4, 7, and 9 into  crbug.com/863401  since they were all SuiteTimedOut failures. We can discuss there.

These all seem to be in the testing steps, assigning to infra deputy.

Copying current sheriffs

There are a range of different issues related to this bug. This bug itself is about too many different things to be actionable. Closing. 

See Issue 862685 Issue 863504 Issue 852083