And I should say, no individual test fails, this looks like a timeout of sorts.

Buildbot link: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/5105

First guess is that the suites pool (where tryjob hwtests run) is fully utilized.

Mostly borne out by https://viceroy.corp.google.com/chromeos/dut_utilization?board=auron_yuna&pool=managed%3A.*&status=Running&topstreams=5&duration=1d&mdb_role=chrome-infra&refresh=-1 at least at the moment.

(I don't know why we have DUTs stuck in "Pending" state)


Also +dgarrett who has been trying to run hwtests from tryjobs, and may be able to comment on whether they have been succeeding lately. There might be a bug in provision suite (+ayatane) affecting tryjob hwtest.

Ah thanks, I hadn't thought that the timeout could be because of resource contention. At least it means that my CL is not killing the devices =D.

I'll disable provision suite for the suites pool, which is probably exacerbating the issue if not the primary cause.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4f09a6d3bfef6281a16aec1da69186f6803da677

commit 4f09a6d3bfef6281a16aec1da69186f6803da677
Author: Allen Li <ayatane@chromium.org>
Date: Thu Jan 25 00:29:24 2018

[autotest] Disable provision suite on pool:suites

Provision suite fundamentally relies on a dedicated pool to work
correctly (e.g., cq pool).  So disable on pool:suites.

Ideally this business logic should live in the suite
scheduler/buildbot config, but its much easier to do here, and serves
as a useful failsafe anyway.

BUG= chromium:804614 
TEST=None

Change-Id: I0e03e9efb140324122d45d7cf83c7f644f5743fe
Reviewed-on: https://chromium-review.googlesource.com/882189
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>

[modify] https://crrev.com/4f09a6d3bfef6281a16aec1da69186f6803da677/server/cros/dynamic_suite/suite.py

The pending DUTs look bad.  Pending is used for tests that have sync count > 1.  DUTs go into Pending as they become ready to run the test, then when enough DUTs are Pending they all go into Starting/Running to run the test together.

I don't know why half the DUTs are stuck in Pending (from what it looks like).