:( 1 more failure for the same CL:

https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/wificell_pre_cq/87

I don't think I see any actual failures (e.g., provisioning). It's just that the build takes way too long:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8955305215816276496

The mistakes about PreCQ failures causing excessive/premature re-rejections might be bug 808683. So perhaps we should focus on only the (legitimate) timeouts here, and not the poor notification logic. It seems like we'll either need to parallelize the builds, increase the timeouts, or reduce the number of configurations we do at once.

(Also bug 810628, but that as considered a duplicate.)

 Issue 601813  has been merged into this issue.

My latest timeout:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8946771502912122064

It also had a failure because we had a missing (locked for repair) veyron_speedy device, and so we couldn't allocate any testing for it.

I think I can confidently declare that wificell-pre-cq is broken in its current state. Among the things that would need to change:
(a) improve device allocation behaviors -- we are simultaneously testing too little and too much; if we can't allocate many devices for this, it's not good to require 1 of each -- we should probably be flexible; but on the other hand, we don't really have a good recent selection of boards in this pool either
(b) make a sane timeout -- if we're running X builds in sequence, we need more than 4 hours
(c) probably more?

Agree that this is not usable at the moment. In terms of expanding the pre-cq pool with more recent boards, there is an existing open bug https://b.corp.google.com/issues/73493055


As for the number of devices / boards of each kind in the pre-cq pool, I am not sure what a good number is as this pretty much depends on how often its used and how many folks try to use it at the same time. The existing setup is suppose to have 2 of the following; winky, speedy, jerry, snow, elm and 1 of the following: lulu, cyan, samus.


Regarding the 4 hour timeout, I am not sure where that comes from (akeshet@ / nxia@ ?), I see 60min max runtime in the suite control file https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/test_suites/control.wificell-pre-cq?l=32

> As for the number of devices / boards of each kind in the pre-cq pool, I am not sure what a good number

I'm not directly asking for more types of boards right now; and not necessarily even allocating more boards. I just want the existing setup to be reliable. For the few small sanity tests we run here, it feels like we don't often care about running on *all* board types all the time. If the alternatives are
(a) fail because a DUT or two were down/locked or
(b) run on an incomplete set of boards
I'd always choose the latter for this suite.

BTW, allocating 1 DUT for a board type that is required, means that this suite will often fail. As soon as a lab tech needs to repair something, swap APs, etc., we're dead. So we definitely either need more than 1, or else modify the suite to do something more flexible.

Related question: is it possible to allocate DUTs to more than one pool? e.g., we probably don't fully utilize both pool:wificell and pool:wificell-pre-cq.

> The existing setup is suppose to have 2 of the following; winky, speedy, jerry, snow, elm and 1 of the following: lulu, cyan, samus.

$ atest host list -b pool:wificell-pre-cq | grep '^chromeos' | wc -l
9

That doesn't add up ;)

[I think we've implicitly stopped using this pre-cq, as it's in bad shape.]