reef-uni builder tests - No enough DUTs - required 4; found 2 |
|||||||
Issue descriptionFor pyro, snappy, and sand, we are failing in the autotest scheduler when provisioning boards. It looks like we only have 6 boards (whereas there are 8 for reef). The normal builder is provisioning the first 4 boards and then the uni-builder can only provision 2 boards. For now, I'm going to change the requirement to 3, so we can run in parallel, but it seems like we should reprovision the lab if possible to help with the transition to uni-builds. Or we'd need to look at something more drastic... like changing the uni builders to run at a different time.
,
Jul 27 2017
Which builder is it you're going after? Can you point to the chromite code? My first thought would be to see if we can set aside a set of DUTs specifically for uni-build testing, so that for now we don't mix-n-match with regular BVT testing.
,
Jul 27 2017
Proposed change is here: https://chromium-review.googlesource.com/c/589812/ This will affect release builders.
,
Jul 27 2017
> This will affect release builders. I'm not convinced that this is the right thing to do for all release builders. I'm not even convinced that it's the right thing to do for _any_ builder (but see bug 727834 for a case where I'm likely to change my mind). Again, for now, I think we should be segregating uni-build DUTs from other DUTs.
,
Jul 27 2017
I don't know how hw is provisioned in the lab or what is available in the first place. If we make it a separate pool, we'll still need to do some work in buildbot to set this pool accordingly.
,
Jul 27 2017
It looks like the existing 'reef-uni' builder is scheduling tests against the snappy BVT pool. The BVT pools are sized based on the number of release builders they're expected to serve. By default that's 6. When there's increased load (e.g. from a PFQ builder), we increase the size to improve capacity. The snappy BVT pool is currently not sized to support extra builders. That means the reef-uni builder is able to disrupt the snappy Canary and Beta builds. Before we start scheduling reef-uni builds against snappy (or any other board+pool), we have to answer a few basic questions like "do we have capacity?" In this case, we also need to answer the question "is this stable enough to be sharing with release builders?" We need to disable HWTest stages on reef-uni until those questions can get sorted.
,
Jul 27 2017
,
Jul 27 2017
we're working on a feature to be able to disable model specific testing ... it should be done in the next couple of days. we can be more aggressive about it if necessary, but I'd rather let my other change land, since this helps continue to trace/debug issues with unibuild
,
Jul 27 2017
We need to make sure that the reef-uni builder isn't interfering
with any of the existing release builders. That needs to take
priority.
For now, I think we can solve this with two changes:
1) Change the reef-uni release builder config to use a new
pool for all testing, e.g. "bvt-uni".
2) The primary deputy should allocate a modest number of
devices (no more than 6) to the pool for the selected
board.
,
Jul 27 2017
sounds good ... I'll submit a change to do this
,
Jul 27 2017
https://chromium-review.googlesource.com/c/590688 to switch to bvt-uni pool.
,
Jul 27 2017
+ primary deputy phobbs@ Can we please make the following pool allocations to this new pool: bvt-uni reef - 6 machines pyro - 6 machines snappy - 6 machines sand - 6 machines coral (if you have it) - 6 machines
,
Jul 31 2017
Assigning to current deputy.
,
Jul 31 2017
Richard, do we have so many available DUTs to be allocated to this new pool? Will it affect the CQ performance after the allocation?
,
Aug 1 2017
I met with shapiroc@ earlier today to go over this request,
and other matters. We've filled in allocations for the
boards with sufficient supply:
$ pool_boards bvt-uni
6 reef
6 sand
6 snappy
We're still waiting for a CL from shapiroc@ before we can declare
victory.
,
Aug 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5 commit a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5 Author: C Shapiro <shapiroc@chromium.org> Date: Tue Aug 01 21:39:52 2017 Changing unified builds to separate bvt-uni pool In many cases, the lab is currently only provisioned with 6 DUTs (for bvt), which works if only one builder is ever performing hardware tests. However, with the addition of the unified builder, we're now competing for resources. This switches unified builds to a separate pool, which means we'll need to allocate hardware in the lab accordingly for each bringup. New pool name is bvt-uni BUG= chromium:749731 TEST=none available Change-Id: I5a38bf0111553a5d918801d21022aaef036efc92 Reviewed-on: https://chromium-review.googlesource.com/590688 Commit-Ready: C Shapiro <shapiroc@google.com> Tested-by: C Shapiro <shapiroc@google.com> Reviewed-by: C Shapiro <shapiroc@google.com> [modify] https://crrev.com/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5/cbuildbot/config_dump.json [modify] https://crrev.com/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5/lib/constants.py [modify] https://crrev.com/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5/cbuildbot/chromeos_config.py
,
Aug 7 2017
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by shapiroc@chromium.org
, Jul 27 2017