New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 749731 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

reef-uni builder tests - No enough DUTs - required 4; found 2

Project Member Reported by shapiroc@chromium.org, Jul 27 2017

Issue description

For pyro, snappy, and sand, we are failing in the autotest scheduler when provisioning boards.

It looks like we only have 6 boards (whereas there are 8 for reef).

The normal builder is provisioning the first 4 boards and then the uni-builder can only provision 2 boards.

For now, I'm going to change the requirement to 3, so we can run in parallel, but it seems like we should reprovision the lab if possible to help with the transition to uni-builds.

Or we'd need to look at something more drastic... like changing the uni builders to run at a different time.
 
Status: Started (was: Untriaged)
Components: Infra>Client>ChromeOS
Which builder is it you're going after?  Can you point
to the chromite code?

My first thought would be to see if we can set aside a
set of DUTs specifically for uni-build testing, so that
for now we don't mix-n-match with regular BVT testing.

Proposed change is here:
https://chromium-review.googlesource.com/c/589812/

This will affect release builders.
> This will affect release builders.

I'm not convinced that this is the right thing to do for all
release builders.  I'm not even convinced that it's the right
thing to do for _any_ builder (but see bug 727834 for a case
where I'm likely to change my mind).

Again, for now, I think we should be segregating uni-build DUTs
from other DUTs.

I don't know how hw is provisioned in the lab or what is available in the first place.

If we make it a separate pool, we'll still need to do some work in buildbot to set this pool accordingly.
It looks like the existing 'reef-uni' builder is scheduling
tests against the snappy BVT pool.  The BVT pools are sized
based on the number of release builders they're expected to
serve.  By default that's 6.  When there's increased load
(e.g. from a PFQ builder), we increase the size to improve
capacity.  The snappy BVT pool is currently not sized to
support extra builders.  That means the reef-uni builder is
able to disrupt the snappy Canary and Beta builds.

Before we start scheduling reef-uni builds against snappy
(or any other board+pool), we have to answer a few basic
questions like "do we have capacity?"  In this case, we also
need to answer the question "is this stable enough to be sharing
with release builders?"

We need to disable HWTest stages on reef-uni until those questions
can get sorted.

Cc: akes...@chromium.org
we're working on a feature to be able to disable model specific testing ... it should be done in the next couple of days.

we can be more aggressive about it if necessary, but I'd rather let my other change land, since this helps continue to trace/debug issues with unibuild
We need to make sure that the reef-uni builder isn't interfering
with any of the existing release builders.  That needs to take
priority.

For now, I think we can solve this with two changes:
 1) Change the reef-uni release builder config to use a new
    pool for all testing, e.g. "bvt-uni".
 2) The primary deputy should allocate a modest number of
    devices (no more than 6) to the pool for the selected
    board.

sounds good ... I'll submit a change to do this
https://chromium-review.googlesource.com/c/590688 to switch to bvt-uni pool.
Owner: pho...@chromium.org
+ primary deputy phobbs@

Can we please make the following pool allocations to this new pool:
bvt-uni

reef - 6 machines
pyro - 6 machines
snappy - 6 machines
sand - 6 machines
coral (if you have it) - 6 machines
Cc: pho...@chromium.org
Owner: shuqianz@chromium.org
Assigning to current deputy.
Richard, do we have so many available DUTs to be allocated to this new pool? Will it affect the CQ performance after the allocation?
Owner: shapiroc@chromium.org
I met with shapiroc@ earlier today to go over this request,
and other matters.  We've filled in allocations for the
boards with sufficient supply:

$ pool_boards bvt-uni
      6 reef
      6 sand
      6 snappy

We're still waiting for a CL from shapiroc@ before we can declare
victory.

Project Member

Comment 16 by bugdroid1@chromium.org, Aug 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5

commit a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5
Author: C Shapiro <shapiroc@chromium.org>
Date: Tue Aug 01 21:39:52 2017

Changing unified builds to separate bvt-uni pool

In many cases, the lab is currently only provisioned with 6 DUTs (for bvt), which
works if only one builder is ever performing hardware tests.  However,
with the addition of the unified builder, we're now competing for
resources.

This switches unified builds to a separate pool, which means we'll need
to allocate hardware in the lab accordingly for each bringup.

New pool name is bvt-uni

BUG= chromium:749731 
TEST=none available

Change-Id: I5a38bf0111553a5d918801d21022aaef036efc92
Reviewed-on: https://chromium-review.googlesource.com/590688
Commit-Ready: C Shapiro <shapiroc@google.com>
Tested-by: C Shapiro <shapiroc@google.com>
Reviewed-by: C Shapiro <shapiroc@google.com>

[modify] https://crrev.com/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5/cbuildbot/config_dump.json
[modify] https://crrev.com/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5/lib/constants.py
[modify] https://crrev.com/a0cc29183bf8d8df50ce0df8c70a6f11d27ee3e5/cbuildbot/chromeos_config.py

Status: Fixed (was: Started)

Sign in to add a comment