chromeos2-row1-rack4-host1 host is down.
issue: https://b.corp.google.com/issues/29274611


chromeos2-row1-rack4-host2 and 
chromeos2-row1-rack4-host3 are working fine.

chromeos2-row1-rack4-host4: jack plugger is not connected to audio board
https://b.corp.google.com/issues/29276776

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3f87e0b3aea250ca1050a8325aa3e35cc9c3aafc

commit 3f87e0b3aea250ca1050a8325aa3e35cc9c3aafc
Author: Sridhar Sonti <sontis@chromium.org>
Date: Fri Jun 10 23:44:19 2016

Created new control files for bvt-perbuild test suite.

BUG= chromium:615455 
TEST=None

Change-Id: I908f12415d9f27096be607dc76c1b663e2bbabc9
Reviewed-on: https://chromium-review.googlesource.com/351680
Commit-Ready: Sridhar Sonti <sontis@chromium.org>
Tested-by: Sridhar Sonti <sontis@chromium.org>
Reviewed-by: Kalin Stoyanov <kalin@chromium.org>

[add] https://crrev.com/3f87e0b3aea250ca1050a8325aa3e35cc9c3aafc/server/site_tests/audio_AudioBasicHeadphone/control.bvt
[add] https://crrev.com/3f87e0b3aea250ca1050a8325aa3e35cc9c3aafc/server/site_tests/audio_AudioBasicExternalMicrophone/control.bvt

The tests did not started running b/c the boards we have with chameleon are not on pool:bvt.

+Dan, Infra label

What should be the preferred way to do it?
1) add these chromeos2- DUTs to pool:bvt
2) Add chameleon boards to chromeos4- DUTs

Adding chromeos2 duts to bvt pool is fine. There is no blocking issue.

Thanks Dan.

The pool:bvt seems to be monitored by the pool_balance script. I recalled that there is a rule to filter out the DUTs with multiple pools. As these devices have multiple pools labeled, I am not sure if it will cause issues. Dan, please help to clarify.

Yes, the jerry DUTs in Atlantis with chameleon boards are on pool:suites. No wonder they haven't started bvt tests. 

+Richard

right, bvt pool is affected by pool_balance script. The balance is executed manually by infra deputy though. Will these duts with chameleon be in multiple pools? 

I am no tsure about the 'multiple pools' state. I added these four DUTs to pool:bvt today, aiming to start few chameleon tests running as part of bvt-perbuild.

The intent is Chameleon DUTs are in both pool:suites and
pool:chameleon.  The balance_pool script won't change DUTs
with two pool: labels.

Currently, it's not supported to put Chameleon DUTs into a
critical pool like pool:bvt.  Devices in critical pools
are considered to be fungible; distinctions like "chameleon"
break that requirement.  For that reason, it's also not
supported to put Chameleon tests into any BVT test suite.

It wouldn't be terribly hard to arrange for a Chameleon test
suite to be requested by canary/release builders, but it
requires changes in both chromite and Autotest.

Why do you require that this test run on request from the
builder, instead of from suite_scheduler?

Hmmm...  I see we now have some veyron_jerry DUTs with
chameleon in pool:bvt.  Who did that?  We'll need to fix it...

I did assign the pool:bvt to these four DUTs as I mentioned in c#9.

> I did assign the pool:bvt to these four DUTs as I mentioned in c#9.

For future reference:  Generally it's a bad idea to manually apply
labels to DUTs, especially pool labels.  Even when it's the right thing,
these actions should be coordinated with the Infra Deputy.

Good point. I am removing the label from these four boards.

Now, how do we proceed from here?

Start by answering the question in c#10:

> Why do you require that this test run on request from the
> builder, instead of from suite_scheduler?

Hmm, I see they started running the bvt tests and the few chameleon tests ran as part of bvt-perbuild.
So, what would be the right thing to do here?
- leave as is?
- remove pool:suites?
- remove pool:bvt, and find way to get chameleon labels on chromeos4 DUTs that are already on pool:bvt (this will involve chameleon installation)?

Re c#15(c#10) It is chameleon team goal to start running basic and short/stable chameleon tests on BVT to catch audio issues earlier than post-build stage.

...audio and display issues...

What fraction of boards have a chameleon attached? How many chameleon-dependent tests are you going to add?

I'm concerned that even if you add a few chameleon boards to bvt pool now, over time the balance_pool script will diffuse them around, and eventually we may end up with 0 (which will cause your test not to  run) or, worse 1 (which may cause that test to fail if only 1 DUT is eligible to run it).

For this goal(chameleon on bvt) I am planning to have two different board-family boards by four DUTs. Total of 8 DUTs, for running two-to-five audio and two-to-four display tests.

For a different goal(current non-bvt suites) I am planning on expanding chameleon footprint in lab to two boards per board-family and each board per four DUTs. Total of about ~ 12x2x4 = 100 boards - b/28384559. 

> Re c#15(c#10) It is chameleon team goal to start running basic and
> short/stable chameleon tests on BVT to catch audio issues earlier
> than post-build stage.

In principle, a per-build out-of-band test should run within a few
hours (or maybe up to a day?) after the build.  Is the problem that
you don't want to wait those few hours?  Or is the problem that the
tests don't reliably run often enough?  I would expect that a dedicated
pool for the suite ought to run reliably, but I don't know about the
latency.

If you really have a requirement (for whatever reason) that the tests
be started by the builder rather than by the suite scheduler, then
I think this is the necessary solution:
  * In autotest, create a suite that contains just the chameleon tests
    you want to run.
  * In chromite change the releas builders you care about to request
    running that suite against pool:chameleon (or some other chosen
    pool) in a new asynchronous HWTest phase.
  * Make sure the pool you want to use has enough chameleon DUTs to
    support the test load.

The intention is to get issues like  issue 617273 (broken feature) and issue 588315(framework) visible earlier, having out-of-band tests not executed for some boards for more than a day, as there might not be a "good" build produced.

Thanks for the suggestion on eventual solution. I realize better get the second goal(larger chameleon footprint for out-of-band suites) strapped before I do any expansion to BVT.

I'll remove the pool:bvt label from chromeos2-row1-rack4-host1-4.

can you confirm what additional work is needed here?