New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 813916 link

Starred by 1 user

Issue metadata

Status: Verified
Last visit > 30 days ago
Closed: Mar 2018
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Sign in to add a comment

"Not enough DUTs for board: expresso"

Project Member Reported by, Feb 20 2018

Issue description

expresso-release has been failing for ~2 weeks, it can't find DUTs

The HWTest stages show:

NotEnoughDutsError: Not enough DUTs for board: expresso, pool: bvt; required: 4, found: 3


Assigning to deputy.

There are 6 expresso in pool:bvt. 3 of them are locked by afaris@ for the reason "battery swelling: send out for replacement".

Checked other pools. Most of them are affected by the battery swelling issue.
It looks like we have 0 spares. 3 DUTs aren't enough to run the tests before timeout.

I'm not sure what to do except push for the replacements.

Status: Started (was: Assigned)
This is still happening:

If we don't expect to get replacements anytime soon, can we make this release builder experimental for now?

Comment 6 by, Mar 20 2018

Still failing:

I will put up a CL to mark this builder (and maybe some others) as experimental.
> I will put up a CL to mark this builder (and maybe some others) as experimental.

Wait, I've just checked inventory.  The database records some 30
expresso DUTs.  I don't know if they all work, but that's enough.

However, the automated inventory believes there are only 11 DUTs,
so there's a disconnect somewhere.  We should sort out the infra
problem first...

Digging deeper into the inventory, most of the expresso units aren't
working because of battery problems.  Replacements are on order.
When you look at what's left over, it's woefully inadequate:
$ atest host list -b board:expresso --unlocked | count_labels -p
      3 bvt
      2 cts
      1 performance
      1 suites
      1 wificell
      1 wifichaos

For now, I think the right answer is to re-assign the cts, performance,
and suites pools to bvt, so that we can at least cover the release

All right after fighting (repeatedly) with balance-pool, I broke down
and manually reassigned pool labels until all working DUTs are in the
BVT pool:
    $ dut-status -b expresso -p bvt
    hostname                       S   last checked         URL
    chromeos4-row4-rack9-host5     OK  2018-03-20 12:00:55
    chromeos4-row4-rack9-host6     OK  2018-03-20 12:01:01
    chromeos4-row4-rack8-host20    OK  2018-03-20 12:05:46
    chromeos4-row4-rack9-host8     OK  2018-03-20 12:05:46
    chromeos2-row6-rack9-host6     OK  2018-03-20 11:59:55
    chromeos2-row6-rack9-host3     OK  2018-03-20 12:07:16

I'll note that I'm continuing to fight with the system:  Even after
assigning DUTs to pools, they keep forgetting their pool label.

OK.  The system _seems_ to have finally decided to remember the
pool assignments.  Here's where we stand with pool assignments:
    $ atest host list -b board:expresso | count_labels -p
          6 bvt
         25 suites
          1 wificell
          1 wifichaos

Everything in pool:suites is broken.  One DUT seems to be stuck in
repair; the others are all locked and awaiting replacement.

Status: Verified (was: Started)

Sign in to add a comment