New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Verified
Last visit > 30 days ago
Closed: Mar 2018
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Sign in to add a comment

Issue 813916: "Not enough DUTs for board: expresso"

Reported by, Feb 20 2018 Project Member

Issue description

expresso-release has been failing for ~2 weeks, it can't find DUTs

The HWTest stages show:

NotEnoughDutsError: Not enough DUTs for board: expresso, pool: bvt; required: 4, found: 3


Assigning to deputy.

Comment 1 by, Feb 20 2018

There are 6 expresso in pool:bvt. 3 of them are locked by afaris@ for the reason "battery swelling: send out for replacement".

Checked other pools. Most of them are affected by the battery swelling issue.

Comment 2 by, Feb 20 2018

It looks like we have 0 spares. 3 DUTs aren't enough to run the tests before timeout.

I'm not sure what to do except push for the replacements.

Comment 3 by, Feb 20 2018

Status: Started (was: Assigned)

Comment 4 by, Feb 26 2018


Comment 5 by, Feb 28 2018

This is still happening:

If we don't expect to get replacements anytime soon, can we make this release builder experimental for now?

Comment 6 by, Mar 20 2018

Still failing:

I will put up a CL to mark this builder (and maybe some others) as experimental.

Comment 7 by, Mar 20 2018

> I will put up a CL to mark this builder (and maybe some others) as experimental.

Wait, I've just checked inventory.  The database records some 30
expresso DUTs.  I don't know if they all work, but that's enough.

However, the automated inventory believes there are only 11 DUTs,
so there's a disconnect somewhere.  We should sort out the infra
problem first...

Comment 8 by, Mar 20 2018

Digging deeper into the inventory, most of the expresso units aren't
working because of battery problems.  Replacements are on order.
When you look at what's left over, it's woefully inadequate:
$ atest host list -b board:expresso --unlocked | count_labels -p
      3 bvt
      2 cts
      1 performance
      1 suites
      1 wificell
      1 wifichaos

For now, I think the right answer is to re-assign the cts, performance,
and suites pools to bvt, so that we can at least cover the release

Comment 9 by, Mar 20 2018

All right after fighting (repeatedly) with balance-pool, I broke down
and manually reassigned pool labels until all working DUTs are in the
BVT pool:
    $ dut-status -b expresso -p bvt
    hostname                       S   last checked         URL
    chromeos4-row4-rack9-host5     OK  2018-03-20 12:00:55
    chromeos4-row4-rack9-host6     OK  2018-03-20 12:01:01
    chromeos4-row4-rack8-host20    OK  2018-03-20 12:05:46
    chromeos4-row4-rack9-host8     OK  2018-03-20 12:05:46
    chromeos2-row6-rack9-host6     OK  2018-03-20 11:59:55
    chromeos2-row6-rack9-host3     OK  2018-03-20 12:07:16

I'll note that I'm continuing to fight with the system:  Even after
assigning DUTs to pools, they keep forgetting their pool label.

Comment 10 by, Mar 20 2018

OK.  The system _seems_ to have finally decided to remember the
pool assignments.  Here's where we stand with pool assignments:
    $ atest host list -b board:expresso | count_labels -p
          6 bvt
         25 suites
          1 wificell
          1 wifichaos

Everything in pool:suites is broken.  One DUT seems to be stuck in
repair; the others are all locked and awaiting replacement.

Comment 11 by, Mar 21 2018

Status: Verified (was: Started)

Sign in to add a comment