New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 763111 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Sep 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

162 ChromeOS duts don't have pool labels

Project Member Reported by dshi@chromium.org, Sep 7 2017

Issue description

$./cli/atest host list | grep -v "pool:" | grep  "os:cros" | wc -l
162

The duts are not running any tests, some starting at 8/19
I checked several veyron_jerry, seem to be working just fine. I remember some had pool:suites label. Somehow the pool labels are gone and leave the duts doing nothing.
 

Comment 1 by dshi@chromium.org, Sep 7 2017

Cc: jrbarnette@chromium.org
+jrbarnette

not sure if balance_pool can cause that.
> not sure if balance_pool can cause that.

Probably.  Ultimately, I think the problem is shards.  We've made some
recent changes that might make this happen less...

On the topic of the numbers there's this:
$ atest host list -b os:cros | grep -v pool: | grep board: | count_labels -b | sort -n | tail -10
      5 lulu
      6 veyron_jerry
      7 kevin
      7 stout
      7 sumo
      7 terra
      8 sand
      9 parrot
     10 heli
     10 rikku

Cc: -jrbarnette@chromium.org
Owner: jrbarnette@chromium.org
How do we plan to handle these DUTs?
Status: Started (was: Untriaged)
> How do we plan to handle these DUTs?

We'll re-add them to pool:suites, once I've done some sanity checks.

OK.  All the DUTs that seemed safe to have pool:suites got it.

There's a handful of DUTs that for various reasons couldn't be
added:

$ atest host list -b os:cros | grep -v pool: | grep board: | cut -c 1-46,203-
chromeos3-row1-rack1-host9     Ready           bluetooth, webcam, hw_video_acc_h264, hw_video_acc_vp8, hw_video_acc_enc_h264, ec:cros, storage:mmc, touchpad, power:battery, hw_video_acc_vp9, internal_display, hw_jpeg_acc_dec, board:celes, os:cros, phase:DVT, variant:celes, sku:celes_intel_celeron_n3050_4Gb, cts_abi_x86, cts_abi_arm
chromeos3-row1-rack2-host12    Ready           bluetooth, storage:mmc, power:battery, board:whirlwind, os:cros, phase:EVT, sku:whirlwind_cpu_qualcomm_1Gb, cts_abi_arm
chromeos1-row3-rack11-host1    Ready           servo, bluetooth, webcam, hw_video_acc_h264, hw_video_acc_enc_h264, ec:cros, storage:mmc, board:kip, power:battery, hw_jpeg_acc_dec, os:cros, cts_abi_x86, cts_abi_arm
chromeos1-row2-rack9-host4     Ready           servo, bluetooth, webcam, hw_video_acc_h264, hw_video_acc_vp8, hw_video_acc_enc_h264, ec:cros, storage:mmc, accel:cros-ec, power:battery, hw_video_acc_vp9, internal_display, board:cyan, hw_jpeg_acc_dec, fwro-version:cyan-firmware/R46-7287.57.32, os:cros, fwrw-version:cyan-firmware/R46-7287.57.32, cts_abi_x86, cts_abi_arm, arc, cros-version:cyan-release/R56-9000.91.0
chromeos4-row5-rack1-host9     Ready           servo, bluetooth, webcam, lightsensor, hw_video_acc_h264, board:peppy, ec:cros, chameleon, storage:ssd, video_glitch_detection, touchpad, power:battery, chameleon:hdmi, audio_loopback_dongle, hw_video_acc_vp9, internal_display, hw_jpeg_acc_dec, os:cros, variant:peppy, sku:peppy_intel_celeron_2955U_2Gb, phase:PVT, cts_abi_x86, cts_abi_arm, usb_nic:asix, cros-version:peppy-release/R62-9856.0.0
chromeos9-row4-rack7-host1     Ready           servo, bluetooth, webcam, hw_video_acc_h264, board:peppy, hw_video_acc_enc_h264, ec:cros, chameleon, storage:ssd, video_glitch_detection, power:battery, hw_video_acc_vp9, chameleon:, hw_jpeg_acc_dec, os:cros, cts_abi_x86, cts_abi_arm, chameleon:bt_hid
chromeos1-test-station         Repair Failed   servo, bluetooth, webcam, hw_video_acc_h264, hw_video_acc_vp8, hw_video_acc_enc_h264, ec:cros, storage:mmc, accel:cros-ec, power:battery, hw_jpeg_acc_dec, os:cros, cts_abi_x86, cts_abi_arm, arc, board:caroline
chromeos9-test-station         Ready           servo, bluetooth, webcam, hw_video_acc_h264, hw_video_acc_vp8, hw_video_acc_enc_h264, ec:cros, storage:mmc, accel:cros-ec, touchpad, touchscreen, power:battery, hw_video_acc_vp9, internal_display, board:cyan, hw_jpeg_acc_dec, os:cros, variant:cyan, phase:PVT, cts_abi_x86, cts_abi_arm, arc, sku:cyan_intel_celeron_n3160_4Gb, cros-version:cyan-release/R61-9723.0.0

... and, once again, pool: label assignments seem to get
lost or mislaid.  We're back to some DUTs that ought to have
labels not having them:

    $ atest host list -b os:cros | grep '^chromeos[246]-' | grep -v pool: | grep board: | grep -v chameleon | wc -l
    18

At the time of c#5, the count was 0; a week later, 18 DUTs have lost
a pool: label.

For the moment, we can pump out the leaky boat _again_:

$ atest label add -m $M pool:suites
Added to label 'pool:suites' hosts: 
	chromeos2-row3-rack4-host5, chromeos2-row3-rack5-host5,
	chromeos2-row4-rack9-host10, chromeos2-row4-rack9-host11,
	chromeos2-row8-rack1-host3, chromeos2-row8-rack1-host5,
	chromeos2-row8-rack4-host8, chromeos2-row8-rack8-host12,
	chromeos2-row8-rack8-host16, chromeos2-row8-rack8-host20,
	chromeos2-row8-rack9-host16, chromeos4-row10-rack6-host21,
	chromeos4-row11-rack7-host2, chromeos4-row5-rack7-host9,
	chromeos4-row8-rack6-host7, chromeos6-row1-rack15-host11,
	chromeos6-row1-rack16-host15, chromeos6-row1-rack23-host15

Status: Fixed (was: Started)
$ atest host list -b os:cros | grep '^chromeos[246]-' | grep -v pool: | grep board: | grep -v chameleon | wc -l
0

... So, I'm declaring victory.

Sign in to add a comment