Several boards have insufficient DUTs in pool:suites Paygen tests not running (even after kicking off multiple reruns) |
|||||||||
Issue descriptionThe pagygen_au_beta tests for several boards did not run for 11151.45.0, 71.0.3578.71 Sample boards: chell: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=3163325 soraka: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=3163430 We have kicked off several rounds of reruns but they keep getting aborted too: https://stainless.corp.google.com/search?view=matrix&row=board_model&col=build&first_date=2018-11-25&last_date=2018-11-29&suite=%5Epaygen_au_beta&build=R71-11151.45.0&board=chell%7Csoraka%7Cveyron_minnie%7Cveyron_speedy%7Csentry&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=false&exclude_acts=true&exclude_retried=true&exclude_non_production=false There seems to be enough DUTs free. This seems to be happening more frequently so I would like some help figure out what is going on and some tips on how to figure it out myself in future
,
Nov 29
Same for soraka
,
Nov 29
Reruns has been aborted for Monroe & Sentry for this build. Autotest job links: https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=261988642 https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=261988630
,
Nov 29
Minnie and speedy also had the same problem
,
Nov 29
These are all models with no spare DUTs in suites pool. There's not much I can do. Depending on the urgency I can poke the lab tech to prioritize these devices for repair.
,
Nov 30
I just check and result below. Looks like - soraka 5/6 pool suites repair failed - chell 2/3 pool suites repairing - speedy 2/2 repair failed I think we need 1) open ticket to have 'em repair 2) add more dut soraka (5 suites in repair failed state) 10 bvt 3-Cleaning 0-Ready 0-Repair Failed 3-Resetting 0-Running 4-Verifying 6 suites 0-Cleaning 0-Ready 5-Repair Failed 0-Resetting 0-Running 1-Verifying chell (we should be ok here) 5 bvt 4-Ready 0-Repair Failed 1-Repairing 0-Resetting 0-Running 3 suites 0-Ready 1-Repair Failed 2-Repairing 0-Resetting 0-Running veyron-speedy (2 suites and 2 repair failed) 11 bvt 0-Provisioning 9-Ready 2-Repair Failed 0-Resetting 0-Running 2 suites 0-Provisioning 0-Ready 2-Repair Failed 0-Resetting 0-Running
,
Nov 30
,
Nov 30
,
Dec 6
Note that we had to skip minnie in the past two betas. Ping since a while since the last update and stable approaches... Thanks
,
Dec 6
,
Dec 12
Hi, monroe and daisy-skate failed M71-BETA-CHROMEOS-7 (11151.59.0, 71.0.3578.94) for similar. Same issue?
,
Dec 12
Issue 914207 has been merged into this issue.
,
Dec 12
Plan to resolve for monroe? It'll miss stable otherwise. Thanks
,
Dec 12
+ New infra deputy
,
Dec 12
Though it is saying they're OOO
,
Dec 13
Monroe suites pool is dying: 10 bvt 8-Ready 1-Repair Failed 1-Repairing 0-Running 0-Locked 14 suites 0-Ready 10-Repair Failed 4-Repairing 0-Running 0-Locked might need to run it in bvt pool?
,
Dec 14
,
Dec 14
,
Dec 18
,
Dec 18
Can we escalate? Not having DUTS means failing tests means not including the boards in the releases... Thanks
,
Dec 18
Re #20, Monroe suites DUTs are offline, here's the ticket for lab to repair them: https://b.corp.google.com/issues/121209369 |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by ayatane@chromium.org
, Nov 29