Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: WontFix
Owner:
Closed: Feb 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment
wizpig: provision failures on chromeos2-row8-rack8-host9
Project Member Reported by drinkcat@chromium.org, Nov 22 2016 Back to list
I see a lot of failures in wizpig-release recently: https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release

In some of these, the HWTest simply seems to timeout.

./dut_status.py -b wizpig -p bvt
hostname                       S   last checked         URL
chromeos2-row8-rack6-host6     OK  2016-11-21 15:56:52  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack6-host6/291535-reset/
chromeos2-row8-rack8-host20    OK  2016-11-21 15:51:10  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host20/291531-reset/
chromeos2-row8-rack8-host14    OK  2016-11-21 15:42:05  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host14/291527-reset/
chromeos2-row8-rack8-host16    OK  2016-11-21 15:55:54  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host16/291533-reset/
chromeos2-row8-rack8-host19    NO  2016-11-21 16:08:18  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host19/291543-repair/
chromeos2-row8-rack8-host9     OK  2016-11-21 13:04:01  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/291321-repair/

chromeos2-row8-rack8-host19 needs repair, and I think chromeos2-row8-rack8-host9 also has issues (it has not run a good test in 3 days):

./dut_status.py chromeos2-row8-rack8-host9 -f -d 120
    2016-11-21 13:04:01  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/291321-repair/
    2016-11-21 10:02:08  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/291075-provision/
    2016-11-21 10:01:29  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/291073-repair/
    2016-11-21 07:33:17  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/290612-provision/
    2016-11-21 07:32:36  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/290610-repair/
    2016-11-21 04:38:03  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/290270-provision/
    2016-11-21 04:30:54  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/290260-repair/
    2016-11-21 01:34:45  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/289875-provision/
    2016-11-21 01:34:07  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/289873-repair/
    2016-11-20 22:38:25  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/289529-provision/
    2016-11-20 18:07:29  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/289299-repair/
    2016-11-20 15:10:39  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/289124-provision/
    2016-11-20 10:49:45  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/288903-repair/
    2016-11-20 07:52:52  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/288699-provision/
    2016-11-20 06:39:32  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/288648-repair/
    2016-11-20 03:03:16  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/288019-provision/
    2016-11-20 03:02:33  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/288016-repair/
    2016-11-20 00:00:46  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/287552-provision/
    2016-11-19 19:42:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/287291-repair/
    2016-11-19 16:40:30  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/287108-provision/
    2016-11-19 14:15:30  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/287039-cleanup/
    2016-11-19 13:51:10  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/86394555-chromeos-test/
    2016-11-19 13:50:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/286987-reset/
    2016-11-19 13:50:15  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/286983-cleanup/
    2016-11-19 13:25:56  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/86394375-chromeos-test/
    2016-11-19 13:25:33  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/286906-reset/
    2016-11-19 13:25:01  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/286904-cleanup/
    2016-11-19 13:00:43  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/86394202-chromeos-test/
    2016-11-19 13:00:20  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/286834-reset/
    2016-11-19 12:59:49  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack8-host9/286833-cleanup/

So we are probably down to 4 DUTs in pool:bvt.

shuqianz: Can you please help rebalance?

 
Balanced. Only the chromeos2-row8-rack8-host19  is replaced. chromeos2-row8-rack8-host9 is in Ready state, so it won't be removed from bvt pool by the script. I will take a look at the DUT tomorrow to see whether it needs a manual swap
Summary: wizpig: provision failures on chromeos2-row8-rack8-host9 (was: wizpig: Rebalance pool:bvt)
A bit more data:

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/595
Suite fails because:
- Job: wizpig-release/R57-9010.0.0/bvt-inline/security_ChromiumOSLSM (86731395-chromeos-test)
  Scheduled to run on chromeos2-row8-rack8-host9

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/594
Suite fails because:
- Job: wizpig-release/R57-9009.0.0/bvt-inline/security_ASLR (86625930-chromeos-test)
  Scheduled to run on chromeos2-row8-rack8-host9
- Job: wizpig-release/R57-9009.0.0/bvt-inline/security_ASLR (86678915-chromeos-test)
  Scheduled to run on chromeos2-row8-rack8-host14 (not sure what that's about, but appears to be a retry of the above)

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/593
Suite fails because:
- Job: wizpig-release/R57-9008.0.0/bvt-inline/login_OwnershipRetaken (86578628-chromeos-test)
  Scheduled to run on chromeos2-row8-rack8-host9

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/591
Suite fails because:
- Job: wizpig-release/R57-9007.0.0/bvt-inline/provision_AutoUpdate.double (86537288-chromeos-test)
  Scheduled to run on chromeos2-row8-rack8-host9
- Job: wizpig-release/R57-9007.0.0/bvt-inline/provision_AutoUpdate.double (86565373-chromeos-test)
  Scheduled to run on chromeos2-row8-rack6-host6 (retry?)

https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/590
Suite fails because:
- Job: wizpig-release/R57-9005.0.0/bvt-inline/login_OwnershipRetaken (86416368-chromeos-test)
  Scheduled to run on chromeos2-row8-rack8-host9

I think the pattern is quite clear, maybe rebalancing won't help if anything still gets scheduled on chromeos2-row8-rack8-host9.

Maybe we should lock chromeos2-row8-rack8-host9 for the time being?
Yeah, please lock it tonight to avoid more failures caused by this DUT.
Labels: -Pri-1 Pri-2
Looks good now, wizpig-release turned green:
https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/599

We should investigate what is wrong with chromeos2-row8-rack8-host9, though.
Owner: dshi@chromium.org
Pass to deputy
Comment 7 by autumn@chromium.org, Nov 29 2016
Labels: -current-issue
Comment 8 by dshi@chromium.org, Feb 6 2017
Status: WontFix
Sign in to add a comment