Recent HWTest all encounter issues in HWTest:
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-release/builds/878/steps/HWTest%20%5Bmoblab%5D/logs/stdio
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-release/builds/879/steps/HWTest%20%5Bmoblab%5D/logs/stdio
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-release/builds/881/steps/HWTest%20%5Bmoblab%5D/logs/stdio
12:59:43: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmp_CX3lr/tmpXT7lCt/temp_summary.json --raw-cmd --task-name guado_moblab-release/R52-8215.0.0-moblab --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 10800 --io-timeout 10800 --hard-timeout 10800 --expiration 1200 -- /usr/local/autotest/site_utils/run_suite.py --build guado_moblab-release/R52-8215.0.0 --board guado_moblab --suite_name moblab --pool bvt --num 1 --file_bugs False --priority Build --timeout_mins 120 --retry True --max_retries 10 --offload_failures_only False -m 60582847
14:07:21: INFO: Refreshing due to a 401 (attempt 1/2)
14:07:21: INFO: Refreshing access_token
15:03:00: ERROR: wait_cmd has lab failures: cwd=None.
Exception will be raised in the next json_dump run.
Autotest instance: cautotest
...
04-19-2016 [13:59:49] printing summary of incomplete jobs (1):
moblab_SmokeSuite: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=60582849
The suite job has another 0:59:41.701376 till timeout.
The suite job has another 0:29:35.981907 till timeout.
The suite job has another -1 day, 23:59:35.578245 till timeout.
Suite timed out. Started on 04-19-2016 [12:59:41], timed out on 04-19-2016 [15:02:39]
Suite job [ FAILED ]
Suite job ABORT:
Suite timings:
Downloads started at 2016-04-19 12:59:36
Payload downloads ended at 2016-04-19 12:59:41
Suite started at 2016-04-19 12:59:53
Artifact downloads ended (at latest) at 2016-04-19 12:59:54
Testing started at None
Testing ended at None
...
Attempting to diagnose pool: bvt
host: chromeos2-row5-rack10-host1, status: Repair Failed, locked: False diagnosis: Failed repair
host: chromeos2-row5-rack10-host8, status: Repairing, locked: False diagnosis: Failed repair
host: chromeos2-row5-rack10-host9, status: Running, locked: False diagnosis: Working
host: chromeos2-row5-rack10-host10, status: Repair Failed, locked: False diagnosis: Failed repair
host: chromeos2-row5-rack10-host11, status: Repair Failed, locked: False diagnosis: Failed repair
host: chromeos2-row5-rack10-host12, status: Repairing, locked: False diagnosis: Unused
Reason: Suite job failed or provisioning failed.
...
@@@STEP_FAILURE@@@
15:03:14: ERROR: ** HWTest did not complete due to infrastructure issues (code 3) **
-----
As of now, out of the 6 guado_moblab in pool:bvt, 4 of them are "Repair Failed", one "Repairing", one "Cleaning", no sure if this is normal.
Comment 1 by benhenry@google.com
, Apr 26 2016Labels: -Infra-ChromeOS