CQ repeatedly failing with exceptions in HWTest |
|||
Issue descriptionThe CrOS CQ has been continuously failing and retrying since around 2017-09-30 14:00 PDT. See e.g. the history on https://chromium-review.googlesource.com/c/chromiumos/platform/tast-tests/+/693814. It looks like most (all?) boards are failing, typically with exceptions in HWTest stages (e.g. https://luci-milo.appspot.com/buildbot/chromeos/wolf-paladin/15819). I don't see anything particularly helpful in the output from that stage, though: @@@BUILD_STEP@HWTest [bvt-inline]@@@ ************************************************************ @@@STEP_LINK@stdout-->stdio@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fwolf-paladin%2F15819%2F%2B%2Frecipes%2Fsteps%2FHWTest__bvt-inline_%2F0%2Fstdout@@@ ** Start Stage HWTest [bvt-inline] - Sat, 30 Sep 2017 13:58:03 -0700 (PDT) ** ** Stage that runs tests in the Autotest lab. ************************************************************ 13:58:03: INFO: Created cidb engine bot@130.211.191.11 for pid 29423 13:58:03: INFO: Running cidb query on pid 29423, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f7ce5f87dd0> 13:58:03: INFO: Waiting up to forever for payloads and test artifacts ... Preconditions for the stage successfully met. Beginning to execute stage... 14:02:09: INFO: Running cidb query on pid 29423, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f7ce5d88090> 14:02:09: INFO: RunCommand: /b/c/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpv9Ylyb/tmph6O8oo/temp_summary.json --raw-cmd --task-name wolf-paladin/R63-9990.0.0-rc2-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:bvt-inline' '--tags=build:wolf-paladin/R63-9990.0.0-rc2' '--tags=task_name:wolf-paladin/R63-9990.0.0-rc2-bvt-inline' '--tags=board:wolf' -- /usr/local/autotest/site_utils/run_suite.py --build wolf-paladin/R63-9990.0.0-rc2 --board wolf --suite_name suite_attr_wrapper --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --offload_failures_only False --suite_args "{'attr_filter': u'(suite:bvt-inline) and (subsystem:default)'}" --job_keyvals "{'cidb_build_stage_id': 57732649L, 'cidb_build_id': 1902955, 'datastore_parent_key': ('Build', 1902955, 'BuildStage', 57732649L)}" -c Autotest instance created: cautotest-prod 09-30-2017 [14:02:15] Submitted create_suite_job rpc 09-30-2017 [14:03:27] Created suite job: http://cautotest-prod.corp.google.com/afe/#tab_id=view_job&object_id=146017790 @@@STEP_LINK@Link to suite@http://cautotest-prod.corp.google.com/afe/#tab_id=view_job&object_id=146017790@@@ --create_and_return was specified, terminating now. Will return from run_suite with status: OK 14:03:30: INFO: RunCommand: /b/c/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpv9Ylyb/tmpjtRyoI/temp_summary.json --raw-cmd --task-name wolf-paladin/R63-9990.0.0-rc2-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:bvt-inline' '--tags=build:wolf-paladin/R63-9990.0.0-rc2' '--tags=task_name:wolf-paladin/R63-9990.0.0-rc2-bvt-inline' '--tags=board:wolf' -- /usr/local/autotest/site_utils/run_suite.py --build wolf-paladin/R63-9990.0.0-rc2 --board wolf --suite_name suite_attr_wrapper --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --offload_failures_only False --suite_args "{'attr_filter': u'(suite:bvt-inline) and (subsystem:default)'}" --job_keyvals "{'cidb_build_stage_id': 57732649L, 'cidb_build_id': 1902955, 'datastore_parent_key': ('Build', 1902955, 'BuildStage', 57732649L)}" -m 146017790 14:09:25: INFO: Refreshing due to a 401 (attempt 1/2) 14:09:25: INFO: Refreshing access_token 14:13:04: INFO: Refreshing due to a 401 (attempt 1/2) 14:13:04: INFO: Refreshing access_token -------------------------------------------------------------------------------- started: Sat Sep 30 14:07:07 2017 ended: Sat Sep 30 14:34:22 2017 duration: 27 mins, 15 secs status: EXCEPTION status reason: step was interrupted. I see other similar recent bug reports that may be related: issue 757351 issue 760016 b/64876483
,
Oct 16 2017
,
Jan 22 2018
,
Jan 23 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by sheriffbot@chromium.org
, Oct 16 2017