New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 770567 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

CQ repeatedly failing with exceptions in HWTest

Project Member Reported by derat@chromium.org, Oct 1 2017

Issue description

The CrOS CQ has been continuously failing and retrying since around 2017-09-30 14:00 PDT. See e.g. the history on https://chromium-review.googlesource.com/c/chromiumos/platform/tast-tests/+/693814.

It looks like most (all?) boards are failing, typically with exceptions in HWTest stages (e.g. https://luci-milo.appspot.com/buildbot/chromeos/wolf-paladin/15819). I don't see anything particularly helpful in the output from that stage, though:

@@@BUILD_STEP@HWTest [bvt-inline]@@@
************************************************************
@@@STEP_LINK@stdout-->stdio@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fwolf-paladin%2F15819%2F%2B%2Frecipes%2Fsteps%2FHWTest__bvt-inline_%2F0%2Fstdout@@@
** Start Stage HWTest [bvt-inline] - Sat, 30 Sep 2017 13:58:03 -0700 (PDT)
** 
** Stage that runs tests in the Autotest lab.
************************************************************
13:58:03: INFO: Created cidb engine bot@130.211.191.11 for pid 29423
13:58:03: INFO: Running cidb query on pid 29423, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f7ce5f87dd0>
13:58:03: INFO: Waiting up to forever for payloads and test artifacts ...
Preconditions for the stage successfully met. Beginning to execute stage...
14:02:09: INFO: Running cidb query on pid 29423, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f7ce5d88090>
14:02:09: INFO: RunCommand: /b/c/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpv9Ylyb/tmph6O8oo/temp_summary.json --raw-cmd --task-name wolf-paladin/R63-9990.0.0-rc2-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:bvt-inline' '--tags=build:wolf-paladin/R63-9990.0.0-rc2' '--tags=task_name:wolf-paladin/R63-9990.0.0-rc2-bvt-inline' '--tags=board:wolf' -- /usr/local/autotest/site_utils/run_suite.py --build wolf-paladin/R63-9990.0.0-rc2 --board wolf --suite_name suite_attr_wrapper --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --offload_failures_only False --suite_args "{'attr_filter': u'(suite:bvt-inline) and (subsystem:default)'}" --job_keyvals "{'cidb_build_stage_id': 57732649L, 'cidb_build_id': 1902955, 'datastore_parent_key': ('Build', 1902955, 'BuildStage', 57732649L)}" -c
Autotest instance created: cautotest-prod
09-30-2017 [14:02:15] Submitted create_suite_job rpc
09-30-2017 [14:03:27] Created suite job: http://cautotest-prod.corp.google.com/afe/#tab_id=view_job&object_id=146017790
@@@STEP_LINK@Link to suite@http://cautotest-prod.corp.google.com/afe/#tab_id=view_job&object_id=146017790@@@
--create_and_return was specified, terminating now.
Will return from run_suite with status: OK
14:03:30: INFO: RunCommand: /b/c/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpv9Ylyb/tmpjtRyoI/temp_summary.json --raw-cmd --task-name wolf-paladin/R63-9990.0.0-rc2-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:bvt-inline' '--tags=build:wolf-paladin/R63-9990.0.0-rc2' '--tags=task_name:wolf-paladin/R63-9990.0.0-rc2-bvt-inline' '--tags=board:wolf' -- /usr/local/autotest/site_utils/run_suite.py --build wolf-paladin/R63-9990.0.0-rc2 --board wolf --suite_name suite_attr_wrapper --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --offload_failures_only False --suite_args "{'attr_filter': u'(suite:bvt-inline) and (subsystem:default)'}" --job_keyvals "{'cidb_build_stage_id': 57732649L, 'cidb_build_id': 1902955, 'datastore_parent_key': ('Build', 1902955, 'BuildStage', 57732649L)}" -m 146017790
14:09:25: INFO: Refreshing due to a 401 (attempt 1/2)
14:09:25: INFO: Refreshing access_token
14:13:04: INFO: Refreshing due to a 401 (attempt 1/2)
14:13:04: INFO: Refreshing access_token

--------------------------------------------------------------------------------
started: Sat Sep 30 14:07:07 2017
ended: Sat Sep 30 14:34:22 2017
duration: 27 mins, 15 secs
status: EXCEPTION
status reason: step was interrupted.



I see other similar recent bug reports that may be related:

issue 757351
 issue 760016 
b/64876483
 
Project Member

Comment 1 by sheriffbot@chromium.org, Oct 16 2017

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 2 by ntang@chromium.org, Oct 16 2017

Status: Fixed (was: Assigned)
The issue in the lab has been fixed, similar to b/760016

Comment 3 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Comment 4 by dchan@chromium.org, Jan 23 2018

Status: Fixed (was: Archived)

Sign in to add a comment