New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 893224 link

Starred by 2 users

Issue metadata

Status: Duplicate
Merged: issue 891758
Owner: ----
Closed: Oct 8
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

grunt-release builders fail since 11124.0.0 in HWTest

Project Member Reported by djkurtz@google.com, Oct 8

Issue description

The last successful grunt-release build was R71-11124.0.0 @ 2018-10-03 11:13.  Since then every build has failed in HWTest:
https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=grunt-release

The most recent run (R71-11138.0.0 [0]) failed with "HWTest [sanity] [grunt] / TestLabFailure / [Test-Logs]: Suite job: ABORT".  Note, however, that HWTest [sanity] [careena] passed.
[0] https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933241305994312032

AFAICT, it looks like stdout shows that the test suite was scheduled but just never ran and eventually timed out after 90 minutes.

  Will return from run_suite with status: OK
  06:23:20: INFO: Re-run swarming_cmd to avoid buildbot salency check.
  06:23:20: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpQeOQ6E/tmp3x13aX/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name grunt-release/R71-11138.0.0-sanity --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:sanity' '--tags=build:grunt-release/R71-11138.0.0' '--tags=task_name:grunt-release/R71-11138.0.0-sanity' '--tags=board:grunt' -- /usr/local/autotest/site_utils/run_suite.py --build grunt-release/R71-11138.0.0 --board grunt --model grunt --suite_name sanity --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 1 --suite_min_duts 1 --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 95065195L, 'cidb_build_id': 3016594, 'datastore_parent_key': ('Build', 3016594, 'BuildStage', 95065195L)}" -m 246463015
  07:55:50: INFO: RetriableHttp: attempt 1 receiving status 503, will retry
  08:00:21: INFO: Re-run swarming_cmd to avoid buildbot salency check.
  08:00:21: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpQeOQ6E/tmp3x13aX/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name grunt-release/R71-11138.0.0-sanity --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:sanity' '--tags=build:grunt-release/R71-11138.0.0' '--tags=task_name:grunt-release/R71-11138.0.0-sanity' '--tags=board:grunt' -- /usr/local/autotest/site_utils/run_suite.py --build grunt-release/R71-11138.0.0 --board grunt --model grunt --suite_name sanity --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 1 --suite_min_duts 1 --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 95065195L, 'cidb_build_id': 3016594, 'datastore_parent_key': ('Build', 3016594, 'BuildStage', 95065195L)}" -m 246463015
  08:25:46: INFO: OAuth token TTL expired, auto-refreshing (attempt 1/2)
  08:25:46: INFO: Refreshing access_token
  08:32:35: INFO: OAuth token TTL expired, auto-refreshing (attempt 1/2)
  08:32:35: INFO: Refreshing access_token
  08:38:59: INFO: OAuth token TTL expired, auto-refreshing (attempt 1/2)
  08:38:59: INFO: Refreshing access_token
  08:57:54: INFO: RetriableHttp: attempt 1 receiving status 503, will retry
  09:03:42: INFO: RetriableHttp: attempt 1 receiving status 503, will retry
  09:24:24: WARNING: Exception is not retriable return code: 3; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpQeOQ6E/tmp3x13aX/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name grunt-release/R71-11138.0.0-sanity --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:sanity' '--tags=build:grunt-release/R71-11138.0.0' '--tags=task_name:grunt-release/R71-11138.0.0-sanity' '--tags=board:grunt' -- /usr/local/autotest/site_utils/run_suite.py --build grunt-release/R71-11138.0.0 --board grunt --model grunt --suite_name sanity --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 1 --suite_min_duts 1 --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 95065195L, 'cidb_build_id': 3016594, 'datastore_parent_key': ('Build', 3016594, 'BuildStage', 95065195L)}" -m 246463015
  Triggered task: grunt-release/R71-11138.0.0-sanity

 
possibly related to  crbug.com/891758 
Mergedinto: 891758
Status: Duplicate (was: Unconfirmed)
Yep, same bug. Aviv just said he's looking at it in the oncall handoff meeting.

Sign in to add a comment