New issue
Advanced search Search tips

Issue 721887 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 715415
Owner: ----
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 715415



Sign in to add a comment

Not enough winky DUTs

Project Member Reported by ayatane@chromium.org, May 12 2017

Issue description

https://luci-milo.appspot.com/buildbot/chromeos/winky-paladin/1599

12:12:46: INFO: RunCommand: /b/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmprrONpz/tmptE4SFH/temp_summary.json --raw-cmd --task-name winky-paladin/R60-9545.0.0-rc1-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:bvt-inline' '--tags=build:winky-paladin/R60-9545.0.0-rc1' '--tags=task_name:winky-paladin/R60-9545.0.0-rc1-bvt-inline' '--tags=board:winky' -- /usr/local/autotest/site_utils/run_suite.py --build winky-paladin/R60-9545.0.0-rc1 --board winky --suite_name bvt-inline --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 10 --minimum_duts 4 --offload_failures_only True --job_keyvals "{'datastore_parent_key': ('Build', 1513338, 'BuildStage', 44662020L)}" -c
12:12:54: WARNING: Exception is not retriable return code: 3; command: /b/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmprrONpz/tmptE4SFH/temp_summary.json --raw-cmd --task-name winky-paladin/R60-9545.0.0-rc1-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:bvt-inline' '--tags=build:winky-paladin/R60-9545.0.0-rc1' '--tags=task_name:winky-paladin/R60-9545.0.0-rc1-bvt-inline' '--tags=board:winky' -- /usr/local/autotest/site_utils/run_suite.py --build winky-paladin/R60-9545.0.0-rc1 --board winky --suite_name bvt-inline --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 10 --minimum_duts 4 --offload_failures_only True --job_keyvals "{'datastore_parent_key': ('Build', 1513338, 'BuildStage', 44662020L)}" -c
Priority was reset to 100
Triggered task: winky-paladin/R60-9545.0.0-rc1-bvt-inline
chromeos-server22-34: 3616e5660e8dcf10 3
  Autotest instance: cautotest
  Bug filing disabled due to missing imports.
  Can't file pool: cq, board: winky in critical state
  TestLabException: Not enough DUTs for board: winky, pool: cq; required: 4, found: 0, suite: bvt-inline, build: winky-paladin/R60-9545.0.0-rc1
  Traceback (most recent call last):
    File "/usr/local/autotest/site_utils/run_suite.py", line 1805, in main
      code, output_dict = main_without_exception_handling(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 1623, in main_without_exception_handling
      raise e
  NotEnoughDutsError: Not enough DUTs for board: winky, pool: cq; required: 4, found: 0, suite: bvt-inline, build: winky-paladin/R60-9545.0.0-rc1
  Will return from run_suite with status: INFRA_FAILURE

 
Doesn't look like there's an actual DUT shortage

% dut-status -b winky -p cq
hostname                       S   last checked         URL
chromeos4-row3-rack13-host5    OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack13-host5/512576-verify/
chromeos4-row3-rack13-host15   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack13-host15/512569-verify/
chromeos4-row3-rack13-host17   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack13-host17/512572-verify/
chromeos4-row3-rack13-host19   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack13-host19/512579-verify/
chromeos4-row3-rack13-host21   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack13-host21/512565-verify/
chromeos4-row3-rack12-host5    OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host5/512571-verify/
chromeos4-row3-rack12-host7    OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host7/512566-verify/
chromeos4-row3-rack12-host9    OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host9/512557-verify/
chromeos4-row3-rack12-host11   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host11/512555-verify/
chromeos4-row3-rack12-host15   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host15/512562-verify/
chromeos4-row3-rack12-host13   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host13/512573-verify/
chromeos4-row3-rack12-host19   OK  2017-05-12 12:47:23  http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row3-rack12-host19/512552-verify/

Comment 2 by xixuan@chromium.org, May 12 2017

Blockedon: 715415
Mergedinto: 715415
Status: Duplicate (was: Untriaged)
winky "outage" caused by RPC downtime

Sign in to add a comment