New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 816563 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

ultima-release: NotEnoughDutsError

Project Member Reported by ejcaruso@chromium.org, Feb 26 2018

Issue description

From https://luci-milo.appspot.com/buildbot/chromeos/ultima-release/1984:

04:40:07: WARNING: Exception is not retriable return code: 3; command: /b/c/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpSNYcVq/tmpHNS82N/temp_summary.json --raw-cmd --task-name ultima-release/R66-10440.0.0-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:bvt-inline' '--tags=build:ultima-release/R66-10440.0.0' '--tags=task_name:ultima-release/R66-10440.0.0-bvt-inline' '--tags=board:ultima' -- /usr/local/autotest/site_utils/run_suite.py --build ultima-release/R66-10440.0.0 --board ultima --suite_name bvt-inline --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 4 --suite_min_duts 6 --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 71750595L, 'cidb_build_id': 2331675, 'datastore_parent_key': ('Build', 2331675, 'BuildStage', 71750595L)}" -c
Priority was reset to 100
Triggered task: ultima-release/R66-10440.0.0-bvt-inline
chromeos-golo-server2-99: 3beaf211dd99d310 3
  Autotest instance created: cautotest-prod
  TestLabException: Not enough DUTs for board: ultima, pool: bvt; required: 4, found: 3
  Traceback (most recent call last):
    File "/usr/local/autotest/site_utils/run_suite.py", line 2034, in _run_task
      return _run_suite(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 1775, in _run_suite
      options.skip_duts_check)
    File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 330, in check_dut_availability
      hosts=hosts)
  NotEnoughDutsError: Not enough DUTs for board: ultima, pool: bvt; required: 4, found: 3
  Will return from run_suite with status: INFRA_FAILURE
cmd=['/b/c/cbuild/repository/chromite/third_party/swarming.client/swarming.py', 'run', '--swarming', 'chromeos-proxy.appspot.com', '--task-summary-json', '/tmp/cbuildbot-tmpSNYcVq/tmpHNS82N/temp_summary.json', '--raw-cmd', '--task-name', u'ultima-release/R66-10440.0.0-bvt-inline', '--dimension', 'os', 'Ubuntu-14.04', '--dimension', 'pool', 'default', '--print-status-updates', '--timeout', '14400', '--io-timeout', '14400', '--hard-timeout', '14400', '--expiration', '1200', u'--tags=priority:Build', u'--tags=suite:bvt-inline', u'--tags=build:ultima-release/R66-10440.0.0', u'--tags=task_name:ultima-release/R66-10440.0.0-bvt-inline', u'--tags=board:ultima', '--', '/usr/local/autotest/site_utils/run_suite.py', '--build', u'ultima-release/R66-10440.0.0', '--board', u'ultima', '--suite_name', u'bvt-inline', '--pool', u'bvt', '--file_bugs', 'True', '--priority', 'Build', '--timeout_mins', '180', '--retry', 'True', '--max_retries', '5', '--minimum_duts', '4', '--suite_min_duts', '6', '--offload_failures_only', 'False', '--job_keyvals', "{'cidb_build_stage_id': 71750595L, 'cidb_build_id': 2331675, 'datastore_parent_key': ('Build', 2331675, 'BuildStage', 71750595L)}", '-c']
Autotest instance created: cautotest-prod
TestLabException: Not enough DUTs for board: ultima, pool: bvt; required: 4, found: 3
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/run_suite.py", line 2034, in _run_task
    return _run_suite(options)
  File "/usr/local/autotest/site_utils/run_suite.py", line 1775, in _run_suite
    options.skip_duts_check)
  File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 330, in check_dut_availability
    hosts=hosts)
NotEnoughDutsError: Not enough DUTs for board: ultima, pool: bvt; required: 4, found: 3
Will return from run_suite with status: INFRA_FAILURE
04:40:07: INFO: Running cidb query on pid 23642, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f59894b3f50>
04:40:07: ERROR: ** HWTest did not complete due to infrastructure issues (code 3) **
04:40:07: INFO: Translating result ** HWTest did not complete due to infrastructure issues (code 3) ** to fail.

Assigning to deputy.
 

Comment 1 by nxia@chromium.org, Feb 26 2018

5 duts were triggered to run verify jobs around 04:39:35 so couldn't serve the hwtests. but we have enough good duts in the bvt pool. will leave this bug open for now and close it when the next run approves there're enough duts in the pool

Status: Fixed (was: Assigned)
Looks like run https://luci-milo.appspot.com/buildbot/chromeos/ultima-release/1985 had enough DUTs but is encountering a different issue, so I'll close this and open a new bug for that.

Sign in to add a comment