New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 898505 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Closed: Oct 24
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

eve-paladin: Not enough DUTs for requirements, required: 4, found: 3

Project Member Reported by benzh@chromium.org, Oct 24

Issue description

Started to happen last evening:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=3066190
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=3066777
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=3067111
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931795758833977248


************************************************************
** Start Stage HWTest [provision] - Tue, 23 Oct 2018 18:06:37 -0700 (PDT)
** 
** Stage that runs tests in the Autotest lab.
************************************************************
18:06:37: INFO: Created cidb engine bot@130.211.191.11 for pid 18967
18:06:37: INFO: Running cidb query on pid 18967, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f47e373ea10>
18:06:37: INFO: Waiting up to forever for payloads and test artifacts ...
Preconditions for the stage successfully met. Beginning to execute stage...
18:15:44: INFO: Running cidb query on pid 18967, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f47e1feced0>
18:15:44: INFO: Re-run swarming_cmd to avoid buildbot salency check.
18:15:44: INFO: RunCommand: /b/swarming/w8gAeRL/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w8gAeRL/ir/tmp/t/cbuildbot-tmpDnfnow/tmpVbo8_R/temp_summary.json --print-status-updates --timeout 9000 --raw-cmd --task-name eve-paladin/R72-11186.0.0-rc2-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:provision' '--tags=build:eve-paladin/R72-11186.0.0-rc2' '--tags=task_name:eve-paladin/R72-11186.0.0-rc2-provision' '--tags=board:eve' -- /usr/local/autotest/site_utils/run_suite.py --build eve-paladin/R72-11186.0.0-rc2 --board eve --suite_name provision --pool cq --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 96813906L, 'cidb_build_id': 3066190, 'datastore_parent_key': ('Build', 3066190, 'BuildStage', 96813906L)}" --test_args "{'fast': 'True'}" -c
18:15:49: WARNING: Exception is not retriable return code: 3; command: /b/swarming/w8gAeRL/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w8gAeRL/ir/tmp/t/cbuildbot-tmpDnfnow/tmpVbo8_R/temp_summary.json --print-status-updates --timeout 9000 --raw-cmd --task-name eve-paladin/R72-11186.0.0-rc2-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:provision' '--tags=build:eve-paladin/R72-11186.0.0-rc2' '--tags=task_name:eve-paladin/R72-11186.0.0-rc2-provision' '--tags=board:eve' -- /usr/local/autotest/site_utils/run_suite.py --build eve-paladin/R72-11186.0.0-rc2 --board eve --suite_name provision --pool cq --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 96813906L, 'cidb_build_id': 3066190, 'datastore_parent_key': ('Build', 3066190, 'BuildStage', 96813906L)}" --test_args "{'fast': 'True'}" -c
Triggered task: eve-paladin/R72-11186.0.0-rc2-provision
chromeos-golo-server5-57: 40bc75d1f27c6d10 3
  Autotest instance created: cautotest-prod
  TestLabException: Not enough DUTs for requirements: ('board:eve', 'pool:cq'); required: 4, found: 2
  Traceback (most recent call last):
    File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
      return _run_suite(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite
      options.skip_duts_check)
    File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability
      hosts=hosts)
  NotEnoughDutsError: Not enough DUTs for requirements: ('board:eve', 'pool:cq'); required: 4, found: 2
  Will return from run_suite with status: INFRA_FAILURE
 
Mergedinto: 897885
Status: Duplicate (was: Untriaged)
 Issue 898504  has been merged into this issue.
Owner: ayatane@chromium.org
Status: Assigned (was: Duplicate)
Components: -Infra>Client>ChromeOS>CI Infra>Client>ChromeOS>Test
Labels: -Pri-3 Pri-1
Status: Duplicate (was: Assigned)
c#4 looks like a mass edit that incorrectly updated this issue's status.
Status: Assigned (was: Duplicate)
Oh oops, forget what I said. I'll go get coffee first.
Mergedinto: -897885 882562
Status: Duplicate (was: Assigned)
This was caused by the DUT locking issue.
peach_pit-chrome-pfq just failed due to not enough DUTs:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931764561629971072

12:44:04: INFO: RunCommand: /b/swarming/wDRjInV/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/wDRjInV/ir/tmp/t/cbuildbot-tmpRXv9hI/tmpUHhuNn/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name peach_pit-chrome-pfq/R72-11189.0.0-rc1-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:peach_pit-chrome-pfq/R72-11189.0.0-rc1' '--tags=task_name:peach_pit-chrome-pfq/R72-11189.0.0-rc1-provision' '--tags=board:peach_pit' -- /usr/local/autotest/site_utils/run_suite.py --build peach_pit-chrome-pfq/R72-11189.0.0-rc1 --board peach_pit --suite_name provision --pool bvt --file_bugs True --priority PFQ --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 4 --suite_min_duts 3 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 96909650L, 'cidb_build_id': 3068629, 'datastore_parent_key': ('Build', 3068629, 'BuildStage', 96909650L)}" -c
  12:44:08: WARNING: Exception is not retriable return code: 3; command: /b/swarming/wDRjInV/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/wDRjInV/ir/tmp/t/cbuildbot-tmpRXv9hI/tmpUHhuNn/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name peach_pit-chrome-pfq/R72-11189.0.0-rc1-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:peach_pit-chrome-pfq/R72-11189.0.0-rc1' '--tags=task_name:peach_pit-chrome-pfq/R72-11189.0.0-rc1-provision' '--tags=board:peach_pit' -- /usr/local/autotest/site_utils/run_suite.py --build peach_pit-chrome-pfq/R72-11189.0.0-rc1 --board peach_pit --suite_name provision --pool bvt --file_bugs True --priority PFQ --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 4 --suite_min_duts 3 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 96909650L, 'cidb_build_id': 3068629, 'datastore_parent_key': ('Build', 3068629, 'BuildStage', 96909650L)}" -c
  Triggered task: peach_pit-chrome-pfq/R72-11189.0.0-rc1-provision
  chromeos-golo-server5-217: 40c06c8713e9a310 3
    Autotest instance created: cautotest-prod
    TestLabException: Not enough DUTs for requirements: ('board:peach_pit', 'pool:bvt'); required: 4, found: 3
    Traceback (most recent call last):
      File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
        return _run_suite(options)
      File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite
        options.skip_duts_check)
      File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability
        hosts=hosts)
    NotEnoughDutsError: Not enough DUTs for requirements: ('board:peach_pit', 'pool:bvt'); required: 4, found: 3
    Will return from run_suite with status: INFRA_FAILURE

Sign in to add a comment