wrong queue.

Just occurred again, at least it found 3 out of 4 this time.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924121980180518432

T************************************************************
** Start Stage HWTest [provision] - Wed, 16 Jan 2019 20:49:06 -0800 (PST)
** 
** Stage that runs tests in the Autotest lab.
************************************************************
20:49:06: INFO: Created cidb engine bot@cosmic-strategy-646:cidb-gen2 for pid 13939
20:49:06: INFO: Running cidb query on pid 13939, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f3c57fd3410>
20:49:06: INFO: Waiting up to forever for payloads and test artifacts ...
Preconditions for the stage successfully met. Beginning to execute stage...
20:49:06: INFO: Running cidb query on pid 13939, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f3c567ef850>
20:49:06: INFO: Re-run swarming_cmd to avoid buildbot salency check.
20:49:06: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpj6JRcc/tmp9vgEqi/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name peach_pit-chrome-pfq/R73-11595.0.0-rc2-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:peach_pit-chrome-pfq/R73-11595.0.0-rc2' '--tags=task_name:peach_pit-chrome-pfq/R73-11595.0.0-rc2-provision' '--tags=board:peach_pit' -- /usr/local/autotest/site_utils/run_suite.py --build peach_pit-chrome-pfq/R73-11595.0.0-rc2 --board peach_pit --suite_name provision --pool bvt --file_bugs True --priority PFQ --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 4 --suite_min_duts 3 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 107551559L, 'cidb_build_id': 3366469, 'datastore_parent_key': ('Build', 3366469, 'BuildStage', 107551559L)}" -c
20:49:11: WARNING: Exception is not retriable return code: 3; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpj6JRcc/tmp9vgEqi/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name peach_pit-chrome-pfq/R73-11595.0.0-rc2-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:peach_pit-chrome-pfq/R73-11595.0.0-rc2' '--tags=task_name:peach_pit-chrome-pfq/R73-11595.0.0-rc2-provision' '--tags=board:peach_pit' -- /usr/local/autotest/site_utils/run_suite.py --build peach_pit-chrome-pfq/R73-11595.0.0-rc2 --board peach_pit --suite_name provision --pool bvt --file_bugs True --priority PFQ --timeout_mins 180 --retry True --max_retries 5 --minimum_duts 4 --suite_min_duts 3 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 107551559L, 'cidb_build_id': 3366469, 'datastore_parent_key': ('Build', 3366469, 'BuildStage', 107551559L)}" -c
Triggered task: peach_pit-chrome-pfq/R73-11595.0.0-rc2-provision
chromeos-golo-server5-170: 4272f5b65b373110 3
  Autotest instance created: cautotest-prod
  TestLabException: Not enough DUTs for requirements: ('board:peach_pit', 'pool:bvt'); required: 4, found: 3
  Traceback (most recent call last):
    File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
      return _run_suite(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite
      options.skip_duts_check)
    File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability
      hosts=hosts)
  NotEnoughDutsError: Not enough DUTs for requirements: ('board:peach_pit', 'pool:bvt'); required: 4, found: 3
  Will return from run_suite with status: INFRA_FAILURE

akeshet@ Could you please triage this issue? This is currently the only builder blocking the PFQ.

Believed fixed, I manually rebalanced this pool yesterday.

Yes, fixed. Thanks!