New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 892189 link

Starred by 0 users

Issue metadata

Status: Duplicate
Owner:
OOO
Closed: Oct 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

TestLabFailure for caroline-tot-chrome-pfq-informational.

Project Member Reported by sammiequon@chromium.org, Oct 4

Issue description

06:44:30: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmp7qf5ux/tmpd_Hq4y/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657' '--tags=task_name:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision' '--tags=board:caroline' -- /usr/local/autotest/site_utils/run_suite.py --build caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657 --board caroline --suite_name provision --pool continuous --file_bugs True --priority PFQ --timeout_mins 180 --retry False --minimum_duts 1 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 94668969L, 'cidb_build_id': 3005657, 'datastore_parent_key': ('Build', 3005657, 'BuildStage', 94668969L)}" -c
06:44:36: WARNING: Exception is not retriable return code: 3; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmp7qf5ux/tmpd_Hq4y/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657' '--tags=task_name:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision' '--tags=board:caroline' -- /usr/local/autotest/site_utils/run_suite.py --build caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657 --board caroline --suite_name provision --pool continuous --file_bugs True --priority PFQ --timeout_mins 180 --retry False --minimum_duts 1 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 94668969L, 'cidb_build_id': 3005657, 'datastore_parent_key': ('Build', 3005657, 'BuildStage', 94668969L)}" -c
Triggered task: caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision
chromeos-golo-server2-205: 405824287d6d2310 3
  Autotest instance created: cautotest-prod
  TestLabException: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0
  Traceback (most recent call last):
    File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
      return _run_suite(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite
      options.skip_duts_check)
    File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability
      hosts=hosts)
  NotEnoughDutsError: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0
  Will return from run_suite with status: INFRA_FAILURE
cmd=['/b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py', 'run', '--swarming', 'chromeos-proxy.appspot.com', '--task-summary-json', '/b/swarming/w/ir/tmp/t/cbuildbot-tmp7qf5ux/tmpd_Hq4y/temp_summary.json', '--print-status-updates', '--timeout', '14400', '--raw-cmd', '--task-name', u'caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision', '--dimension', 'os', 'Ubuntu-14.04', '--dimension', 'pool', 'default', '--io-timeout', '14400', '--hard-timeout', '14400', '--expiration', '1200', u'--tags=priority:PFQ', u'--tags=suite:provision', u'--tags=build:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657', u'--tags=task_name:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision', u'--tags=board:caroline', '--', '/usr/local/autotest/site_utils/run_suite.py', '--build', u'caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657', '--board', u'caroline', '--suite_name', u'provision', '--pool', u'continuous', '--file_bugs', 'True', '--priority', 'PFQ', '--timeout_mins', '180', '--retry', 'False', '--minimum_duts', '1', '--suite_args', "{u'num_required': 1}", '--offload_failures_only', 'False', '--job_keyvals', "{'cidb_build_stage_id': 94668969L, 'cidb_build_id': 3005657, 'datastore_parent_key': ('Build', 3005657, 'BuildStage', 94668969L)}", '-c']
Autotest instance created: cautotest-prod
TestLabException: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
    return _run_suite(options)
  File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite
    options.skip_duts_check)
  File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability
    hosts=hosts)
NotEnoughDutsError: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0
Will return from run_suite with status: INFRA_FAILURE
06:44:36: INFO: Running cidb query on pid 12581, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7fc4bce29b10>
06:44:36: ERROR: ** HWTest did not complete due to infrastructure issues (code 3) **
06:44:36: INFO: Translating result ** HWTest did not complete due to infrastructure issues (code 3) ** to fail.
06:44:36: INFO: Running cidb query on pid 12581, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7fc4bcddc3d0>
06:44:36: INFO: Running cidb query on pid 12581, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7fc4bcdcca50>
************************************************************
** Finished Stage HWTest [provision] - Thu, 04 Oct 2018 06:44:36 -0700 (PDT)
************************************************************
06:44:36: ERROR: BaseException in _RunParallelStages <class 'chromite.lib.failures_lib.TestLabFailure'>: ** HWTest did not complete due to infrastructure issues (code 3) **
Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 702, in Run
    self.PerformStage()
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/test_stages.py", line 280, in PerformStage
    raise cmd_result.to_raise
TestLabFailure: ** HWTest did not complete due to infrastructure issues (code 3) **
Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 120, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 678, in RunParallelSteps
    return [queue.get_nowait() for queue in queues]
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 675, in RunParallelSteps
    pass
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 561, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
BackgroundFailure: <class 'chromite.lib.failures_lib.TestLabFailure'>: ** HWTest did not complete due to infrastructure issues (code 3) **
Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 702, in Run
    self.PerformStage()
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/test_stages.py", line 280, in PerformStage
    raise cmd_result.to_raise
TestLabFailure: ** HWTest did not complete due to infrastructure issues (code 3) **
 
Owner: gu...@chromium.org
+guocb

guocb, is this something you can look at?
We have to update labstation for some reason. This requires lock DUTs to give labstation a chance to reboot.

$ atest host list -b board:caroline,pool:continuous -s Ready
Host                          Status  Shard                               Locked  Lock Reason                                              Locked by      Platform  Labels
chromeos6-row2-rack21-host9   Ready   cros-full-0005.mtv.corp.google.com  True    reboot for servo host chromeos6-row2-rack21-labstation1  chromeos-test  caroline  board:caroline, ...

See issue 887020
Status: Started (was: Untriaged)
Mergedinto: 891956
Status: Duplicate (was: Started)

Sign in to add a comment