TestLabFailure for caroline-tot-chrome-pfq-informational. |
|||
Issue description06:44:30: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmp7qf5ux/tmpd_Hq4y/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657' '--tags=task_name:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision' '--tags=board:caroline' -- /usr/local/autotest/site_utils/run_suite.py --build caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657 --board caroline --suite_name provision --pool continuous --file_bugs True --priority PFQ --timeout_mins 180 --retry False --minimum_duts 1 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 94668969L, 'cidb_build_id': 3005657, 'datastore_parent_key': ('Build', 3005657, 'BuildStage', 94668969L)}" -c [1;33m06:44:36: WARNING: Exception is not retriable return code: 3; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmp7qf5ux/tmpd_Hq4y/temp_summary.json --print-status-updates --timeout 14400 --raw-cmd --task-name caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:provision' '--tags=build:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657' '--tags=task_name:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision' '--tags=board:caroline' -- /usr/local/autotest/site_utils/run_suite.py --build caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657 --board caroline --suite_name provision --pool continuous --file_bugs True --priority PFQ --timeout_mins 180 --retry False --minimum_duts 1 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 94668969L, 'cidb_build_id': 3005657, 'datastore_parent_key': ('Build', 3005657, 'BuildStage', 94668969L)}" -c Triggered task: caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision chromeos-golo-server2-205: 405824287d6d2310 3 Autotest instance created: cautotest-prod TestLabException: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0 Traceback (most recent call last): File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task return _run_suite(options) File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite options.skip_duts_check) File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability hosts=hosts) NotEnoughDutsError: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0 Will return from run_suite with status: INFRA_FAILURE cmd=['/b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py', 'run', '--swarming', 'chromeos-proxy.appspot.com', '--task-summary-json', '/b/swarming/w/ir/tmp/t/cbuildbot-tmp7qf5ux/tmpd_Hq4y/temp_summary.json', '--print-status-updates', '--timeout', '14400', '--raw-cmd', '--task-name', u'caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision', '--dimension', 'os', 'Ubuntu-14.04', '--dimension', 'pool', 'default', '--io-timeout', '14400', '--hard-timeout', '14400', '--expiration', '1200', u'--tags=priority:PFQ', u'--tags=suite:provision', u'--tags=build:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657', u'--tags=task_name:caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657-provision', u'--tags=board:caroline', '--', '/usr/local/autotest/site_utils/run_suite.py', '--build', u'caroline-tot-chrome-pfq-informational/R71-11126.0.0-b3005657', '--board', u'caroline', '--suite_name', u'provision', '--pool', u'continuous', '--file_bugs', 'True', '--priority', 'PFQ', '--timeout_mins', '180', '--retry', 'False', '--minimum_duts', '1', '--suite_args', "{u'num_required': 1}", '--offload_failures_only', 'False', '--job_keyvals', "{'cidb_build_stage_id': 94668969L, 'cidb_build_id': 3005657, 'datastore_parent_key': ('Build', 3005657, 'BuildStage', 94668969L)}", '-c'][0m Autotest instance created: cautotest-prod TestLabException: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0 Traceback (most recent call last): File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task return _run_suite(options) File "/usr/local/autotest/site_utils/run_suite.py", line 1738, in _run_suite options.skip_duts_check) File "/usr/local/autotest/site_utils/diagnosis_utils.py", line 320, in check_dut_availability hosts=hosts) NotEnoughDutsError: Not enough DUTs for requirements: ('board:caroline', 'pool:continuous'); required: 1, found: 0 Will return from run_suite with status: INFRA_FAILURE 06:44:36: INFO: Running cidb query on pid 12581, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7fc4bce29b10> [1;31m06:44:36: ERROR: ** HWTest did not complete due to infrastructure issues (code 3) **[0m 06:44:36: INFO: Translating result ** HWTest did not complete due to infrastructure issues (code 3) ** to fail. 06:44:36: INFO: Running cidb query on pid 12581, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7fc4bcddc3d0> 06:44:36: INFO: Running cidb query on pid 12581, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7fc4bcdcca50> ************************************************************ ** Finished Stage HWTest [provision] - Thu, 04 Oct 2018 06:44:36 -0700 (PDT) ************************************************************ [1;31m06:44:36: ERROR: BaseException in _RunParallelStages <class 'chromite.lib.failures_lib.TestLabFailure'>: ** HWTest did not complete due to infrastructure issues (code 3) ** Traceback (most recent call last): File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run self._task(*self._task_args, **self._task_kwargs) File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 702, in Run self.PerformStage() File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/test_stages.py", line 280, in PerformStage raise cmd_result.to_raise TestLabFailure: ** HWTest did not complete due to infrastructure issues (code 3) ** Traceback (most recent call last): File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 120, in _RunParallelStages parallel.RunParallelSteps(steps) File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 678, in RunParallelSteps return [queue.get_nowait() for queue in queues] File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 675, in RunParallelSteps pass File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 561, in ParallelTasks raise BackgroundFailure(exc_infos=errors) BackgroundFailure: <class 'chromite.lib.failures_lib.TestLabFailure'>: ** HWTest did not complete due to infrastructure issues (code 3) ** Traceback (most recent call last): File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run self._task(*self._task_args, **self._task_kwargs) File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 702, in Run self.PerformStage() File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/test_stages.py", line 280, in PerformStage raise cmd_result.to_raise TestLabFailure: ** HWTest did not complete due to infrastructure issues (code 3) **
,
Oct 4
Similar in cq pool: https://bugs.chromium.org/p/chromium/issues/detail?id=891956
,
Oct 4
We have to update labstation for some reason. This requires lock DUTs to give labstation a chance to reboot. $ atest host list -b board:caroline,pool:continuous -s Ready Host Status Shard Locked Lock Reason Locked by Platform Labels chromeos6-row2-rack21-host9 Ready cros-full-0005.mtv.corp.google.com True reboot for servo host chromeos6-row2-rack21-labstation1 chromeos-test caroline board:caroline, ... See issue 887020
,
Oct 4
,
Oct 4
|
|||
►
Sign in to add a comment |
|||
Comment 1 by sammiequon@chromium.org
, Oct 4