https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931994035364652192
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931976355617128208
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931976324893280640

There were intermittent HWTest provision failures in the last few CQ run on kevin-arcnext-paladin, auron_yuna-paladin, cyan-paladin. All these have the same error signature:
TimeoutException: retry exception (function="create_suite()"), timeout = 600s


************************************************************
22:39:54: INFO: Created cidb engine bot@130.211.191.11 for pid 23843
22:39:54: INFO: Running cidb query on pid 23843, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f3bb6b2c8d0>
22:39:54: INFO: Waiting up to forever for payloads and test artifacts ...
Preconditions for the stage successfully met. Beginning to execute stage...
22:48:26: INFO: Running cidb query on pid 23843, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f3bb5fcbe10>
22:48:26: INFO: Re-run swarming_cmd to avoid buildbot salency check.
22:48:26: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpXxcQWY/tmp0gWeOb/temp_summary.json --print-status-updates --timeout 9000 --raw-cmd --task-name kevin-arcnext-paladin/R72-11181.0.0-rc1-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:provision' '--tags=build:kevin-arcnext-paladin/R72-11181.0.0-rc1' '--tags=task_name:kevin-arcnext-paladin/R72-11181.0.0-rc1-provision' '--tags=board:kevin-arcnext' -- /usr/local/autotest/site_utils/run_suite.py --build kevin-arcnext-paladin/R72-11181.0.0-rc1 --board kevin --suite_name provision --pool cq --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 96585954L, 'cidb_build_id': 3059807, 'datastore_parent_key': ('Build', 3059807, 'BuildStage', 96585954L)}" --test_args "{'fast': 'True'}" -c
22:58:42: WARNING: Exception is not retriable return code: 1; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /b/swarming/w/ir/tmp/t/cbuildbot-tmpXxcQWY/tmp0gWeOb/temp_summary.json --print-status-updates --timeout 9000 --raw-cmd --task-name kevin-arcnext-paladin/R72-11181.0.0-rc1-provision --dimension os Ubuntu-14.04 --dimension pool default --io-timeout 9000 --hard-timeout 9000 --expiration 1200 '--tags=priority:CQ' '--tags=suite:provision' '--tags=build:kevin-arcnext-paladin/R72-11181.0.0-rc1' '--tags=task_name:kevin-arcnext-paladin/R72-11181.0.0-rc1-provision' '--tags=board:kevin-arcnext' -- /usr/local/autotest/site_utils/run_suite.py --build kevin-arcnext-paladin/R72-11181.0.0-rc1 --board kevin --suite_name provision --pool cq --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 5 --minimum_duts 4 --suite_args "{u'num_required': 1}" --offload_failures_only False --job_keyvals "{'cidb_build_stage_id': 96585954L, 'cidb_build_id': 3059807, 'datastore_parent_key': ('Build', 3059807, 'BuildStage', 96585954L)}" --test_args "{'fast': 'True'}" -c
Triggered task: kevin-arcnext-paladin/R72-11181.0.0-rc1-provision
chromeos-golo-server2-22: 40b322c4a2c30610 1
  Autotest instance created: cautotest-prod
  10-21-2018 [22:48:31] Submitted create_suite_job rpc
  Traceback (most recent call last):
    File "/usr/local/autotest/site_utils/run_suite.py", line 2076, in <module>
      sys.exit(main())
    File "/usr/local/autotest/site_utils/run_suite.py", line 2065, in main
      result = _run_task(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
      return _run_suite(options)
    File "/usr/local/autotest/site_utils/run_suite.py", line 1739, in _run_suite
      job_id = create_suite(afe, options)
    File "/usr/local/autotest/client/common_lib/cros/retry.py", line 246, in func_retry
      raise error.TimeoutException(exception_message)
  autotest_lib.client.common_lib.error.TimeoutException: retry exception (function="create_suite()"), timeout = 600s


cmd=['/b/swarming/w/ir/cache/cbuild/repository/chromite/third_party/swarming.client/swarming.py', 'run', '--swarming', 'chromeos-proxy.appspot.com', '--task-summary-json', '/b/swarming/w/ir/tmp/t/cbuildbot-tmpXxcQWY/tmp0gWeOb/temp_summary.json', '--print-status-updates', '--timeout', '9000', '--raw-cmd', '--task-name', u'kevin-arcnext-paladin/R72-11181.0.0-rc1-provision', '--dimension', 'os', 'Ubuntu-14.04', '--dimension', 'pool', 'default', '--io-timeout', '9000', '--hard-timeout', '9000', '--expiration', '1200', u'--tags=priority:CQ', u'--tags=suite:provision', u'--tags=build:kevin-arcnext-paladin/R72-11181.0.0-rc1', u'--tags=task_name:kevin-arcnext-paladin/R72-11181.0.0-rc1-provision', u'--tags=board:kevin-arcnext', '--', '/usr/local/autotest/site_utils/run_suite.py', '--build', u'kevin-arcnext-paladin/R72-11181.0.0-rc1', '--board', u'kevin', '--suite_name', u'provision', '--pool', u'cq', '--file_bugs', 'False', '--priority', 'CQ', '--timeout_mins', '90', '--retry', 'True', '--max_retries', '5', '--minimum_duts', '4', '--suite_args', "{u'num_required': 1}", '--offload_failures_only', 'False', '--job_keyvals', "{'cidb_build_stage_id': 96585954L, 'cidb_build_id': 3059807, 'datastore_parent_key': ('Build', 3059807, 'BuildStage', 96585954L)}", '--test_args', "{'fast': 'True'}", '-c']
Autotest instance created: cautotest-prod
10-21-2018 [22:48:31] Submitted create_suite_job rpc
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/run_suite.py", line 2076, in <module>
    sys.exit(main())
  File "/usr/local/autotest/site_utils/run_suite.py", line 2065, in main
    result = _run_task(options)
  File "/usr/local/autotest/site_utils/run_suite.py", line 2000, in _run_task
    return _run_suite(options)
  File "/usr/local/autotest/site_utils/run_suite.py", line 1739, in _run_suite
    job_id = create_suite(afe, options)
  File "/usr/local/autotest/client/common_lib/cros/retry.py", line 246, in func_retry
    raise error.TimeoutException(exception_message)
autotest_lib.client.common_lib.error.TimeoutException: retry exception (function="create_suite()"), timeout = 600s
22:58:42: INFO: No json dump found, no HWTest results to report
22:58:42: INFO: Running cidb query on pid 23843, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f3bb5fcc410>

22:58:42: ERROR: ** HWTest failed (code 1) **
22:58:42: INFO: Translating result ** HWTest failed (code 1) ** to fail.
22:58:42: INFO: Running cidb query on pid 23843, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f3bb5fcc510>
22:58:42: INFO: Running cidb query on pid 23843, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f3bb5fcc950>
************************************************************
** Finished Stage HWTest [provision] - Sun, 21 Oct 2018 22:58:42 -0700 (PDT)
************************************************************