After pinning chrome to version 59.0.3064.0_rc-r1, the CQ still experienced aborted test but the symptom is different, it shows connection refused.
https://chromium-review.googlesource.com/c/482707/
https://luci-milo.appspot.com/buildbot/chromeos/peppy-paladin/14920
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/113449682-chromeos-test/
04/20 00:55:37.883 INFO | autoserv:0687| Results placed in /usr/local/autotest/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/20172004005531
04/20 00:55:37.883 DEBUG| autoserv:0695| autoserv is running in drone chromeos-server14.mtv.corp.google.com.
04/20 00:55:37.884 DEBUG| autoserv:0696| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/20172004005531 -m chromeos4-row6-rack12-host7 --verbose --lab True --provision --job-labels cros-version:peppy-paladin/R60-9478.0.0-rc2
04/20 00:55:37.884 INFO | pidfile:0016| Logged pid 1188 to /usr/local/autotest/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/20172004005531/.autoserv_execute
04/20 00:55:37.887 DEBUG| autoserv:0406| faulthandler registered on SIGTERM.
04/20 00:55:37.890 DEBUG| base_job:0350| Persistent state global_properties.test_retry now set to 0
04/20 00:55:37.890 DEBUG| base_job:0350| Persistent state global_properties.tag now set to ''
04/20 00:55:38.017 DEBUG| retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:55:48.070 DEBUG| retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:56:08.107 DEBUG| retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:56:48.151 DEBUG| retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:58:08.168 DEBUG| retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 01:00:38.010 ERROR| autoserv:0769| Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
File "/usr/local/autotest/server/autoserv", line 761, in main
use_ssp)
File "/usr/local/autotest/server/autoserv", line 494, in run_autoserv
test_retry, **kwargs)
File "/usr/local/autotest/server/site_server_job.py", line 48, in __init__
super(site_server_job, self).__init__(*args, **dargs)
File "/usr/local/autotest/server/server_job.py", line 331, in __init__
self.machines, self.in_lab, host_attributes)
File "/usr/local/autotest/server/server_job.py", line 102, in get_machine_dicts
afe_host = _create_afe_host(machine, in_lab)
File "/usr/local/autotest/server/server_job.py", line 1440, in _create_afe_host
hosts = afe.get_hosts(hostname=hostname)
File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
hosts = self.run('get_hosts', **query_args)
File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
self, call, **dargs)
File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 114, in GenericRetry
time.sleep(sleep_time)
File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.
Comment 1 by nxia@chromium.org
, Apr 20 2017And this provision failure happened on all peppy dut around 0:55 am nxia@nxia:~/chromiumos/src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome$ dut-status -b peppy -s '2017-04-20 00:00:01' -u '2017-04-20 01:10:00' -p cq -f chromeos4-row9-rack11-host1 2017-04-20 01:00:46 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host1/3159653-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host1/3159587-provision/ chromeos4-row6-rack12-host3 2017-04-20 01:00:45 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host3/3159648-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host3/3159582-provision/ chromeos4-row6-rack12-host7 2017-04-20 01:00:45 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host7/3159647-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/ chromeos4-row6-rack13-host1 2017-04-20 01:00:46 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host1/3159656-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host1/3159590-provision/ chromeos4-row6-rack13-host15 2017-04-20 01:00:46 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host15/3159650-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host15/3159584-provision/ chromeos4-row6-rack13-host19 2017-04-20 01:00:46 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host19/3159655-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host19/3159589-provision/ chromeos4-row9-rack11-host3 2017-04-20 01:00:46 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host3/3159652-repair/ 2017-04-20 00:55:34 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host3/3159586-provision/ nxia@nxia:~/chromiumos/src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome$