Another connection timeout example:

http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=113650262


http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=113650262

04/21 04:08:37.043 DEBUG|        dev_server:0710| The host chromeos2-row1-rack8-host1 (100.115.224.78) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.224.0:19.
04/21 04:08:37.047 DEBUG|        base_utils:0185| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/check_health?"''
04/21 04:08:38.566 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/21 04:08:48.600 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/21 04:09:08.701 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/21 04:09:48.758 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/21 04:11:08.857 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/21 04:13:38.565 DEBUG|              test:0389| Test failed due to Timeout occurred- waited 300 seconds.. Exception log follows the after_iteration_hooks.
04/21 04:13:38.565 DEBUG|              test:0392| starting after_iteration_hooks
04/21 04:13:38.566 DEBUG|              test:0395| after_iteration_hooks completed
04/21 04:13:38.568 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 823, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled TimeoutError: Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once
    force_full_update=force)
  File "/usr/local/autotest/server/afe_utils.py", line 210, in machine_install_and_update_labels
    update_host_attribute(host, attribute, value)
  File "/usr/local/autotest/server/afe_utils.py", line 186, in update_host_attribute
    AFE.set_host_attribute(attribute, value, hostname=host.hostname)
  File "/usr/local/autotest/server/frontend.py", line 570, in set_host_attribute
    self.run('set_host_attribute', attribute=attr, value=val, **dargs)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
    self, call, **dargs)
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 114, in GenericRetry
    time.sleep(sleep_time)
  File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
    raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.

https://luci-milo.appspot.com/buildbot/chromeos/peach_pit-paladin/15502

https://luci-milo.appspot.com/buildbot/chromeos/peach_pit-paladin/15502


04/20 00:56:45.233 INFO |          autoserv:0687| Results placed in /usr/local/autotest/results/hosts/chromeos6-row2-rack11-host17/3090518-provision/20172004005635
04/20 00:56:45.234 DEBUG|          autoserv:0695| autoserv is running in drone chromeos-server10.hot.corp.google.com.
04/20 00:56:45.234 DEBUG|          autoserv:0696| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/hosts/chromeos6-row2-rack11-host17/3090518-provision/20172004005635 -m chromeos6-row2-rack11-host17 --verbose --lab True --provision --job-labels cros-version:peach_pit-paladin/R60-9478.0.0-rc2
04/20 00:56:45.234 INFO |           pidfile:0016| Logged pid 25428 to /usr/local/autotest/results/hosts/chromeos6-row2-rack11-host17/3090518-provision/20172004005635/.autoserv_execute
04/20 00:56:45.235 DEBUG|          autoserv:0406| faulthandler registered on SIGTERM.
04/20 00:56:45.237 DEBUG|          base_job:0350| Persistent state global_properties.test_retry now set to 0
04/20 00:56:45.237 DEBUG|          base_job:0350| Persistent state global_properties.tag now set to ''
04/20 00:56:45.294 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:56:55.345 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:57:15.447 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:57:55.489 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:59:15.591 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 01:01:45.290 ERROR|          autoserv:0769| Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
  File "/usr/local/autotest/server/autoserv", line 761, in main
    use_ssp)
  File "/usr/local/autotest/server/autoserv", line 494, in run_autoserv
    test_retry, **kwargs)
  File "/usr/local/autotest/server/site_server_job.py", line 48, in __init__
    super(site_server_job, self).__init__(*args, **dargs)
  File "/usr/local/autotest/server/server_job.py", line 331, in __init__
    self.machines, self.in_lab, host_attributes)
  File "/usr/local/autotest/server/server_job.py", line 102, in get_machine_dicts
    afe_host = _create_afe_host(machine, in_lab)
  File "/usr/local/autotest/server/server_job.py", line 1440, in _create_afe_host
    hosts = afe.get_hosts(hostname=hostname)
  File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
    hosts = self.run('get_hosts', **query_args)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
    self, call, **dargs)
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 114, in GenericRetry
    time.sleep(sleep_time)
  File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
    raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.