New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 713825 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 714259
Owner:
Last visit > 30 days ago
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Tests got aborted because of connection refused

Project Member Reported by nxia@chromium.org, Apr 20 2017

Issue description

After pinning chrome to version 59.0.3064.0_rc-r1, the CQ still experienced aborted test but the symptom is different, it shows connection refused. 

https://chromium-review.googlesource.com/c/482707/



https://luci-milo.appspot.com/buildbot/chromeos/peppy-paladin/14920

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/113449682-chromeos-test/



04/20 00:55:37.883 INFO |          autoserv:0687| Results placed in /usr/local/autotest/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/20172004005531
04/20 00:55:37.883 DEBUG|          autoserv:0695| autoserv is running in drone chromeos-server14.mtv.corp.google.com.
04/20 00:55:37.884 DEBUG|          autoserv:0696| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/20172004005531 -m chromeos4-row6-rack12-host7 --verbose --lab True --provision --job-labels cros-version:peppy-paladin/R60-9478.0.0-rc2
04/20 00:55:37.884 INFO |           pidfile:0016| Logged pid 1188 to /usr/local/autotest/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/20172004005531/.autoserv_execute
04/20 00:55:37.887 DEBUG|          autoserv:0406| faulthandler registered on SIGTERM.
04/20 00:55:37.890 DEBUG|          base_job:0350| Persistent state global_properties.test_retry now set to 0
04/20 00:55:37.890 DEBUG|          base_job:0350| Persistent state global_properties.tag now set to ''
04/20 00:55:38.017 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:55:48.070 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:56:08.107 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:56:48.151 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 00:58:08.168 DEBUG|        retry_util:0136| <class 'urllib2.URLError'>(<urlopen error [Errno 111] Connection refused>)
04/20 01:00:38.010 ERROR|          autoserv:0769| Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
  File "/usr/local/autotest/server/autoserv", line 761, in main
    use_ssp)
  File "/usr/local/autotest/server/autoserv", line 494, in run_autoserv
    test_retry, **kwargs)
  File "/usr/local/autotest/server/site_server_job.py", line 48, in __init__
    super(site_server_job, self).__init__(*args, **dargs)
  File "/usr/local/autotest/server/server_job.py", line 331, in __init__
    self.machines, self.in_lab, host_attributes)
  File "/usr/local/autotest/server/server_job.py", line 102, in get_machine_dicts
    afe_host = _create_afe_host(machine, in_lab)
  File "/usr/local/autotest/server/server_job.py", line 1440, in _create_afe_host
    hosts = afe.get_hosts(hostname=hostname)
  File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
    hosts = self.run('get_hosts', **query_args)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
    self, call, **dargs)
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 114, in GenericRetry
    time.sleep(sleep_time)
  File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
    raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.


 

Comment 1 by nxia@chromium.org, Apr 20 2017

And this provision failure happened on all peppy dut around 0:55 am


nxia@nxia:~/chromiumos/src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome$ dut-status -b peppy -s '2017-04-20 00:00:01' -u '2017-04-20 01:10:00' -p cq -f
chromeos4-row9-rack11-host1
    2017-04-20 01:00:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host1/3159653-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host1/3159587-provision/
chromeos4-row6-rack12-host3
    2017-04-20 01:00:45  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host3/3159648-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host3/3159582-provision/
chromeos4-row6-rack12-host7
    2017-04-20 01:00:45  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host7/3159647-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack12-host7/3159581-provision/
chromeos4-row6-rack13-host1
    2017-04-20 01:00:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host1/3159656-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host1/3159590-provision/
chromeos4-row6-rack13-host15
    2017-04-20 01:00:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host15/3159650-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host15/3159584-provision/
chromeos4-row6-rack13-host19
    2017-04-20 01:00:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host19/3159655-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row6-rack13-host19/3159589-provision/
chromeos4-row9-rack11-host3
    2017-04-20 01:00:46  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host3/3159652-repair/
    2017-04-20 00:55:34  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host3/3159586-provision/
nxia@nxia:~/chromiumos/src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome$ 

Comment 2 by aut...@google.com, May 16 2017

Owner: nxia@chromium.org
@ nxia - what's needed next here? 

Comment 3 by nxia@chromium.org, May 16 2017

Mergedinto: 714259
Status: Duplicate (was: Untriaged)

Sign in to add a comment