New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 700450 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner: ----
Closed: Mar 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

peppy-paladin failed: TimeoutError at getting HostInfo

Project Member Reported by nxia@chromium.org, Mar 10 2017

Issue description

peppy-paladin failed because of the below error. I see AFE mysql experienced high CPU usage during the time, is the mysql slowness causing the error?

https://uberchromegw.corp.google.com/i/chromeos/builders/peppy-paladin/builds/14496/steps/HWTest%20%5Bbvt-cq%5D/logs/stdio



http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=105779209

03/10 00:04:13.966 INFO |          autoserv:0687| Results placed in /usr/local/autotest/results/105779209-chromeos-test/chromeos4-row9-rack11-host1
03/10 00:04:13.966 DEBUG|          autoserv:0695| autoserv is running in drone chromeos-server14.mtv.corp.google.com.
03/10 00:04:13.967 DEBUG|          autoserv:0696| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/105779209-chromeos-test/chromeos4-row9-rack11-host1 -m chromeos4-row9-rack11-host1 -u chromeos-test -l peppy-paladin/R59-9354.0.0-rc1/bvt-cq/platform_PartitionCheck -c --lab True -P 105779209-chromeos-test/chromeos4-row9-rack11-host1 -n /usr/local/autotest/results/drone_tmp/attach.52078 --verify_job_repo_url
03/10 00:04:13.967 INFO |           pidfile:0016| Logged pid 8009 to /usr/local/autotest/results/105779209-chromeos-test/chromeos4-row9-rack11-host1/.autoserv_execute
03/10 00:04:13.975 DEBUG|          autoserv:0406| faulthandler registered on SIGTERM.
03/10 00:04:13.978 DEBUG|          base_job:0350| Persistent state global_properties.test_retry now set to 0
03/10 00:04:13.978 DEBUG|          base_job:0350| Persistent state global_properties.tag now set to '105779209-chromeos-test/chromeos4-row9-rack11-host1'
03/10 00:04:14.301 DEBUG|         host_info:0193| Refreshing HostInfo using store <autotest_lib.server.hosts.afe_store.AfeStore object at 0x7f85053206d0>
03/10 00:04:14.302 DEBUG|         host_info:0194| Old host_info: None
03/10 00:09:14.304 DEBUG|        retry_util:0129| ending retries with error: <class 'chromite.lib.timeout_util.TimeoutError'>(Timeout occurred- waited 300 seconds.)
03/10 00:09:14.305 ERROR|          autoserv:0769| Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
  File "/usr/local/autotest/server/autoserv", line 761, in main
    use_ssp)
  File "/usr/local/autotest/server/autoserv", line 494, in run_autoserv
    test_retry, **kwargs)
  File "/usr/local/autotest/server/site_server_job.py", line 48, in __init__
    super(site_server_job, self).__init__(*args, **dargs)
  File "/usr/local/autotest/server/server_job.py", line 331, in __init__
    self.machines, self.in_lab, host_attributes)
  File "/usr/local/autotest/server/server_job.py", line 107, in get_machine_dicts
    'host_info_store': _create_host_info_store(machine, in_lab),
  File "/usr/local/autotest/server/server_job.py", line 1459, in _create_host_info_store
    host_info_store.get(force_refresh=True)
  File "/usr/local/autotest/server/hosts/host_info.py", line 142, in get
    return self._get_uncached()
  File "/usr/local/autotest/server/hosts/host_info.py", line 196, in _get_uncached
    info = self._refresh_impl()
  File "/usr/local/autotest/server/hosts/afe_store.py", line 37, in _refresh_impl
    hosts = self._afe.get_hosts(hostname=self._hostname)
  File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
    hosts = self.run('get_hosts', **query_args)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
    self, call, **dargs)
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 81, in _run
    return super(RetryingAFE, self).run(call, **dargs)
  File "/usr/local/autotest/server/frontend.py", line 107, in run
    result = utils.strip_unicode(rpc_call(**dargs))
  File "/usr/local/autotest/frontend/afe/json_rpc/proxy.py", line 114, in __call__
    respdata = urllib2.urlopen(request).read()
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
    raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.
 

Comment 1 by xixuan@chromium.org, Mar 10 2017

Mergedinto: 695287
Status: Duplicate (was: Untriaged)
It's a duplicate of 695287 I think.

Sign in to add a comment