peppy-paladin failed because of the below error. I see AFE mysql experienced high CPU usage during the time, is the mysql slowness causing the error?
https://uberchromegw.corp.google.com/i/chromeos/builders/peppy-paladin/builds/14496/steps/HWTest%20%5Bbvt-cq%5D/logs/stdio
http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=105779209
03/10 00:04:13.966 INFO | autoserv:0687| Results placed in /usr/local/autotest/results/105779209-chromeos-test/chromeos4-row9-rack11-host1
03/10 00:04:13.966 DEBUG| autoserv:0695| autoserv is running in drone chromeos-server14.mtv.corp.google.com.
03/10 00:04:13.967 DEBUG| autoserv:0696| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/105779209-chromeos-test/chromeos4-row9-rack11-host1 -m chromeos4-row9-rack11-host1 -u chromeos-test -l peppy-paladin/R59-9354.0.0-rc1/bvt-cq/platform_PartitionCheck -c --lab True -P 105779209-chromeos-test/chromeos4-row9-rack11-host1 -n /usr/local/autotest/results/drone_tmp/attach.52078 --verify_job_repo_url
03/10 00:04:13.967 INFO | pidfile:0016| Logged pid 8009 to /usr/local/autotest/results/105779209-chromeos-test/chromeos4-row9-rack11-host1/.autoserv_execute
03/10 00:04:13.975 DEBUG| autoserv:0406| faulthandler registered on SIGTERM.
03/10 00:04:13.978 DEBUG| base_job:0350| Persistent state global_properties.test_retry now set to 0
03/10 00:04:13.978 DEBUG| base_job:0350| Persistent state global_properties.tag now set to '105779209-chromeos-test/chromeos4-row9-rack11-host1'
03/10 00:04:14.301 DEBUG| host_info:0193| Refreshing HostInfo using store <autotest_lib.server.hosts.afe_store.AfeStore object at 0x7f85053206d0>
03/10 00:04:14.302 DEBUG| host_info:0194| Old host_info: None
03/10 00:09:14.304 DEBUG| retry_util:0129| ending retries with error: <class 'chromite.lib.timeout_util.TimeoutError'>(Timeout occurred- waited 300 seconds.)
03/10 00:09:14.305 ERROR| autoserv:0769| Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
File "/usr/local/autotest/server/autoserv", line 761, in main
use_ssp)
File "/usr/local/autotest/server/autoserv", line 494, in run_autoserv
test_retry, **kwargs)
File "/usr/local/autotest/server/site_server_job.py", line 48, in __init__
super(site_server_job, self).__init__(*args, **dargs)
File "/usr/local/autotest/server/server_job.py", line 331, in __init__
self.machines, self.in_lab, host_attributes)
File "/usr/local/autotest/server/server_job.py", line 107, in get_machine_dicts
'host_info_store': _create_host_info_store(machine, in_lab),
File "/usr/local/autotest/server/server_job.py", line 1459, in _create_host_info_store
host_info_store.get(force_refresh=True)
File "/usr/local/autotest/server/hosts/host_info.py", line 142, in get
return self._get_uncached()
File "/usr/local/autotest/server/hosts/host_info.py", line 196, in _get_uncached
info = self._refresh_impl()
File "/usr/local/autotest/server/hosts/afe_store.py", line 37, in _refresh_impl
hosts = self._afe.get_hosts(hostname=self._hostname)
File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
hosts = self.run('get_hosts', **query_args)
File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
self, call, **dargs)
File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 122, in GenericRetry
ret = functor(*args, **kwargs)
File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 81, in _run
return super(RetryingAFE, self).run(call, **dargs)
File "/usr/local/autotest/server/frontend.py", line 107, in run
result = utils.strip_unicode(rpc_call(**dargs))
File "/usr/local/autotest/frontend/afe/json_rpc/proxy.py", line 114, in __call__
respdata = urllib2.urlopen(request).read()
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.
Comment 1 by xixuan@chromium.org
, Mar 10 2017Status: Duplicate (was: Untriaged)