lumpy-chrome-pfq, reef-chrome-pfq failed with provision failure |
|||
Issue descriptionlumpy: https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-chrome-pfq/builds/9991 reef: https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq/builds/349 Selected error messages: @@@STEP_LINK@[Auto-Bug]: provision: FAIL: Unhandled TimeoutError: Timeout occurred- waited 300 seconds., 144 reports@https://code.google.com/p/chromium/issues/detail?id=613447@@@ @@@STEP_LINK@[Test-Logs]: provision: FAIL: Unhandled TimeoutError: Timeout occurred- waited 300 seconds.@http://cautotest/tko/retrieve_logs.cgi?job=/results/113533359-chromeos-test/@@@ There are a lot of lines in the log showing the status of the host devices are repair failed. Not sure if it's related.
,
Apr 20 2017
This is (essentially) bug 708679. The symptom has shifted a bit,
but key elements remain. The signature of this problem from the
provision and repair logs is in the traceback below. The most
important line is this one:
File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
hosts = self.run('get_hosts', **query_args)
The marker is the call to 'self.run(', that times out, regardless of
whether it's "get_hosts", or some other call. That's the RPC call
timing out because of a problem on the shard.
We've already restarted the shard: it serves both lumpy and reef.
====
Unhandled TimeoutError: Timeout occurred- waited 300 seconds.
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/test.py", line 817, in _call_test_function
return func(*args, **dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 470, in execute
dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
postprocess_profiled_run, args, dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 380, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once
force_full_update=force)
File "/usr/local/autotest/server/afe_utils.py", line 201, in machine_install_and_update_labels
clear_host_attributes_before_provision(host)
File "/usr/local/autotest/server/afe_utils.py", line 170, in clear_host_attributes_before_provision
update_host_attribute(host, attribute, None)
File "/usr/local/autotest/server/afe_utils.py", line 187, in update_host_attribute
if get_host_attribute(host, attribute, use_local_value=False) != value:
File "/usr/local/autotest/server/afe_utils.py", line 151, in get_host_attribute
hosts = AFE.get_hosts(hostname=host.hostname)
File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
hosts = self.run('get_hosts', **query_args)
File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
self, call, **dargs)
File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 122, in GenericRetry
ret = functor(*args, **kwargs)
File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 81, in _run
return super(RetryingAFE, self).run(call, **dargs)
File "/usr/local/autotest/server/frontend.py", line 107, in run
result = utils.strip_unicode(rpc_call(**dargs))
File "/usr/local/autotest/frontend/afe/json_rpc/proxy.py", line 114, in __call__
respdata = urllib2.urlopen(request).read()
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 300 seconds.
,
Apr 20 2017
|
|||
►
Sign in to add a comment |
|||
Comment 1 by x...@chromium.org
, Apr 20 2017Owner: nxia@chromium.org
Status: Assigned (was: Available)