New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 713932 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Last visit > 30 days ago
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

lumpy-chrome-pfq, reef-chrome-pfq failed with provision failure

Project Member Reported by x...@chromium.org, Apr 20 2017

Issue description

lumpy: https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-chrome-pfq/builds/9991
reef: https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq/builds/349

Selected error messages:
@@@STEP_LINK@[Auto-Bug]: provision: FAIL: Unhandled TimeoutError: Timeout occurred- waited 300 seconds., 144 reports@https://code.google.com/p/chromium/issues/detail?id=613447@@@
@@@STEP_LINK@[Test-Logs]: provision: FAIL: Unhandled TimeoutError: Timeout occurred- waited 300 seconds.@http://cautotest/tko/retrieve_logs.cgi?job=/results/113533359-chromeos-test/@@@

There are a lot of lines in the log showing the status of the host devices are repair failed. Not sure if it's related.

 

Comment 1 by x...@chromium.org, Apr 20 2017

Cc: -nxia@chromium.org
Owner: nxia@chromium.org
Status: Assigned (was: Available)
This does look like infra issues. Assign to current infra deputy for now.
Components: Infra>Client>ChromeOS
This is (essentially) bug 708679.  The symptom has shifted a bit,
but key elements remain.  The signature of this problem from the
provision and repair logs is in the traceback below.  The most
important line is this one:
    File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
      hosts = self.run('get_hosts', **query_args)

The marker is the call to 'self.run(', that times out, regardless of
whether it's "get_hosts", or some other call.  That's the RPC call
timing out because of a problem on the shard.

We've already restarted the shard: it serves both lumpy and reef.

====
Unhandled TimeoutError: Timeout occurred- waited 300 seconds.
  Traceback (most recent call last):
    File "/usr/local/autotest/client/common_lib/test.py", line 817, in _call_test_function
      return func(*args, **dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 470, in execute
      dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
      postprocess_profiled_run, args, dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 380, in _call_run_once
      self.run_once(*args, **dargs)
    File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once
      force_full_update=force)
    File "/usr/local/autotest/server/afe_utils.py", line 201, in machine_install_and_update_labels
      clear_host_attributes_before_provision(host)
    File "/usr/local/autotest/server/afe_utils.py", line 170, in clear_host_attributes_before_provision
      update_host_attribute(host, attribute, None)
    File "/usr/local/autotest/server/afe_utils.py", line 187, in update_host_attribute
      if get_host_attribute(host, attribute, use_local_value=False) != value:
    File "/usr/local/autotest/server/afe_utils.py", line 151, in get_host_attribute
      hosts = AFE.get_hosts(hostname=host.hostname)
    File "/usr/local/autotest/server/frontend.py", line 538, in get_hosts
      hosts = self.run('get_hosts', **query_args)
    File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 111, in run
      self, call, **dargs)
    File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 122, in GenericRetry
      ret = functor(*args, **kwargs)
    File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 81, in _run
      return super(RetryingAFE, self).run(call, **dargs)
    File "/usr/local/autotest/server/frontend.py", line 107, in run
      result = utils.strip_unicode(rpc_call(**dargs))
    File "/usr/local/autotest/frontend/afe/json_rpc/proxy.py", line 114, in __call__
      respdata = urllib2.urlopen(request).read()
    File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
      return _opener.open(url, data, timeout)
    File "/usr/lib/python2.7/urllib2.py", line 404, in open
      response = self._open(req, data)
    File "/usr/lib/python2.7/urllib2.py", line 422, in _open
      '_open', req)
    File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
      result = func(*args)
    File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
      return self.do_open(httplib.HTTPConnection, req)
    File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
      r = h.getresponse(buffering=True)
    File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
      response.begin()
    File "/usr/lib/python2.7/httplib.py", line 444, in begin
      version, status, reason = self._read_status()
    File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
      line = self.fp.readline(_MAXLINE + 1)
    File "/usr/lib/python2.7/socket.py", line 476, in readline
      data = self._sock.recv(self._rbufsize)
    File "/usr/local/autotest/site-packages/chromite/lib/timeout_util.py", line 62, in kill_us
      raise TimeoutError(error_message % {'time': max_run_time})
  TimeoutError: Timeout occurred- waited 300 seconds.
Mergedinto: 708679
Status: Duplicate (was: Assigned)

Sign in to add a comment