New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 704420 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

some autotest unittests are full of slow pointless elasticsearch retry churn

Project Member Reported by akes...@chromium.org, Mar 23 2017

Issue description

Example:
~/chromiumos/src/third_party/autotest/files/frontend/afe$ ./rpc_interface_unittest.py 

<snip>

  File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.connect()
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 141, in connect
    conn = self._new_conn()
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 120, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
ProtocolError: ('Connection aborted.', error(110, 'Connection timed out'))
DEBUG:elasticsearch:> {"time_recorded": 1490246895.047299, "changed_by": "autotest_system", "hostname": "host9", "locked": true, "lock_reason": "Testing forward to shard"}
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
INFO:urllib3.connectionpool:Starting new HTTP connection (4): 172.25.65.90
WARNING:elasticsearch:POST http://172.25.65.90:9200/cautotest/lock_history [status:N/A request:15.023s]
Traceback (most recent call last):
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/elasticsearch/connection/http_urllib3.py", line 74, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connectionpool.py", line 559, in urlopen
    _pool=self, _stacktrace=stacktrace)
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/util/retry.py", line 223, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connectionpool.py", line 308, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1017, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.connect()
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 141, in connect
    conn = self._new_conn()
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 120, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
ProtocolError: ('Connection aborted.', error(110, 'Connection timed out'))
DEBUG:elasticsearch:> {"time_recorded": 1490246895.047299, "changed_by": "autotest_system", "hostname": "host9", "locked": true, "lock_reason": "Testing forward to shard"}
ERROR:root:ConnectionError(('Connection aborted.', error(110, 'Connection timed out'))) caused by: ProtocolError(('Connection aborted.', error(110, 'Connection timed out')))
.........
----------------------------------------------------------------------
Ran 51 tests in 681.206s

Most of the time looks to me like it is spent in retry loops on elasticsearch metrics, which a unittest really shouldn't even be sending.
 

Comment 1 by dshi@chromium.org, Mar 23 2017

Cc: -dshi@chromium.org
Owner: dshi@chromium.org
The test instance of metadb was down, that's why the retrying. I will remove the instance name from global_config, so non-prod setting won't try to upload to metadb.
Project Member

Comment 2 by bugdroid1@chromium.org, Mar 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ace5189e88ce40bbec62162b5aee13d6173f3a42

commit ace5189e88ce40bbec62162b5aee13d6173f3a42
Author: Dan Shi <dshi@google.com>
Date: Sat Mar 25 02:38:06 2017

[autotest] Remove test instance of metadb in global config

There is no need for uploading metadata in non-prod settings.

BUG= chromium:704420 
TEST=None

Change-Id: Ibe33b01502161568421ffd618bbf52d9348f8d23
Reviewed-on: https://chromium-review.googlesource.com/458677
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/ace5189e88ce40bbec62162b5aee13d6173f3a42/global_config.ini

Project Member

Comment 3 by bugdroid1@chromium.org, Mar 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ace5189e88ce40bbec62162b5aee13d6173f3a42

commit ace5189e88ce40bbec62162b5aee13d6173f3a42
Author: Dan Shi <dshi@google.com>
Date: Sat Mar 25 02:38:06 2017

[autotest] Remove test instance of metadb in global config

There is no need for uploading metadata in non-prod settings.

BUG= chromium:704420 
TEST=None

Change-Id: Ibe33b01502161568421ffd618bbf52d9348f8d23
Reviewed-on: https://chromium-review.googlesource.com/458677
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/ace5189e88ce40bbec62162b5aee13d6173f3a42/global_config.ini

Comment 4 by dshi@chromium.org, Mar 25 2017

Status: Fixed (was: Untriaged)

Comment 5 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 6 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Comment 7 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment