some autotest unittests are full of slow pointless elasticsearch retry churn |
|||||
Issue description
Example:
~/chromiumos/src/third_party/autotest/files/frontend/afe$ ./rpc_interface_unittest.py
<snip>
File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
self.connect()
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 141, in connect
conn = self._new_conn()
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 120, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
ProtocolError: ('Connection aborted.', error(110, 'Connection timed out'))
DEBUG:elasticsearch:> {"time_recorded": 1490246895.047299, "changed_by": "autotest_system", "hostname": "host9", "locked": true, "lock_reason": "Testing forward to shard"}
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
INFO:urllib3.connectionpool:Starting new HTTP connection (4): 172.25.65.90
WARNING:elasticsearch:POST http://172.25.65.90:9200/cautotest/lock_history [status:N/A request:15.023s]
Traceback (most recent call last):
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/elasticsearch/connection/http_urllib3.py", line 74, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connectionpool.py", line 559, in urlopen
_pool=self, _stacktrace=stacktrace)
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/util/retry.py", line 223, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connectionpool.py", line 516, in urlopen
body=body, headers=headers)
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connectionpool.py", line 308, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1017, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
self.connect()
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 141, in connect
conn = self._new_conn()
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/connection.py", line 120, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
ProtocolError: ('Connection aborted.', error(110, 'Connection timed out'))
DEBUG:elasticsearch:> {"time_recorded": 1490246895.047299, "changed_by": "autotest_system", "hostname": "host9", "locked": true, "lock_reason": "Testing forward to shard"}
ERROR:root:ConnectionError(('Connection aborted.', error(110, 'Connection timed out'))) caused by: ProtocolError(('Connection aborted.', error(110, 'Connection timed out')))
.........
----------------------------------------------------------------------
Ran 51 tests in 681.206s
Most of the time looks to me like it is spent in retry loops on elasticsearch metrics, which a unittest really shouldn't even be sending.
,
Mar 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ace5189e88ce40bbec62162b5aee13d6173f3a42 commit ace5189e88ce40bbec62162b5aee13d6173f3a42 Author: Dan Shi <dshi@google.com> Date: Sat Mar 25 02:38:06 2017 [autotest] Remove test instance of metadb in global config There is no need for uploading metadata in non-prod settings. BUG= chromium:704420 TEST=None Change-Id: Ibe33b01502161568421ffd618bbf52d9348f8d23 Reviewed-on: https://chromium-review.googlesource.com/458677 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/ace5189e88ce40bbec62162b5aee13d6173f3a42/global_config.ini
,
Mar 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ace5189e88ce40bbec62162b5aee13d6173f3a42 commit ace5189e88ce40bbec62162b5aee13d6173f3a42 Author: Dan Shi <dshi@google.com> Date: Sat Mar 25 02:38:06 2017 [autotest] Remove test instance of metadb in global config There is no need for uploading metadata in non-prod settings. BUG= chromium:704420 TEST=None Change-Id: Ibe33b01502161568421ffd618bbf52d9348f8d23 Reviewed-on: https://chromium-review.googlesource.com/458677 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/ace5189e88ce40bbec62162b5aee13d6173f3a42/global_config.ini
,
Mar 25 2017
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by dshi@chromium.org
, Mar 23 2017Owner: dshi@chromium.org