New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 884064 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Sep 14
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug-Regression

Blocking:
issue 884374



Sign in to add a comment

(Flake) DevToolsClientConnectionError: timed out

Project Member Reported by kainino@chromium.org, Sep 14

Issue description

Seen this across these builds just recently:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20(AMD)/7586
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20(AMD)/7593
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20(AMD)/7596

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20(AMD)/4972

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20Retina%20Debug%20(AMD)/33127

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20Retina%20Release%20(AMD)/40089
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20Retina%20Release%20(AMD)/40103

Apparently it's prevalent on these Mac Retina AMD bots.

One stack trace (from 7596):

Traceback (most recent call last):
  <module> at /b/s/w/ir/third_party/catapult/telemetry/telemetry/testing/run_browser_tests.py:359
    ret_code = RunTests(sys.argv[1:])
  RunTests at /b/s/w/ir/third_party/catapult/telemetry/telemetry/testing/run_browser_tests.py:328
    ret, _, _ = runner.run()
  run at /b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py:179
    ret, full_results = self._run_tests(result_set, test_set)
  _run_tests at /b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py:466
    self._run_one_set(self.stats, result_set, test_set)
  _run_one_set at /b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py:511
    test_set.isolated_tests, 1)
  _run_list at /b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py:537
    _setup_process, _teardown_process)
  make_pool at /b/s/w/ir/third_party/catapult/third_party/typ/typ/pool.py:28
    return _AsyncPool(host, jobs, callback, context, pre_fn, post_fn)
  __init__ at /b/s/w/ir/third_party/catapult/third_party/typ/typ/pool.py:188
    self.context_after_pre = pre_fn(self.host, 1, self.context)
  _setup_process at /b/s/w/ir/third_party/catapult/third_party/typ/typ/runner.py:807
    child.context_after_setup = child.setup_fn(child, child.context)
  _SetUpProcess at /b/s/w/ir/third_party/catapult/telemetry/telemetry/testing/run_browser_tests.py:349
    context.test_class.SetUpProcess()
  SetUpProcess at /b/s/w/ir/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:297
    cls.StartBrowser()
  StartBrowser at /b/s/w/ir/content/test/gpu/gpu_tests/gpu_integration_test.py:96
    cls.tab = cls.browser.tabs[0]
  __getitem__ at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/tab_list.py:18
    return self._tab_list_backend.__getitem__(index)
  __getitem__ at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py:56
    self._Update()
  _Update at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py:99
    backends_map = self._devtools_client.GetUpdatedInspectableContexts()
  GetUpdatedInspectableContexts at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py:399
    contexts = self._ListInspectableContexts()
  _ListInspectableContexts at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py:324
    return self._devtools_http.RequestJson('')
  RequestJson at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py:107
    return json.loads(self.Request(path, timeout))
  Request at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py:88
    response = self._conn.getresponse()
  getresponse at /b/s/w/ir/.swarming_module/lib/python2.7/httplib.py:1121
    response.begin()
  begin at /b/s/w/ir/.swarming_module/lib/python2.7/httplib.py:438
    version, status, reason = self._read_status()
  _read_status at /b/s/w/ir/.swarming_module/lib/python2.7/httplib.py:394
    line = self.fp.readline(_MAXLINE + 1)
  readline at /b/s/w/ir/.swarming_module/lib/python2.7/socket.py:480
    data = self._sock.recv(self._rbufsize)
DevToolsClientConnectionError: timed out

 
Searching monorail for "DevToolsClientConnectionError" or "DevToolsClientConnectionError: timed out" gives some results that COULD be related, but none obviously related.
Cc: ellyjo...@chromium.org ccameron@chromium.org
Components: Internals>GPU>Testing Blink>WebGL Platform>DevTools
Labels: -Type-Bug -Pri-2 Pri-1 Type-Bug-Regression
Owner: kainino@chromium.org
Status: Assigned (was: Untriaged)
Fortunately, digging in to a few of the failing shards:

https://chromium-swarm.appspot.com/task?id=3fec8844062b2210&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3fed82405b433010&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3fedeee18cee3010&refresh=10&show_raw=1

indicate they're all failing on a single bot, build211-m1:

https://chromium-swarm.appspot.com/bot?id=build211-m1&sort_stats=total%3Adesc

which is failing basically all tests it runs.

I've taken it out of Swarming with the "Shut down gracefully" button. Could you please file an Infra>Labs ticket about getting it looked at and turned back on? Search for any similar tickets I've filed for the basic template. ("Build infrastructure" type.)

Components: -Infra>Platform>Swarming Infra>Platform>Swarming>Admin
Labels: Hotlist-PixelWrangler
This particular bot was fixed in issue 884067, but I'm now seeing errors like this across many bots. I need to investigate a little more.
Status: Fixed (was: Assigned)
They seem to have stopped for now. Original issue was fixed in 884067 so marking this as fixed.
Blocking: 884374

Sign in to add a comment