benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.v8.runtime_stats.top_25 in telemetry_perf_unittests failing on multiple builders |
||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of shend@chromium.org benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.v8.runtime_stats.top_25 in telemetry_perf_unittests failing on multiple builders Builders failed on: - Mac10.10 Tests: https://build.chromium.org/p/chromium.mac/builders/Mac10.10%20Tests - Mac10.11 Tests: https://build.chromium.org/p/chromium.mac/builders/Mac10.11%20Tests First failure: https://ci.chromium.org/buildbot/chromium.mac/Mac10.11%20Tests/25000 but nothing in blamelist seems suspect. Looks like something wrong with web sockets timing out: [ RUN ] https://www.google.de/search?q=v8@{'cache_temperature': 'hot-browser'} Traceback (most recent call last): File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 116, in _RunStoryAndProcessErrorIfNeeded state.WillRunStory(story) File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 258, in WillRunStory self._current_page, self.browser, self._previous_page) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 188, in EnsurePageCacheTemperature c.PrepareBrowserCache(page, browser, previous_page) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 86, in PrepareBrowserCache cls.PrepareRendererCache(page, tab, previous_page) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 174, in PrepareRendererCache _ClearCacheAndData(tab, page.url) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 62, in _ClearCacheAndData tab.ClearDataForOrigin(url) File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/tab.py", line 287, in ClearDataForOrigin return self._inspector_backend.ClearDataForOrigin(url, timeout) File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 120, in ClearDataForOrigin self._storage.ClearDataForOrigin(url, timeout) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_storage.py", line 24, in ClearDataForOrigin }}, timeout) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 116, in SyncRequest res = self._Receive(timeout) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 155, in _Receive data = self._socket.recv() File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv opcode, data = self.recv_data() File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data opcode, frame = self.recv_data_frame(control_frame) File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame frame = self.recv_frame() File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame return self.frame_buffer.recv_frame() File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame self.recv_header() File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header header = self.recv_strict(2) File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict bytes_ = self.recv(min(16384, shortage)) File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv return recv(self.sock, bufsize) File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv raise WebSocketTimeoutException(message) WebSocketTimeoutException: timed out
,
Apr 5 2018
Sorry, I don't know how to handle this issue. - What build configuration does this bot run (I cannot see any gn-args anywhere)? - What command would I have to run to repro this? - Could we run a functional bisect on the benchmark? ( I don't know how to do this) - Is this result flaky or really failing? (see below the benchmark just runs fine) The bots seem to happily produce output[1], so I don't think there is a real issue here. Based on the revision range of the first failure there is no definitely v8 involved there. [1] https://chromeperf.appspot.com/report?sid=2e5b53ab50fa555eb06a09d2da6850fe198d279d12b9060211f17171db5de63c&start_rev=539088&end_rev=548275
,
Apr 5 2018
Hmm it's been consistently failing on Mac10.11 and flaky on Mac10.10: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=telemetry_perf_unittests&tests=top_25
,
Apr 5 2018
Not sure how to debug/reproduce. There are some instructions here: https://www.chromium.org/developers/telemetry/diagnosing-test-failures but might not be useful. It seems like something wrong with the telemetry infra on Mac, so I'm just going to disable this test and assign someone who might know more about this.
,
Apr 6 2018
Hi Ned, do you happen to know what might be the issue here? Telemetry appears to be trying to connect to some inspector thing via websockets but is timing out (see stack trace at the top).
,
Apr 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ff42d47f48bb746624a0a260b4b86a6369ae36af commit ff42d47f48bb746624a0a260b4b86a6369ae36af Author: Darren Shen <shend@chromium.org> Date: Fri Apr 06 08:13:51 2018 Disable BenchmarkSmokeTest.v8.runtime_stats.top_25 on Mac. Test consistently failing on Mac10.11 and flaky on Mac10.10. TBR=nednguyen@chromium.org No-Try: true No-Presubmit: true No-Tree-Checks: true Bug: 829161 Change-Id: I89fc7186a5c3c76853ec666bca57409e76950d73 Reviewed-on: https://chromium-review.googlesource.com/999232 Commit-Queue: Darren Shen <shend@chromium.org> Reviewed-by: Darren Shen <shend@chromium.org> Cr-Commit-Position: refs/heads/master@{#548710} [modify] https://crrev.com/ff42d47f48bb746624a0a260b4b86a6369ae36af/tools/perf/expectations.config
,
Apr 6 2018
As test is now disabled, removing from sheriff queue.
,
Apr 10 2018
Basically the method clear cache data timed out. So I am guessing that the cache is too big? https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_storage.py?rcl=2c59f678c7ede8a844fb687525d594b71aabe3dd&l=20 This is a problem specific to v8.runtime_stats.top_25, so reassign to cbruni@. I think the solution here is either trying to debug why clearing cache on this page is taking longer, or to extend the cache clearing timeout.
,
Apr 16 2018
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/14fec576c40000
,
Apr 16 2018
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/14fec576c40000
,
May 7 2018
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/1480549bc40000
,
May 7 2018
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/1480549bc40000 Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
,
May 7 2018
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/14b12a3bc40000
,
May 8 2018
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/14b12a3bc40000
,
May 8 2018
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/16882507c40000
,
May 8 2018
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/16988627c40000
,
May 8 2018
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/16988627c40000
,
Jun 4 2018
The benchmarks are enabled again. Closing |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by shend@chromium.org
, Apr 5 2018Owner: cbruni@chromium.org
Status: Assigned (was: Available)