New issue
Advanced search Search tips

Issue 829161 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.v8.runtime_stats.top_25 in telemetry_perf_unittests failing on multiple builders

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Apr 5 2018

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of shend@chromium.org

benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.v8.runtime_stats.top_25 in telemetry_perf_unittests failing on multiple builders

Builders failed on: 
- Mac10.10 Tests: 
  https://build.chromium.org/p/chromium.mac/builders/Mac10.10%20Tests
- Mac10.11 Tests: 
  https://build.chromium.org/p/chromium.mac/builders/Mac10.11%20Tests

First failure:
https://ci.chromium.org/buildbot/chromium.mac/Mac10.11%20Tests/25000

but nothing in blamelist seems suspect.

Looks like something wrong with web sockets timing out:

  [ RUN      ] https://www.google.de/search?q=v8@{'cache_temperature': 'hot-browser'}
  Traceback (most recent call last):
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 116, in _RunStoryAndProcessErrorIfNeeded
      state.WillRunStory(story)
    File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
      return func(*args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 258, in WillRunStory
      self._current_page, self.browser, self._previous_page)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 188, in EnsurePageCacheTemperature
      c.PrepareBrowserCache(page, browser, previous_page)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 86, in PrepareBrowserCache
      cls.PrepareRendererCache(page, tab, previous_page)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 174, in PrepareRendererCache
      _ClearCacheAndData(tab, page.url)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/cache_temperature.py", line 62, in _ClearCacheAndData
      tab.ClearDataForOrigin(url)
    File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
      return func(*args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/tab.py", line 287, in ClearDataForOrigin
      return self._inspector_backend.ClearDataForOrigin(url, timeout)
    File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
      return func(*args, **kwargs)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 120, in ClearDataForOrigin
      self._storage.ClearDataForOrigin(url, timeout)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_storage.py", line 24, in ClearDataForOrigin
      }}, timeout)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 116, in SyncRequest
      res = self._Receive(timeout)
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 155, in _Receive
      data = self._socket.recv()
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv
      opcode, data = self.recv_data()
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data
      opcode, frame = self.recv_data_frame(control_frame)
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame
      frame = self.recv_frame()
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame
      return self.frame_buffer.recv_frame()
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame
      self.recv_header()
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header
      header = self.recv_strict(2)
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict
      bytes_ = self.recv(min(16384, shortage))
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv
      return recv(self.sock, bufsize)
    File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv
      raise WebSocketTimeoutException(message)
  WebSocketTimeoutException: timed out
 

Comment 1 by shend@chromium.org, Apr 5 2018

Components: Tests>Telemetry
Owner: cbruni@chromium.org
Status: Assigned (was: Available)
Hi cbruni, can you take a look at this? Thanks!
Cc: cbruni@chromium.org hablich@chromium.org
Owner: shend@chromium.org
Sorry, I don't know how to handle this issue. 
- What build configuration does this bot run (I cannot see any gn-args anywhere)?
- What command would I have to run to repro this?
- Could we run a functional bisect on the benchmark? ( I don't know how to do this)
- Is this result flaky or really failing? (see below the benchmark just runs fine)

The bots seem to happily produce output[1], so I don't think there is a real issue here.
Based on the revision range of the first failure there is no definitely v8 involved there.


[1] https://chromeperf.appspot.com/report?sid=2e5b53ab50fa555eb06a09d2da6850fe198d279d12b9060211f17171db5de63c&start_rev=539088&end_rev=548275

Comment 3 by shend@chromium.org, Apr 5 2018

Hmm it's been consistently failing on Mac10.11 and flaky on Mac10.10: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=telemetry_perf_unittests&tests=top_25

Comment 4 by shend@chromium.org, Apr 5 2018

Not sure how to debug/reproduce. There are some instructions here: https://www.chromium.org/developers/telemetry/diagnosing-test-failures

but might not be useful. It seems like something wrong with the telemetry infra on Mac, so I'm just going to disable this test and assign someone who might know more about this.

Comment 5 by shend@chromium.org, Apr 6 2018

Owner: nednguyen@chromium.org
Hi Ned, do you happen to know what might be the issue here? Telemetry appears to be trying to connect to some inspector thing via websockets but is timing out (see stack trace at the top).
Project Member

Comment 6 by bugdroid1@chromium.org, Apr 6 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff42d47f48bb746624a0a260b4b86a6369ae36af

commit ff42d47f48bb746624a0a260b4b86a6369ae36af
Author: Darren Shen <shend@chromium.org>
Date: Fri Apr 06 08:13:51 2018

Disable BenchmarkSmokeTest.v8.runtime_stats.top_25 on Mac.

Test consistently failing on Mac10.11 and flaky on Mac10.10.

TBR=nednguyen@chromium.org

No-Try: true
No-Presubmit: true
No-Tree-Checks: true

Bug:  829161 
Change-Id: I89fc7186a5c3c76853ec666bca57409e76950d73
Reviewed-on: https://chromium-review.googlesource.com/999232
Commit-Queue: Darren Shen <shend@chromium.org>
Reviewed-by: Darren Shen <shend@chromium.org>
Cr-Commit-Position: refs/heads/master@{#548710}
[modify] https://crrev.com/ff42d47f48bb746624a0a260b4b86a6369ae36af/tools/perf/expectations.config

Labels: -Sheriff-Chromium
As test is now disabled, removing from sheriff queue.
Owner: cbruni@chromium.org
Basically the method clear cache data timed out. So I am guessing that the cache is too big?

https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_storage.py?rcl=2c59f678c7ede8a844fb687525d594b71aabe3dd&l=20

This is a problem specific to v8.runtime_stats.top_25, so reassign to cbruni@. I think the solution here is either trying to debug why clearing cache on this page is taking longer, or to extend the cache clearing timeout.
Project Member

Comment 10 by 42576172...@developer.gserviceaccount.com, Apr 16 2018

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/14fec576c40000
😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/1480549bc40000

Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/14b12a3bc40000
📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/16988627c40000
Status: Fixed (was: Assigned)
The benchmarks are enabled again. Closing

Sign in to add a comment