system_health.common_mobile/browse:social:pinterest_infinite_scroll and facebook_infinite_scroll in system_health.common_mobile failing on chromium.perf/Android One Perf |
||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of charliea@chromium.org system_health.common_mobile/browse:social:pinterest_infinite_scroll in system_health.common_mobile failing on chromium.perf/Android One Perf Builders failed on: - Android One Perf: https://ci.chromium.org/buildbot/chromium.perf/Android%20One%20Perf Based on the flakiness dashboard (https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=system_health.common_mobile&builder=chromium.perf%3AAndroid%20One%20Perf), it looks like this test may have gotten flakier around 30 builds or so ago. I'm going to launch a bisect with a really wide range and see if it turns up anything and disable it (and facebook_infinite_scroll, which is also failing with the same error. Error looks like: ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing: Error when collecting Chrome tracing on devtools at port localabstract:chrome_devtools_remote: Traceback (most recent call last): File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 239, in CollectAgentTraceData client.CollectChromeTracingData(trace_data_builder) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 462, in CollectChromeTracingData self._tracing_backend.CollectTraceData(trace_data_builder, timeout) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 231, in CollectTraceData self._CollectTracingData(trace_data_builder, timeout) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 259, in _CollectTracingData traceback.format_exc()) TracingUnrecoverableException: Exception raised while collecting tracing data: Traceback (most recent call last): File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 252, in _CollectTracingData self._inspector_websocket.DispatchNotifications(timeout) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 140, in DispatchNotifications self._Receive(timeout) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 155, in _Receive data = self._socket.recv() File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv opcode, data = self.recv_data() File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data opcode, frame = self.recv_data_frame(control_frame) File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame frame = self.recv_frame() File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame return self.frame_buffer.recv_frame() File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame self.recv_header() File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header header = self.recv_strict(2) File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict bytes_ = self.recv(min(16384, shortage)) File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv return recv(self.sock, bufsize) File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 93, in recv "Connection is already closed.") WebSocketConnectionClosedException: Connection is already closed.
,
May 11 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c584828b101f1699d00763310f73fc9c761873b0 commit c584828b101f1699d00763310f73fc9c761873b0 Author: Charlie Andrews <charliea@chromium.org> Date: Fri May 11 14:51:06 2018 Disable {facebook,pinterest}_infinite_scroll on Android One NOTRY=true TBR=perezju@chromium.org Bug: 842163 Change-Id: I148198aaa2e8f6dc4eb7ebeba01ef945f5ad6d4f Reviewed-on: https://chromium-review.googlesource.com/1054166 Reviewed-by: Charlie Andrews <charliea@chromium.org> Commit-Queue: Charlie Andrews <charliea@chromium.org> Cr-Commit-Position: refs/heads/master@{#557881} [modify] https://crrev.com/c584828b101f1699d00763310f73fc9c761873b0/tools/perf/expectations.config
,
May 11 2018
📍 Couldn't reproduce a difference. https://pinpoint-dot-chromeperf.appspot.com/job/12ae8f80240000
,
May 14 2018
It looks like pinpoint probably doesn't run repeats to detect this failure. It's a little hard to tell exactly how many times it runs at each revision - maybe 10? - but this story went from being about 0% flaky to about 10% flaky and, from Pinpoint's perspective, it went from 0% flaky to 0% flaky. Here's a look at the flakiness dashboard: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&testType=system_health.common_mobile&builder=chromium.perf%3AAndroid%20One%20Perf dtu@, what are we supposed to do in circumstances like this? Does pinpoint support any ability to bisect relatively fine-grained changes in flakiness like this?
,
May 17 2018
Yes, sorry, Pinpoint is currently tuned to detect performance regressions, which tend to be fairly large. I think we would be able to detect a 0% to 20% change in failure rate, but 10% is too low with the current thresholds. I filed a bug for having different thresholds. https://github.com/catapult-project/catapult/issues/4459
,
Jun 19 2018
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/11eca5bd240000
,
Jun 20 2018
📍 Couldn't reproduce a difference. https://pinpoint-dot-chromeperf.appspot.com/job/11eca5bd240000
,
Jul 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/13c71c1ea40000
,
Jul 19
Rerunning this now that dtu@ has made it possible for pinpoint to detect more subtle changes in flakiness
,
Jul 19
📍 Couldn't reproduce a difference. https://pinpoint-dot-chromeperf.appspot.com/job/13c71c1ea40000
,
Jul 19
It looks like it passed 50 runs. Maybe it's not flaky anymore (???)
,
Jul 23
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6a6d89bca0bf063ccf0b476cc43ad06e4c8a57c7 commit 6a6d89bca0bf063ccf0b476cc43ad06e4c8a57c7 Author: Charlie Andrews <charliea@chromium.org> Date: Mon Jul 23 15:13:16 2018 Reenable facebook and pinterest infinite scroll on Android One A recent bisect in the linked bug suggests that they aren't flaky anymore. NOTRY=true TBR=nednguyen@chromium.org Bug: 842163 Change-Id: I79442bf895ff3596f1b1c8ae3db8ccb596041e78 Reviewed-on: https://chromium-review.googlesource.com/1146765 Reviewed-by: Charlie Andrews <charliea@chromium.org> Commit-Queue: Charlie Andrews <charliea@chromium.org> Cr-Commit-Position: refs/heads/master@{#577170} [modify] https://crrev.com/6a6d89bca0bf063ccf0b476cc43ad06e4c8a57c7/tools/perf/expectations.config
,
Jul 23
|
||||
►
Sign in to add a comment |
||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, May 11 2018