Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8961750990478208144

=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : v8.browsing_mobile
  Metric       : API:count_avg/browse_social/browse_social_facebook

Revision             Exit Code      N
chromium@515995      1 +- N/A       20      good
chromium@516094      1 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.social.facebook v8.browsing_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8961750990478208144


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8961663920156068240

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8961663840229768752

=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : v8.browsing_mobile
  Metric       : API:count_avg/browse_social/browse_social_pinterest_infinite_scroll

Revision             Exit Code      N
chromium@519310      1 +- N/A       20      good
chromium@519410      1 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.social.pinterest.infinite.scroll v8.browsing_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8961663920156068240


For feedback, file a bug with component Speed>Bisection

=== Auto-CCing suspected CL author altimin@chromium.org ===

Hi altimin@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : Alexander Timin
  Commit : 8f338a43482e0d68ff895d83aec7a360a2ccbc3d
  Date   : Mon Nov 27 14:48:53 2017
  Subject: [scheduler] Replace TaskQueueManagerDelegate with ThreadController.

Bisect Details
  Configuration: android_nexus7_perf_bisect
  Benchmark    : v8.browsing_mobile
  Metric       : API:count_avg/browse_social/browse_social_pinterest_infinite_scroll

Revision             Exit Code      N
chromium@519200      0 +- N/A       20      good
chromium@519250      0 +- N/A       20      good
chromium@519275      0 +- N/A       20      good
chromium@519288      0 +- N/A       20      good
chromium@519294      0 +- N/A       20      good
chromium@519297      0 +- N/A       20      good
chromium@519298      1 +- N/A       20      bad       <--
chromium@519299      1 +- N/A       20      bad
chromium@519300      1 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.social.pinterest.infinite.scroll v8.browsing_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8961663840229768752


For feedback, file a bug with component Speed>Bisection

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fadbe44c45c6f3f0c3413b19282c64c9514ec33e

commit fadbe44c45c6f3f0c3413b19282c64c9514ec33e
Author: Alexander Timin <altimin@chromium.org>
Date: Thu Nov 30 00:31:33 2017

[scheduler] Fix worker scheduler shutdown.

- Use base::TimeTicks::Now as a fallback in SchedulerHelper after
TaskQueueManager was destroyed. This is needed for finalising tracing
after shutdown.
- Use thread-safe flag in WebThreadImplForWorkerScheduler to avoid race
condition.

R=alexclarke@chromium.org
BUG= 788797 

Change-Id: Ia3fe758a9884c11e8f83481a3d81ec8438a08863
Reviewed-on: https://chromium-review.googlesource.com/797040
Commit-Queue: Alexander Timin <altimin@chromium.org>
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Cr-Commit-Position: refs/heads/master@{#520337}
[modify] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/BUILD.gn
[modify] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/DEPS
[rename] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/child/pollable_thread_safe_flag.cc
[rename] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/child/pollable_thread_safe_flag.h
[modify] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/child/scheduler_helper.cc
[modify] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/child/webthread_impl_for_worker_scheduler.cc
[modify] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/child/webthread_impl_for_worker_scheduler.h
[modify] https://crrev.com/fadbe44c45c6f3f0c3413b19282c64c9514ec33e/third_party/WebKit/Source/platform/scheduler/renderer/renderer_scheduler_impl.h

This appears to still be failing. It also affects v8.broswing_mobile-future, with the same failure mode.

Hm, the fix that I landed definitely helped to some degree and I can't reproduce it anymore on my device. Could I ask you to try to repro it and share a stack trace if possible?

 Issue 792026  has been merged into this issue.

Sorry, I pinged this bug as bot health sheriff and do not have hardware locally to try to repro. Perhaps the current failure mode is different, though. Unassigning myself and adding test owners mythria@ and ulan@ to confirm.

I think some of the stories are failing because of this bug:  crbug.com/785446 .

Though I also see failures on facebook_infinite_scroll and pinterest_infinite_scroll that are not related to that bug. For example facebook_infinite_scroll and pinterest_infinite_scroll pages in this log: 
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F817%2F%2B%2Frecipes%2Fsteps%2Fv8.browsing_mobile-future_on_Android%2F0%2Fstdout

The trace looks as follows:

 Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 135, in _RunStoryAndProcessErrorIfNeeded
    test.Measure(state.platform, results)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 261, in Measure
    trace_result, _ = platform.tracing_controller.StopTracing()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 53, in StopTracing
    return self._tracing_controller_backend.StopTracing()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 140, in StopTracing
    '\n'.join(raised_exception_messages))
TracingException: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing
    agent.CollectAgentTraceData(builder)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 250, in CollectAgentTraceData
    '\n'.join(raised_exception_messages))
ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing:
Error when collecting Chrome tracing on devtools at port localabstract:chrome_devtools_remote:
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 239, in CollectAgentTraceData
    client.CollectChromeTracingData(trace_data_builder)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 405, in CollectChromeTracingData
    self._tracing_backend.CollectTraceData(trace_data_builder, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 231, in CollectTraceData
    self._CollectTracingData(trace_data_builder, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 259, in _CollectTracingData
    traceback.format_exc())
TracingUnrecoverableException: Exception raised while collecting tracing data:
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 252, in _CollectTracingData
    self._inspector_websocket.DispatchNotifications(timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 140, in DispatchNotifications
    self._Receive(timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 155, in _Receive
    data = self._socket.recv()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv
    opcode, data = self.recv_data()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame
    frame = self.recv_frame()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame
    self.recv_header()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header
    header = self.recv_strict(2)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv
    return recv(self.sock, bufsize)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 93, in recv
    "Connection is already closed.")
WebSocketConnectionClosedException: Connection is already closed.

altimin@, does this look similar to what you have fixed earlier? If not, I will disable both pages and create a new bug.

It sounds like the hypothesis is that this bug can maybe be re-closed and a new bug filed.  ->altimin to respond to comment 12; can maybe be reassigned to mythria afterwards.

Taking out of sheriff queue for now.

I tried to repro this again on my device and wasn't able to (I was able to repro easily  before the fix).

Re #12: This is just the message that Chrome crashed. The stacktrace from the crash is the most interesting thing, but it's missing.

mythria@, could you please try to run this test on your device (or let's disable it otherwise).

Based on an offline conversation with altimin@ and mythria@, I'm going to send out a CL disabling this benchmark on Android One for now. They'll probably get around to looking at this deeper after Christmas.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/99554ae8ff4454b38f5ae3250c2ae3d87f0a2b3f

commit 99554ae8ff4454b38f5ae3250c2ae3d87f0a2b3f
Author: Charlie Andrews <charliea@chromium.org>
Date: Wed Dec 13 19:16:25 2017

Disable v8.browsing_mobile[-future] on Android One

The benchmark has been failing since around November 27 and, even though
altimin@ was successful in fixing some failures that were reproducible
locally, the benchmark still seems to be failing on the waterfall and
mythria@ doesn't think that it's likely that we'll be able to debug it
before the holiday vacation.

TBR=altimin@chromium.org,mythria@chromium.org,nednguyen@google.com

Bug:  788797 
Change-Id: Ib1b245524afd11400792250f34f0fae6cd952d85
Reviewed-on: https://chromium-review.googlesource.com/824942
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Reviewed-by: Alexander Timin <altimin@chromium.org>
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#523843}
[modify] https://crrev.com/99554ae8ff4454b38f5ae3250c2ae3d87f0a2b3f/tools/perf/benchmarks/v8_browsing.py

 Issue 736526  has been merged into this issue.

I tried running this v8.browsing_mobile on android one locally. I ran it for about 20 times, and apart from facebook_infinite_scroll and tumblr_infinite_scroll all other stories ran without any failures. facebook_infinite_scroll failed 5 out of 20 times and tumlr_infinite_scroll failed 6 out of 20 times. So, I am going to disable these two pages and enable the benchmark and see how it does on the bots. I will try to investigate the failures on facebook and tumblr. They always fail with the following failure:

Traceback (most recent call last):
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 127, in _RunStoryAndProcessErrorIfNeeded
    test.Measure(state.platform, results)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 261, in Measure
    trace_result, _ = platform.tracing_controller.StopTracing()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 53, in StopTracing
    return self._tracing_controller_backend.StopTracing()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 140, in StopTracing
    '\n'.join(raised_exception_messages))
TracingException: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing
    agent.CollectAgentTraceData(builder)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 250, in CollectAgentTraceData
    '\n'.join(raised_exception_messages))
ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing:
Error when collecting Chrome tracing on devtools at port localabstract:chrome_devtools_remote:
Traceback (most recent call last):
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 239, in CollectAgentTraceData
    client.CollectChromeTracingData(trace_data_builder)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 458, in CollectChromeTracingData
    self._tracing_backend.CollectTraceData(trace_data_builder, timeout)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 231, in CollectTraceData
    self._CollectTracingData(trace_data_builder, timeout)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 259, in _CollectTracingData
    traceback.format_exc())
TracingUnrecoverableException: Exception raised while collecting tracing data:
Traceback (most recent call last):
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 252, in _CollectTracingData
    self._inspector_websocket.DispatchNotifications(timeout)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 140, in DispatchNotifications
    self._Receive(timeout)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 155, in _Receive
    data = self._socket.recv()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv
    opcode, data = self.recv_data()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame
    self.recv_header()
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header
    header = self.recv_strict(2)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv
    return recv(self.sock, bufsize)
  File "/usr/local/google/code/clankium/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 93, in recv
    "Connection is already closed.")
WebSocketConnectionClosedException: Connection is already closed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6d25f4059bbfb5c6567f74cc917233f408564634

commit 6d25f4059bbfb5c6567f74cc917233f408564634
Author: Mythri Alle <mythria@chromium.org>
Date: Mon Jan 08 18:34:02 2018

Enable v8.browsing_mobile benchmark on Android one

Enable v8.browsing_mobile on Android one. This was failing frequently
because of  crbug.com/785446 . Now, that it is fixed enabling this
benchmark again.

Disable facebook_infinite_scroll and pinterest_infinite_scroll that
fail locally. These pages also fail on the v8.browsing_mobile-future
benchmarks on the bots consistently. So it is safer to disable them
for now.

Bug:  chromium:788797 
Change-Id: I86dabec4aaebf3ea5299105de77a9a87cb1936f0
Reviewed-on: https://chromium-review.googlesource.com/854292
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#527680}
[modify] https://crrev.com/6d25f4059bbfb5c6567f74cc917233f408564634/tools/perf/expectations.config

I think infinite scroll pages are a problem on android because of the large traces they generate. We have another bug to track this.