The failing story seems to be extra_large_texture_uploads.html

It looks like the first failure of this type occurred around 6760c7cce26a88e252edd85f894e1b4dd0d50a21.

Since then, it's failed 117 times out of the approximately 185 runs for an overall failure rate of 0.63. I want to kick off a bisect around this CL. Assuming that we want to have a 95% chance of the bisect successfully identifying the culprit, that means that we need to figure out how many runs back we have to go such that there's a 95% chance that this 63% flake would have happened at least once. In other words:

0.63^N <= .05
ln(0.63^N) <= ln(0.05)
N * ln(0.63) <= ln(0.05)
N >= ln(.05) / ln(0.63) (remembering to reverse the sign of the inequality because a log of a number less than 1 is negative, and multiplying or dividing by a negative reverses the inequality)
N >= 6.48
N=7

In other words, we need to go back 7 runs from the first failure in order to be 95% sure that the introduction of the failure was in those 7 runs. 

Now how many CLs are in those 7 runs? 74 + 128 + 37 + 38 + 69 + 93 + 91 = 530. Yowza. (If we're looking for good reasons to decrease the cycle time, here's one.) 

A bisect will need to run at ceil(log2(530)) revisions in that range, which is ceil(9.04) or 10 revisions. Assuming that we want a 95% chance for the bisect to succeed, how confident do we have to be that we'll reproduce the bisect at each individual revision?

.95 < X^10
10th root of (.95) < X
.9948 < X

In other words, we need to be at least ~99.5% confident that we'll reproduce the failure at each of those 10 revisions in order to be 95% confident that we'll successfully reproduce the failure at each of the 10 revisions required to bisect a 530 revision range.

Now, how does that translate to the number times that we need to run that particular test at each revision in the range?

0.63 ^ X <= (1 - .9948)
0.63 ^ X <= .0052
ln(0.63 ^ X) <= ln(.0052)
X * ln(0.63) <= ln(.0052)
X >= ln(.0052) / ln(0.63) (remembering to reverse the sign again)
X >= 11.38
X = 12

In other words, the bisect parameters should be:

12 runs at each revision
Good revision 7 before the first failure (bead99874c0f63f1ddf66963cd1af613b4a6531e)
Bad revision (6760c7cce26a88e252edd85f894e1b4dd0d50a21)
Test smoothness.tough_texture_upload_cases/extra_large_texture_uploads.html

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8960200801636520048

While this bisect is running, I'm going to go ahead and disable this story on Android One.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3fc1564b8fee52398b8c3391978b6d90202b47e1

commit 3fc1564b8fee52398b8c3391978b6d90202b47e1
Author: Charlie Andrews <charliea@chromium.org>
Date: Thu Dec 14 21:39:51 2017

e# Enter a description of the change.
Disable flaky smoothness.tough_texture_cases story on Android One

TBR=vmiura@chromium.org, nednguyen@google.com

Bug:  795060 
Change-Id: I9dbaa1e3563b60ad95f20f4ee18935637ee09925
Reviewed-on: https://chromium-review.googlesource.com/827530
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#524192}
[modify] https://crrev.com/3fc1564b8fee52398b8c3391978b6d90202b47e1/tools/perf/benchmarks/smoothness.py

I think this test has been flaky.  extra_large_texture_uploads story likely exceeds some memory limit, or takes too long to run on these devices.

It looks like, for some reason, the test run is hanging indefinitely (for ~20 hours now) when things go south. That's not good.

=== BISECT JOB RESULTS ===
Bisect failed for unknown reasons

Please contact the team (see below) and report the error.


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.tough_texture_upload_cases
  Metric       : jank_count/extra_large_texture_uploads.html


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=extra.large.texture.uploads.html smoothness.tough_texture_upload_cases

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8960200801636520048


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8959847026793514960

Kicking it off again to see if it was just a flake (I'll pass it on to the dashboard team if it was)

=== BISECT JOB RESULTS ===
Bisect was unable to run to completion

Error: INFRA_FAILURE

The bisect was able to narrow the range, you can try running with:
  good_revision: ef44a1f72e225797a49e6856ca7e9180f0e5b7b5
  bad_revision : d4af8b9a73a5bd126a7791ad42eeaa0b0ad75947

If failures persist contact the team (see below) and report the error.


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.tough_texture_upload_cases
  Metric       : frame_times/extra_large_texture_uploads.html

Revision             Exit Code      N
chromium@511420      1 +- N/A       20      good
chromium@511759      1 +- N/A       20      good
chromium@511929      1 +- N/A       20      good
chromium@511972      1 +- N/A       20      good
chromium@511993      1 +- N/A       20      good
chromium@512014      0 +- N/A       20      bad
chromium@512098      0 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=extra.large.texture.uploads.html smoothness.tough_texture_upload_cases

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8959847026793514960


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8959753173847439584

Well... it got further. I wonder if this failure is similar to https://bugs.chromium.org/p/chromium/issues/detail?id=755981, where the Telemetry test seems to fail but doesn't terminate. In this case, though, it doesn't hit the I/O timeout because there's some sort of heartbeat that's still coming out of this code (https://cs.chromium.org/chromium/src/third_party/android_tools/sdk/platform-tools/systrace/catapult/devil/devil/android/perf/surface_stats_collector.py?type=cs&q=%22dumpsys+SurfaceFlinger%22&sq=package:chromium&l=107)

=== Auto-CCing suspected CL author kmarshall@chromium.org ===

Hi kmarshall@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : Kevin Marshall
  Commit : 8cc60afe97c6c6adc5fa4e90cefe86ef8f6a9118
  Date   : Thu Oct 26 22:49:50 2017
  Subject: Fuchsia: restrict loglistener scope to monitoring deployed instance.

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.tough_texture_upload_cases
  Metric       : frame_times/extra_large_texture_uploads.html

Revision             Exit Code      N
chromium@511993      1 +- N/A       10      good
chromium@511994      0 +- N/A       10      bad       <--
chromium@511995      0 +- N/A       10      bad
chromium@511996      0 +- N/A       10      bad
chromium@511999      0 +- N/A       10      bad
chromium@512004      0 +- N/A       10      bad
chromium@512014      0 +- N/A       10      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=extra.large.texture.uploads.html smoothness.tough_texture_upload_cases

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8959753173847439584


For feedback, file a bug with component Speed>Bisection

Looks like this found the wrong culprit - the one it found was a fuchsia CL. Sorry kmarshall@.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8958486258287782496

=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.tough_texture_upload_cases
  Metric       : frame_times/extra_large_texture_uploads.html

Revision             Exit Code      N
chromium@511993      0 +- N/A       20      good
chromium@512014      0 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=extra.large.texture.uploads.html smoothness.tough_texture_upload_cases

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8958486258287782496


For feedback, file a bug with component Speed>Bisection

Bah. Having a heck of a time bisecting this one. Assigning to vmiura@ for investigation.

I can take a look

Android one perf bot is gone, and we don't see the same issues on the Android Go perf bot (the successor to Android One perf), so closing this out.