All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=677421

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgv8ThpQoM


Bot(s) for this bug's original alert(s):

android-nexus7v2

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991948697554534976

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@439370  1.26863  6.98612  30  good
chromium@439754  1.76956  12.1214  30  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 677421

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests thread_times.key_mobile_sites_smooth
Test Metric: thread_raster_cpu_time_per_frame/thread_raster_cpu_time_per_frame
Relative Change: 39.49%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3568
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991948697554534976


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5322249316335616

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8990641666934886624

=== Auto-CCing suspected CL author jmadill@chromium.org ===

Hi jmadill@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : jmadill
  Commit : 19fa73af253fb8af1199767812814eb2c3f98295
  Date   : Tue Dec 20 00:55:43 2016
  Subject: Roll ANGLE d967122..37ee8a6

Bisect Details
  Configuration: android_nexus7_perf_bisect
  Benchmark    : thread_times.key_mobile_sites_smooth
  Metric       : thread_raster_cpu_time_per_frame/thread_raster_cpu_time_per_frame
  Change       : 37.06% | 1.32630984059 -> 1.81786888125

Revision             Result                    N
chromium@439370      1.32631 +- 0.460989       6      good
chromium@439562      1.32274 +- 0.384379       6      good
chromium@439610      1.63345 +- 0.839638       9      good
chromium@439634      1.8176 +- 0.122693        9      good
chromium@439641      1.82398 +- 0.0983583      6      good
chromium@439642      1.9105 +- 0.0540582       6      bad       <--
chromium@439643      1.8921 +- 0.0871679       9      bad
chromium@439644      1.89844 +- 0.0744101      9      bad
chromium@439646      1.90843 +- 0.0943553      9      bad
chromium@439658      1.9096 +- 0.102763        6      bad
chromium@439754      1.81787 +- 0.066893       6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests thread_times.key_mobile_sites_smooth

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8990641666934886624

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6435443437993984


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Possibly related to Olli's compiler refactoring?

Perf people, any idea why the bisect script isn't recursing into the ANGLE roll? It does this for some other regressions.

If there's repro steps we can look at this. CC'ing Yuly who works a lot with Android ANGLE.

Simon, can you look into why the bisect didn't descend into the ANGLE roll?

Yep, logged crbug.com/680993 to track that.

From the table in #6, looks to me like the bisect logic is wrong.
There were 2 big increases in time: chromium@439610 +0.31071 and chromium@439634 +0.18415, yet chromium@439642 is blamed, though it increases the time only by +0.08652. Also, one can see that in chromium@439754 the time is back to what it was in chromium@439634, before ANGLE roll.
Also, I understand that +- is the error range? Than I don't think that there was any regression at all. 1.32631 + 0.460989 is about the same as 1.9105 - 0.0540582.

This bug has an owner, but was in a state that our triage picked up. Marking as Assigned.

Simon can you respond to Yuly's comment in #11?

re: #c11

Yeah definitely looks like it got tripped up. Pinpoint should be better at handling cases like this where there are potentially multiple movements in the range. I'd guess there was a regression here:

chromium@439562      1.32274 +- 0.384379
chromium@439610      1.63345 +- 0.839638
chromium@439634      1.8176 +- 0.122693

And another that appeared slightly later but went away:

chromium@439641      1.82398 +- 0.0983583
chromium@439642      1.9105 +- 0.0540582
chromium@439658      1.9096 +- 0.102763
chromium@439754      1.81787 +- 0.066893

I can try rekicking this with a better metric as well, since it looks like only a single page regressed.

If you could do that and assign to the blamed new owner it would be appreciated, I'm not as familiar with the bisect tools.

Actually took a quick peak at another android bot during the same period, and there was exactly the same regression on the other android bots. Probably didn't get grouped with them since the 7v2's range is almost 400 commits (possibly the bot was down or the test was broken at the time?). Anyway, can just dupe against  crbug.com/677420 

Great, thank you Simon.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982620126833436160

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: android_nexus7_perf_bisect
  Benchmark    : thread_times.key_mobile_sites_smooth
  Metric       : thread_raster_cpu_time_per_frame/thread_raster_cpu_time_per_frame

Revision             Result                  N
chromium@439370      1.26863 +- 6.98612      30      good
chromium@439754      1.76956 +- 12.1214      30      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests thread_times.key_mobile_sites_smooth

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982620126833436160

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5322249316335616


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!