All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=661932

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo8KJzgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg4-zepwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg4-mxvAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo535kAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo6P7swsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg46CAtAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg46mPuAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw5eYhAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw-e-wAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo7ni6AoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw4XRmgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg49zArAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw6-a5wsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo9elpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg45rWuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg49r0oQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg48Cm8AoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw6_lnwgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg49PPpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo4nj7woM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw5OGtggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw6aTgwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo57WtwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo9ruuwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw_6DlwgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw_2zrgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw92f6AsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo5PmuAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg45irpwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw6_l7wsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo72DngoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw_7ZxgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw8-94woM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg466uvQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw9uG_QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw-e-gAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo6ir4ggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo5nStAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo6bz7goM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo5a-igoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgw-eq7gsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgo_X4pwoM


Bot(s) for this bug's original alert(s):

android-nexus5
android-nexus6
android-nexus7v2
chromium-rel-win7-dual
chromium-rel-win7-gpu-ati
chromium-rel-win7-gpu-intel
chromium-rel-win7-gpu-nvidia
chromium-rel-win7-x64-dual
chromium-rel-win8-dual

Bisect wasn't automatically kicked off, running manually.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996912552849894464

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7036
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996910094477170368

Test failed. Trying on a different bot with a smaller revision range.

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_x64_perf_bisect/builds/1527
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994182618795757200

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994182613622036672

=== PERF REGRESSION ===


=== Auto-CCing suspected CL author ksakamoto@chromium.org ===

Hi ksakamoto@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [loading metric] filter out events during cache warming
Author  : ksakamoto
Commit description:
  
The regexp in prepareTelemetryInternalEventPredicate did not match
the titles of events logged by cache_temperature.py.

Note for perf sheriffs: Improvements are expected in page_cycler_v2
benchmarks.

BUG=catapult:#2962

Review-Url: https://codereview.chromium.org/2467763002
Commit  : 0d3575d73a471a53e1d1d363a349ad4536f293fc
Date    : Tue Nov 01 12:17:02 2016


===== TESTED REVISIONS =====
Revision                             Mean     Std Dev  N  Good?
chromium@428991                      1484.63  288.429  8  good
chromium@428992                      1493.99  431.469  8  good
chromium@428992,catapult@0d3575d73a  2086.82  163.618  5  bad    <--
chromium@428993                      1987.96  528.198  8  bad
chromium@428995                      2023.26  609.306  8  bad
chromium@428998                      2086.15  135.821  5  bad
chromium@429004                      2068.3   40.8791  5  bad
chromium@429017                      2035.88  121.821  5  bad
chromium@429043                      2050.37  112.911  5  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 661932

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.nationalgeographic.com. page_cycler_v2.basic_oopif
Test Metric: timeToFirstContentfulPaint_avg/pcv1-warm/http___www.nationalgeographic.com_
Relative Change: 38.11%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3530
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994182618795757200


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5839486607425536

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [loading metric] filter out events during cache warming
Author  : ksakamoto
Commit description:
  
The regexp in prepareTelemetryInternalEventPredicate did not match
the titles of events logged by cache_temperature.py.

Note for perf sheriffs: Improvements are expected in page_cycler_v2
benchmarks.

BUG=catapult:#2962

Review-Url: https://codereview.chromium.org/2467763002
Commit  : 0d3575d73a471a53e1d1d363a349ad4536f293fc
Date    : Tue Nov 01 12:17:02 2016


===== TESTED REVISIONS =====
Revision                             Mean     Std Dev  N  Good?
chromium@428975                      808.718  307.15   8  good
chromium@428985                      833.987  347.921  8  good
chromium@428990                      817.221  306.806  8  good
chromium@428992                      833.63   194.896  8  good
chromium@428992,catapult@0d3575d73a  1799.2   661.554  5  bad    <--
chromium@428993                      1794.18  508.232  5  bad
chromium@428994                      1833.59  483.035  5  bad
chromium@429012                      1827.48  550.834  5  bad

Bisect job ran on: android_nexus6_perf_bisect
Bug ID: 661932

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.nationalgeographic.com. page_cycler_v2.basic_oopif
Test Metric: timeToFirstContentfulPaint_avg/pcv1-warm/http___www.nationalgeographic.com_
Relative Change: 125.97%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2797
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994182613622036672


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6418317068206080

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

This is expected. My CL fixed a bug that PCv2 collected metrics from cache-warming page loads too.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8984378730883946656

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: win_perf_bisect
  Benchmark    : page_cycler_v2.typical_25
  Metric       : timeToFirstContentfulPaint_avg/pcv1-warm/http___www.fifa.com_


To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='http...www\.fifa\.com.' page_cycler_v2.typical_25

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984378730883946656

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4603053158694912


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!