All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=660245

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg_dTSowoM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-dual

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997602141356819280

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@427333  330.591  3.37121  18  good
chromium@427402  330.491  5.97084  18  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases_extra
Test Metric: seek/garden2_10s.webm_seek_warm
Relative Change: 1.26%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7027
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997602141356819280


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5787920470900736

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996608974716725680

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7039
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

@robertocn, the build seems to be failing during execution of a bisect python script. Do you know who owns? How to proceed?

INFO:root:Running ['git.bat', 'rev-parse', '--verify', 'd6020a7ef29d3f7fb77d16a1cd6b32c7b2f09dc2^{commit}']
fatal: Needed a single revision

... (much later)

INFO:remote_run:Command ['C:\\b\\depot_tools\\python276_bin\\python.exe', 'C:\\b\\.remote_run_cipd\\recipes.py', '--verbose', 'remote', '--repository', 'https://chromium.googlesource.com/chromium/tools/build.git', '--revision', 'origin/master', '--workdir', 'C:\\b\\rr\\tmpta_1zo\\rw', '--', '--verbose', 'run', '--properties-file', 'C:\\b\\rr\\tmpta_1zo\\remote_run_properties.json', '--workdir', 'C:\\b\\rr\\tmpta_1zo\\w', '--output-result-json', 'C:\\b\\rr\\tmpta_1zo\\recipe_result.json', u'bisection/desktop_bisect'] finished with exit code 1.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8992796572614258064

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8992796564338316608

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@427333  339.02   14.2401  21  good
chromium@427402  338.687  23.0057  21  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.garden2.10s.webm media.tough_video_cases_extra
Test Metric: seek/video.html?src_garden2_10s.webm
Relative Change: 0.10%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7078
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8992796572614258064


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4933104509124608

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: No values were found while testing the reference range.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean  Std Dev  N  Good?
chromium@427333  N/A   N/A      0  good
chromium@427402  N/A   N/A      0  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=garden2.10s.webm.seek.warm media.tough_video_cases_extra
Test Metric: seek/garden2_10s.webm_seek_warm
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7079
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8992796564338316608


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6389585095950336

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Ping robertocn@ - any advice on how to progress here?

+simonhatch, perezju: output from a test run. Looks like --story-filter is filtering out all tests:

https://uberchromegw.corp.google.com/i/tryserver.chromium.perf/builders/win_perf_bisect/builds/7079/steps/Gathering%20reference%20values.Performance%20Test%204%20of%206/logs/Captured%20Output

C:\b\depot_tools\python276_bin\python.exe C:\b\rr\tmpn2uomk\rw\checkout\scripts\slave\runisolatedtest.py --test_name "" --builder_name win_perf_bisect --checkout_dir C:\b\c\b\win_perf_bisect C:\b\c\b\win_perf_bisect\src\tools\perf\run_benchmark -- C:\b\depot_tools\python276_bin\python.exe C:\b\c\b\win_perf_bisect\src\tools\perf\run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=garden2.10s.webm.seek.warm media.tough_video_cases_extra --output-dir=C:\b\rr\tmpn2uomk\w\bisect_results\f5928602-31bf-4ac4-b5fc-42be6abffb18

C:\b\depot_tools\python276_bin\python.exe C:\b\c\b\win_perf_bisect\src\tools\perf\run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=garden2.10s.webm.seek.warm media.tough_video_cases_extra --output-dir=C:\b\rr\tmpn2uomk\w\bisect_results\f5928602-31bf-4ac4-b5fc-42be6abffb18
INFO:root:No usb serial port discovered. Available ones are: [('COM1', 'Communications Port (COM1)', 'ACPI\\PNP0501\\37000FC1'), ('COM2', 'Communications Port (COM2)', 'ACPI\\PNP0501\\37000FC2')]
INFO:root:Chose browser: PossibleDesktopBrowser(type=release, executable=C:\b\c\b\win_perf_bisect\src\out\Release\chrome.exe, flash=C:\b\c\b\win_perf_bisect\src\third_party\adobe\flash\binaries\ppapi\win_x64\pepflashplayer.dll)
INFO:root:Chose browser: PossibleDesktopBrowser(type=release, executable=C:\b\c\b\win_perf_bisect\src\out\Release\chrome.exe, flash=C:\b\c\b\win_perf_bisect\src\third_party\adobe\flash\binaries\ppapi\win_x64\pepflashplayer.dll)
[  PASSED  ] 0 tests.



I'll try re-running without --story-filter in the short term.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8992740621458739648

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8992740616489662416

That story filter doesn't seem valid, how was it picked? When I go to the dashboard it automatically populates the story-filter field with "garden2.10s.webm.seek.warm"

But digging through the log it seems like that's not a valid option, most of the stories look something more like: video.html?src=tulip2.ogg&type=audio

That sounds like a bug on how we build the story filter. Could you file a catapult bug on me to get it fixed? I'll have a look at it tomorrow.

Done!

https://github.com/catapult-project/catapult/issues/3103

Thanks for taking a look! Any workaround for me in the meantime?

You can try running with nothing in the story-filter field until this is fixed for any other bisects. There's a new bisect running from sullivan (#c13) for this issue so just wait on the result.

+ned as FYI

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@427333  329.456  12.4464  21  good
chromium@427402  330.584  16.8538  21  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases_extra
Test Metric: seek/garden2_10s.webm_seek_warm
Relative Change: 0.34%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7081
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8992740616489662416


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5273684275625984

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: 
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@427333  332.22   34.4969  21  good
chromium@427368  329.615  12.6454  21  unknown
chromium@427402  328.255  7.1788   21  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases_extra
Test Metric: seek/video.html?src_garden2_10s.webm
Relative Change: 1.19%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7080
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8992740621458739648


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5907005254926336

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8992684370062743152

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8992684257823117056

Perhaps we have a 2 part root cause.

Started a new bisect between 427333 and 427368
https://chromeperf.appspot.com/buildbucket_job_status/8992684370062743152

Started a second bisect between 427368 and 427402
https://chromeperf.appspot.com/buildbucket_job_status/8992684257823117056

(Both with story filter cleared)

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7083
Failure reason: the build has failed.
Additional errors:
Bisect aborted early for lack of confidence.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7084
Failure reason: the build has failed.
Additional errors:
Bisect aborted early for lack of confidence.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Now restarting the bisects described in Comment 26

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991437600698618176

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991437580738255904

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@427368  331.522  15.8131  21  good
chromium@427402  331.673  20.8986  21  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.garden2.10s.webm media.tough_video_cases_extra
Test Metric: seek/video.html?src_garden2_10s.webm
Relative Change: 0.05%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7092
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991437580738255904


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5836217650774016

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@427333  338.202  15.252   21  good
chromium@427368  338.814  19.9861  21  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases_extra
Test Metric: seek/video.html?src_garden2_10s.webm
Relative Change: 0.18%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7091
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991437600698618176


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5341487884140544

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

All of the "means" in these recent runs seem to be elevated to match the post-regression values. I'll do another bisect now with the start revision pulled further back. 

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991380717685909824

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@426991  329.905  14.9684  21  good
chromium@427402  329.62   15.7124  21  bad

Bisect job ran on: win_perf_bisect
Bug ID: 660245

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases_extra
Test Metric: seek/video.html?src_garden2_10s.webm
Relative Change: 0.09%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7093
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991380717685909824


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6357040954343424

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Mean value is still in the post-regression range even after moving the start revision way back into the "good" range. I notice that all of the tests in this alert are on win7 bots. Perhaps an OS update across all bots hurt seeking performance irrespective of chrome. 

I notice a similar struggle with win7 (and bisect reproduce-ability) in  Issue 660232 . Standing by for advice on that bug. Not sure if there's more to do here. 

+sullivan

https://chromeperf.appspot.com/report?sid=5599dd918a4cbfd560f7321e649ba9131465b71eb5cac7ab880b7038a590bc51&start_rev=423533&end_rev=441641

This really looks like a false alarm, the ref build moved by the same amount, albeit at just slightly different range. Adding sullivan@ for any ideas why those might not line up.

It looks like the ref build "caught up" in the next run, so maybe:

Run 1:
ref build runs, no regression
<something happens on bot>
ToT test runs, regression

Run 2:
ref build runs, regression
ToT test runs, regression

This only happened on one machine, so likely a bot issue.

Ah cool thanks, then chcunningham@ this can probably be safely WontFix'd.

Thanks all. I didn't realize we could access the ref this way! I used to find the ref overlayed on the non-ref graph. This isn't happening for any of the graphs in the original alert. Did something change there?

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8984378575737039136

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: win_perf_bisect
  Benchmark    : media.tough_video_cases_extra
  Metric       : seek/video.html?src_garden2_10s.webm

Revision             Result                  N
chromium@427333      332.265 +- 22.0431      21      good
chromium@427402      330.413 +- 13.6833      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.garden2.10s.webm media.tough_video_cases_extra

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984378575737039136

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4933104509124608


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8984276568749255664

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: win_perf_bisect
  Benchmark    : media.tough_video_cases_extra
  Metric       : seek/garden2_10s.webm_seek_warm

Revision             Result                  N
chromium@427333      332.805 +- 20.1819      21      good
chromium@427402      333.094 +- 21.6708      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases_extra

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984276568749255664

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4933104509124608


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8983623006921043744

=== Auto-CCing suspected CL author xunjieli@chromium.org ===

Hi xunjieli@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : xunjieli
  Commit : 28a187760807a40049084627465be82a5d1e3118
  Date   : Tue Oct 25 15:33:14 2016
  Subject: Switch to use net::FilterSourceStream from net::Filter

Bisect Details
  Configuration: win_perf_bisect
  Benchmark    : media.tough_video_cases_extra
  Metric       : seek/video.html?src_garden2_10s.webm
  Change       : 1.63% | 333.585833333 -> 328.16

Revision             Result                  N
chromium@427333      333.586 +- 7.96768      6       good
chromium@427351      331.417 +- 7.40003      14      good
chromium@427356      333.177 +- 12.3449      14      good
chromium@427357      333.78 +- 27.2016       14      good
chromium@427358      329.127 +- 10.3692      14      bad       <--
chromium@427359      329.658 +- 10.2546      14      bad
chromium@427368      329.095 +- 11.7154      14      bad
chromium@427402      328.16 +- 3.71201       6       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.garden2.10s.webm media.tough_video_cases_extra

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8983623006921043744

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4933104509124608


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

I am not quite sure why I am cc-ed or why my CL is in the range. Let me know if there's anything I can do.

Sorry - not sure why additional bisects were kicked off months after the WontFix. This was determined to be noise back in Jan. Please ignore

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982632168566340000

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982624478779958368

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982619791756193792

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982580255582485024