All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=780890

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=5a5392c1f158874eb4665b12046489ed4a1bdf5c6ed75bb6f13620136399f4b9


Bot(s) for this bug's original alert(s):

chromium-rel-win7-gpu-nvidia

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8964017611673504192

=== Auto-CCing suspected CL author jbroman@chromium.org ===

Hi jbroman@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Jeremy Roman
  Commit : 863386d2cb0b88e41d8f720227930b9b1ad071cc
  Date   : Tue Oct 31 19:25:38 2017
  Subject: [c++] Permit heterogeneous container lookup and use it in base::CommandLine.

Bisect Details
  Configuration: winx64nvidia_perf_bisect
  Benchmark    : media_perftests
  Metric       : sinc_resampler_convolve/optimized_unaligned
  Change       : 1.46% | 130371.448881 -> 132279.91216

Revision             Result                 N
chromium@512503      130371 +- 1324.82      6      good
chromium@512801      130479 +- 1245.11      6      good
chromium@512876      130182 +- 1460.52      6      good
chromium@512895      130084 +- 895.299      6      good
chromium@512904      130448 +- 1037.52      9      good
chromium@512905      130540 +- 675.038      9      good
chromium@512906      129610 +- 23252.4      9      bad       <--
chromium@512907      132422 +- 810.408      6      bad
chromium@512909      132393 +- 781.545      6      bad
chromium@512913      132346 +- 713.862      6      bad
chromium@512950      132228 +- 629.109      6      bad
chromium@513098      132280 +- 1075.01      6      bad

To Run This Test
  .\src\out\Release_x64\media_perftests.exe --single-process-tests

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8964017611673504192


For feedback, file a bug with component Speed>Bisection

 Issue 780903  has been merged into this issue.

Looks like bisect is confused and the culprit is actually https://chromium.googlesource.com/chromium/src/+/7b88e3ebd719f7d5c8d8454b7fdd365b8b9a9adf 

Simon, could you please take a look?

Bot doesn't repro same values as the waterfall, first thing that stands out is that the test seems to be getting run with "--single-process-tests" whereas the waterfall version doesn't. Could this make a difference?

Oh wow, if the waterfall isn't running with --single-process-tests, then it could mean that it's running multiple tests are the same time, which would cause things to be really flaky. Are you sure the waterfall version isn't doing that?

+Dale is expert on media_perftests

perftests should always be run as --single-process-tests, otherwise performance will be non-deterministic if they're not running with that on the waterfall that is a bug. Simply adding tests will change paired orderings and thus results.

Kinda looks like the waterfall isn't run with that:

['.\\media_perftests.exe', '--verbose', '--test-launcher-print-test-stdio=always']

https://chromium-swarm.appspot.com/task?id=398da3bb9c093610&refresh=10&show_raw=1

Re-titling and assigning to myself for investigation.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f54ed709bc183da3994cd7fc783ac80d6b87250e

commit f54ed709bc183da3994cd7fc783ac80d6b87250e
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Wed Nov 15 00:41:37 2017

Make gtest perf test isolated and deterministic.

Also boy scout rule: prevent misspellings by putting flags
used multiple times into variables.

Bug:  780890 
Change-Id: I6ab20a95eb8699216fc1adc9097561e53a99ff71
Reviewed-on: https://chromium-review.googlesource.com/761402
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516523}
[modify] https://crrev.com/f54ed709bc183da3994cd7fc783ac80d6b87250e/testing/scripts/run_gtest_perf_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b7ffb463b11ff16a6af17b216701bc5788521b06

commit b7ffb463b11ff16a6af17b216701bc5788521b06
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Wed Nov 15 18:14:54 2017

Revert "Make gtest perf test isolated and deterministic."

This reverts commit f54ed709bc183da3994cd7fc783ac80d6b87250e.

Reason for revert: caused crbug/785291

Original change's description:
> Make gtest perf test isolated and deterministic.
> 
> Also boy scout rule: prevent misspellings by putting flags
> used multiple times into variables.
> 
> Bug:  780890 
> Change-Id: I6ab20a95eb8699216fc1adc9097561e53a99ff71
> Reviewed-on: https://chromium-review.googlesource.com/761402
> Commit-Queue: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Emily Hanley <eyaich@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#516523}

TBR=dalecurtis@chromium.org,dpranke@chromium.org,kbr@chromium.org,crouleau@chromium.org,eyaich@chromium.org

Change-Id: I68ddb7ec3fa9687170dcab2e43af39197cd51af2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  780890 
Reviewed-on: https://chromium-review.googlesource.com/772190
Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
Commit-Queue: Caleb Rouleau <crouleau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516746}
[modify] https://crrev.com/b7ffb463b11ff16a6af17b216701bc5788521b06/testing/scripts/run_gtest_perf_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3d7c42fb6fc47f96f70f0a7d256a280522ed20ba

commit 3d7c42fb6fc47f96f70f0a7d256a280522ed20ba
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Thu Nov 16 01:34:17 2017

[media_perftests] use --single-process-tests flag.

https://chromium-review.googlesource.com/c/chromium/src/+/761402
was reverted because it caused timeouts ( bug 785291 ). Better to
start small as kbr@ recommended in the initial CL :(

Bug:  780890 
Change-Id: I073e6f2ebc912dfdd897f404d69be1fc5a17f386
Reviewed-on: https://chromium-review.googlesource.com/773399
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516944}
[modify] https://crrev.com/3d7c42fb6fc47f96f70f0a7d256a280522ed20ba/testing/buildbot/gn_isolate_map.pyl

(infra bug triager here, triaging infra bugs)