All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=714896

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgovXWtAoM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-x64-dual

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981385463220905968

=== Auto-CCing suspected CL author mlippautz@chromium.org ===

Hi mlippautz@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : mlippautz
  Commit : 6d89de7b9e5fc3a5f9d0fa2cef51864191d5ff5f
  Date   : Tue Apr 04 16:11:56 2017
  Subject: Reland of [heap] Refactor evacuation verifier

Bisect Details
  Configuration: win_x64_perf_bisect
  Benchmark    : v8.infinite_scroll_tbmv2
  Metric       : v8-gc-scavenger_max/flickr
  Change       : 14.39% | 4.11635714286 -> 4.7085

Revision                           Result                   N
chromium@461933                    4.11636 +- 1.57252       14      good
chromium@461952                    4.22036 +- 1.6885        14      good
chromium@461962                    4.30657 +- 1.60523       14      good
chromium@461962,v8@1769c7034b      4.87733 +- 0.104151      6       good
chromium@461962,v8@6d89de7b9e      4.7871 +- 0.32327        21      bad       <--
chromium@461962,v8@0bd9f1b8e6      4.82421 +- 0.401099      14      bad
chromium@461962,v8@d0e5df3e17      4.8275 +- 0.291066       14      bad
chromium@461963                    4.79886 +- 0.884614      14      bad
chromium@461964                    4.80236 +- 0.354318      14      bad
chromium@461965                    4.8351 +- 0.369304       21      bad
chromium@461967                    4.82781 +- 0.360989      21      bad
chromium@461971                    4.80993 +- 0.876516      14      bad
chromium@462008                    4.7085 +- 1.15462        14      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=flickr v8.infinite_scroll_tbmv2

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8981385463220905968

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5904975454011392


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981336461322574320

I disagree with the bisect, this looks like 1769c7034ba4203c6301d5db5e3081b47585af0d. vogelheim, can you take a look?

sullivan, any idea how the bisect got this so wrong?

Simon, any ideas what happened with the bisect?

#5: Thanks a lot! I was confused and under the same impression but kicked of another bisect before replying here. 

In the last compare step, when examining chromium@461962,v8@6d89de7b9e:

With:
chromium@461962                    4.30657 +- 1.60523       14      good

  "result": {
    "U": 16,
    "p": 0.035447892552460725,
    "significance": "NEED_MORE_DATA"
  },


And with:
chromium@461962,v8@6d89de7b9e      4.7871 +- 0.32327        21      bad       <--

  "result": {
    "U": 18,
    "p": 0.00944014954530159,
    "significance": "REJECT"
  },

Looks like the MWU test incorrectly classifies chromium@461962,v8@6d89de7b9e as being different from chromium@461962,v8@1769c7034b, and since chromium@461962 was a "good" revision, chromium@461962,v8@6d89de7b9e is declared the culprit. I don't see off hand anything it did specifically "wrong", just that the data here seemed perfectly suited to throw it off?

Ideally if the bisect had some validation, like it used to perform where it would revert the cl locally, and then verify that the regression disappeared, we'd avoid this kind of issue.


+dtu for ideas on how we might tweak MWU to better handle cases like this

=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : vogelheim
  Commit : 1769c7034ba4203c6301d5db5e3081b47585af0d
  Date   : Tue Apr 04 15:38:17 2017
  Subject: [parser/ast] Reduce memory consumption for AstConsString.

Bisect Details
  Configuration: win_x64_perf_bisect
  Benchmark    : v8.infinite_scroll_tbmv2
  Metric       : v8-gc-scavenger_max/flickr
  Change       : 16.34% | 4.18564285714 -> 4.86966666667

Revision                           Result                    N
chromium@461933                    4.18564 +- 1.58762        14      good
chromium@461952                    4.02667 +- 0.523495       6       good
chromium@461962                    4.0855 +- 0.631038        6       good
chromium@461962,v8@1769c7034b      4.85217 +- 0.318378       6       bad       <--
chromium@461962,v8@6d89de7b9e      4.76311 +- 0.781892       9       bad
chromium@461962,v8@0bd9f1b8e6      4.8115 +- 0.215554        6       bad
chromium@461962,v8@d0e5df3e17      4.8795 +- 0.227815        6       bad
chromium@461963                    4.8395 +- 0.270188        6       bad
chromium@461964                    5.02 +- 0.429632          6       bad
chromium@461965                    4.88933 +- 0.343542       6       bad
chromium@461967                    4.81417 +- 0.0993722      6       bad
chromium@461971                    4.89817 +- 0.242212       6       bad
chromium@462008                    4.86967 +- 0.41625        9       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=flickr v8.infinite_scroll_tbmv2

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8981336461322574320

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5066123344609280


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!