All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=633537

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgxq_brQsM


Bot(s) for this bug's original alert(s):

chromium-rel-win10

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9005450679591115408

=== Auto-CCing suspected CL author hpayer@chromium.org ===

Hi hpayer@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [heap] Reland "Remove black pages and use black areas instead."
Author  : hpayer
Commit description:
  
BUG=chromium:630969, chromium:630386 

Review-Url: https://codereview.chromium.org/2186863005
Cr-Commit-Position: refs/heads/master@{#38195}
Commit  : 205457b1aa3811021f603f8a4201b5363507841d
Date    : Mon Aug 01 09:05:04 2016


===== TESTED REVISIONS =====
Revision                       Mean      Std Dev  N  Good?
chromium@408962                13692928  0.0      5  good
chromium@408962,v8@dc505196e6  13692928  0.0      5  good
chromium@408962,v8@205457b1aa  14741504  0.0      5  bad    <--
chromium@408962,v8@e9887b6aad  14741504  0.0      5  bad
chromium@408963                14741504  0.0      5  bad
chromium@408964                14741504  0.0      5  bad
chromium@408966                14741504  0.0      5  bad
chromium@408969                14741504  0.0      5  bad
chromium@408975                14741504  0.0      5  bad
chromium@408987                14741504  0.0      5  bad

Bisect job ran on: winx64_10_perf_bisect
Bug ID: 633537

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop
Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:v8:effective_size_avg/load_search_yahoo
Relative Change: 7.66%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_10_perf_bisect/builds/634
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005450679591115408


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5851256828985344

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/d2cf6e2956dc4523639ead25569443a07a4e113d

commit d2cf6e2956dc4523639ead25569443a07a4e113d
Author: hpayer <hpayer@chromium.org>
Date: Wed Aug 03 18:08:54 2016

[heap] Temporarily use old live object iterator to investigate Win 10 memory regression.

BUG= chromium:633537 

Review-Url: https://codereview.chromium.org/2205373002
Cr-Commit-Position: refs/heads/master@{#38311}

[modify] https://crrev.com/d2cf6e2956dc4523639ead25569443a07a4e113d/src/heap/mark-compact-inl.h
[modify] https://crrev.com/d2cf6e2956dc4523639ead25569443a07a4e113d/test/cctest/heap/test-heap.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/62f2e7e8b01404a4253290533d98bb454e820199

commit 62f2e7e8b01404a4253290533d98bb454e820199
Author: hpayer <hpayer@chromium.org>
Date: Thu Aug 04 08:22:36 2016

Revert "[heap] Temporarily use old live object iterator to investigate Win 10 memory regression."

This reverts commit d2cf6e2956dc4523639ead25569443a07a4e113d.

BUG= chromium:633537 

Review-Url: https://codereview.chromium.org/2207383002
Cr-Commit-Position: refs/heads/master@{#38328}

[modify] https://crrev.com/62f2e7e8b01404a4253290533d98bb454e820199/src/heap/mark-compact-inl.h
[modify] https://crrev.com/62f2e7e8b01404a4253290533d98bb454e820199/test/cctest/heap/test-heap.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/04fda1f686405a6ec774470be011806cc23a02b0

commit 04fda1f686405a6ec774470be011806cc23a02b0
Author: hpayer <hpayer@chromium.org>
Date: Fri Aug 05 10:55:56 2016

[heap] Temporarily do not use size to iterate live object to investigate Win 10 memory regression.

BUG= chromium:633537 

Review-Url: https://codereview.chromium.org/2211893002
Cr-Commit-Position: refs/heads/master@{#38369}

[modify] https://crrev.com/04fda1f686405a6ec774470be011806cc23a02b0/src/heap/mark-compact-inl.h
[modify] https://crrev.com/04fda1f686405a6ec774470be011806cc23a02b0/test/cctest/heap/test-heap.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/2b10616b778e627f221949eff73e4f67100a78d3

commit 2b10616b778e627f221949eff73e4f67100a78d3
Author: hpayer <hpayer@chromium.org>
Date: Mon Aug 08 09:18:17 2016

[heap] Temporarily use old live object iterator to investigate Win 10 memory regression.

BUG= chromium:633537 

Review-Url: https://codereview.chromium.org/2224823003
Cr-Commit-Position: refs/heads/master@{#38423}

[modify] https://crrev.com/2b10616b778e627f221949eff73e4f67100a78d3/src/heap/mark-compact-inl.h

 Issue 633225  has been merged into this issue.

The Win 10 bot seems to be flaky on that metric. Our investigation did not find anything. Closing this one for now.

From the traces:

BEFORE regression: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_38-2016-08-01_10-01-47-12867.html
AFTER regression: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_39-2016-08-01_12-36-27-94180.html

it seems that the increase is due to +1.0 MiB in v8/isolate_*/heap_spaces/code_space.

hpayer: What kind of investigation have you done? Although the number is bi-modal over the whole revision range, the regression reproduces perfectly (#3). I think it would be worth investigating what exactly is causing the isolate to use an extra megabyte for code space.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003617938510994048

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [heap] Reland "Remove black pages and use black areas instead."
Author  : hpayer
Commit description:
  
BUG=chromium:630969, chromium:630386 

Review-Url: https://codereview.chromium.org/2186863005
Cr-Commit-Position: refs/heads/master@{#38195}
Commit  : 205457b1aa3811021f603f8a4201b5363507841d
Date    : Mon Aug 01 09:05:04 2016


===== TESTED REVISIONS =====
Revision                       Mean      Std Dev  N  Good?
chromium@408962                13902643  468937   5  good
chromium@408962,v8@dc505196e6  13692928  0.0      5  good
chromium@408962,v8@205457b1aa  14741504  0.0      5  bad    <--
chromium@408962,v8@e9887b6aad  14741504  0.0      5  bad
chromium@408963                14741504  0.0      5  bad
chromium@408964                14741504  0.0      5  bad
chromium@408966                14741504  0.0      5  bad
chromium@408969                14741504  0.0      5  bad
chromium@408975                14741504  0.0      5  bad
chromium@408987                14741504  0.0      5  bad

Bisect job ran on: winx64_10_perf_bisect
Bug ID: 633537

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop
Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:v8:effective_size_avg/load_search_yahoo
Relative Change: 6.03%
Score: 95.0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_10_perf_bisect/builds/674
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003617938510994048


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5254267095482368

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

hpayer: The regression still reproduces perfectly. Could you please look into why there is +1.0 MiB increase in v8/isolate_*/heap_spaces/code_space?

We bisected it down (#4 to #7) to the change of the iterator. It perfectly bisects but it does not explain why code space would use one extra page. The iterator is returning before and after the change exactly the same objects, so behavior should be unchanged. We do not have a Win 10 machine for local investigation and unfortunately it does not reproduce on any other machines. 

Temporarily declaring bankruptcy on the *desktop* system health benchmark.
The number of alerts became unmanageable and the overall process needs to be improved to make it sustainable.
The alerts have been turned off and I'm archiving the outstanding regressions.
Note: this is just about desktop, the mobile system health stays up.