All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=654226

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9b73qQkM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-dual

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8999350450673351632

=== Auto-CCing suspected CL author proberge@chromium.org ===

Hi proberge@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Send a TrackedPreference incident when registry pref validation fails.
Author  : proberge
Commit description:
  
Note that the new incident types (BYPASS_CLEARED and BYPASS_CHANGED) are only
used when external (registry) validation fails and existing
(secure_preferences-based) validation succeeds.

BUG=624858

Committed: https://crrev.com/1ba5bcd1a74380bb398eeab3787ede5620b5130c
Review-Url: https://codereview.chromium.org/2384213002
Cr-Original-Commit-Position: refs/heads/master@{#422545}
Cr-Commit-Position: refs/heads/master@{#423285}
Commit  : c34dbf9b2c3434449a30384ce215a40c45fdae05
Date    : Wed Oct 05 20:53:18 2016


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@423149  107770  1092.78  5  good
chromium@423221  107467  707.322  5  good
chromium@423257  108289  830.487  5  good
chromium@423275  108218  718.655  5  good
chromium@423284  107424  1023.72  5  good
chromium@423285  124400  1091.24  5  bad    <--
chromium@423286  123480  536.648  5  bad
chromium@423288  124343  931.734  5  bad
chromium@423292  123772  499.94   5  bad

Bisect job ran on: win_perf_bisect
Bug ID: 654226

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests storage.indexeddb_endure
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 14.85%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6992
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8999350450673351632


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5883690758963200

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

From https://chromeperf.appspot.com/group_report?bug_id=654226 it's clear that my change (https://codereview.chromium.org/2384213002) is responsible - the regression temporarily showed up Oct 3 and 4, then my change was reverted and the regression stopped. After the rollforward on Oct 5 the regression appeared again.

My change doesn't directly touch memory usage; I suspect that my change, which enables a new incident trigger, is incorrectly triggering a Tracked Preference incident (which somehow eats 15 MB (!)). 

There's either an issue with the logic in https://codereview.chromium.org/2204943002/ or an issue with how the Windows build bot manages the Windows registry and profiles between runs. I'll take a look.

Only seems to be happening on chromium-rel-win7-dual:
https://chromeperf.appspot.com/report?sid=6c8b5fe1e6ec6070011ee29efe5a4a502e27389f5fdac8a12e528116f68e9afb

The test being storage.indexeddb_endure is a red herring. 

memory:chrome:browser_process:reported_by_os:peak_resident_size_avg has the same regression:
https://chromeperf.appspot.com/report?sid=c7ae76421eb57a798b8d36b847f252402f873280af777174a654e21a13a3a7c1&start_rev=421996&end_rev=424402

To expand on comment #5, the regression isn't seen on win7-x64-dual; only on win7-dual.

https://chromeperf.appspot.com/report?sid=c2cfab238c3ffb5dbdececb9b67fd0c12607395c7ab12ed0d4a00372a2a5a045&start_rev=421996&end_rev=424402

Yet another graph:
https://chromeperf.appspot.com/report?sid=72a73503f6ddb1c6201f62ea2e5fbb2d7298a430a38210f02a66596d2139a769&start_rev=421996&end_rev=424402

Some of system_health.memory_desktop's subtests are showing a lower peak_resident_size_avg. The "load" tests have the regression while the "browse" tests have a comparable improvement.

I'm not sure what this means. Maybe my change caused some memory allocation to happen earlier in the Chrome startup. Still don't know why this only occurs on 32-bit Win7.

Tried a benchmark with a patch to make Chrome crash if an TrackedPref incident is seen.

https://codereview.chromium.org/2415783003/ (patch set 3)

Unfortunately, memory measurements failed:
http://storage.googleapis.com/chromium-telemetry/html-results/results-2016-10-14_09-29-12

What's next? What's the plan to close this out?

This dropped off of my radar. As it only happens on one bot (32-bit win7) and the memory use appears to have moved instead of being added (see #8), I'm closing this as WontFix.

Here's some (Stable, Beta) graphs that suggest the memory usage increase doesn't impact Chrome users:
https://uma.googleplex.com/p/chrome/variations/?sid=92404693b474a47de434d1a9fdfe49e2
https://uma.googleplex.com/p/chrome/variations/?sid=b18a94509b2cc4369ac0682ddfa8fdbd