All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=663887

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg09uUsQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg07Kw_wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg09aUqQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgk9uZtgsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg06W5uAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg0_34tQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg0-TprQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgs-j6_QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg0_vOoQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg08v-swsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg04rqrwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg052_vAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgs_izpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgs5y5owoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg04qvqAkM


Bot(s) for this bug's original alert(s):

android-nexus5X
android-nexus6

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996436268068760512

=== Auto-CCing suspected CL author fdoray@chromium.org ===

Hi fdoray@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Revert of Enable SequencedWorkerPool to TaskScheduler redirection in testing config. (patchset #2 id:20001 of https://codereview.chromium.org/2446603002/ )
Author  : fdoray
Commit description:
  
Reason for revert:
This caused a 1.1% regression in system_health.memory_mobile. https://crbug.com/661520

Original issue's description:
> Enable SequencedWorkerPool to TaskScheduler redirection in testing config.
>
> With this CL, redirection of SequencedWorkerPools to TaskScheduler will
> be tested on bots.
>
> Before a similar CL https://codereview.chromium.org/2353973002/ landed,
> SequencedWorkerPool tasks didn't generate tracing events. With
> redirection to TaskScheduler enabled, the same tasks started to generate
> tracing events. That affected the battor.power_cases/cpu_time_percentage_max
> graph https://crbug.com/656629 and the CL was reverted. Now that
> SequencedWorkerPool tasks generate tracing events
> https://codereview.chromium.org/2429863002/, enabling redirection
> to TaskScheduler shouldn't affect any performance graph. Note that the
> battor.power_cases/cpu_time_percentage_max graph didn't go up when tracing
> was enabled in SequencedWorkerPools because of
> https://codereview.chromium.org/2420133002/ which reduced the CPU
> consumption of a task running in the blocking pool.
>
> BUG= 622400 
>
> Committed: https://crrev.com/9b3162c323b6529cbba17fb419129fea26b75257
> Cr-Commit-Position: refs/heads/master@{#427375}

TBR=gab@chromium.org,rkaplow@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 622400 , 661520

Review-Url: https://codereview.chromium.org/2488513003
Cr-Commit-Position: refs/heads/master@{#430616}
Commit  : 5a6dfbbcae66fe80acefd8e1f0d812cdc4551bb6
Date    : Tue Nov 08 15:33:28 2016


===== TESTED REVISIONS =====
Revision         Mean      Std Dev  N  Good?
chromium@430615  40776785  265971   5  good
chromium@430616  41716771  496604   5  bad    <--
chromium@430617  41777495  418680   5  bad
chromium@430618  41891107  197035   5  bad
chromium@430621  41720499  263179   5  bad

Bisect job ran on: android_nexus6_perf_bisect
Bug ID: 663887

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile_stress
Test Metric: memory:chrome:all_processes:reported_by_chrome:malloc:effective_size_avg/background/after_http_yandex_ru_touchsearch_text_science
Relative Change: 2.31%
Score: 99.5

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2736
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996436268068760512


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5886090987175936

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

FYI. Looks like you regressed other metrics on the revert.

¯\_(ツ)_/¯

@rschoen: how confident are we in these alerts? I wouldn't expect either the landing nor the revert of this CL to have changed anything significant memory wise..

Yeah, that's a very high-confidence bisect there. Even more so if there were alerts on the land and the revert.

You can try things out on the perf trybots if you'd like to experiment?

Ok thanks, are there tools that can highlight where the new memory chunks are coming from?

Yep! Documentation at https://chromium.googlesource.com/chromium/src/+/master/components/tracing/docs/memory_infra.md
You can look at the charts linked in #1--click "Trace" in the tooltip before/after the regression to get traces before/after.

Okay thanks, @fdoray to investigate (re-opening despite revert as we still don't know what's happening -- feel free to remove labels that affect the perf dashboards).

Roger that.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/53a4e4af0525c0112bc6b3d03c9f4d2534212830

commit 53a4e4af0525c0112bc6b3d03c9f4d2534212830
Author: gab <gab@chromium.org>
Date: Thu Dec 22 16:03:34 2016

Enable BrowserScheduler.RedirectSequencedWorkerPools experiment on trunk.

It previously caused a memory regression which might since have been
addressed, trying again...

BUG= 622400 , 661520,  663887 

Review-Url: https://codereview.chromium.org/2597493002
Cr-Commit-Position: refs/heads/master@{#440433}

[modify] https://crrev.com/53a4e4af0525c0112bc6b3d03c9f4d2534212830/testing/variations/fieldtrial_testing_config.json

Relanded this CL with a few precursor memory improvements and it appears to now be improving memory across the board :) : 

https://chromeperf.appspot.com/group_report?rev=440433 

(a few red deltas in there but they all seem to have associated bugs not related to this CL -- relaunched a few bisects to make sure).

In particular the regression in this very issue is now an clear improvement with this CL :) :
https://chromeperf.appspot.com/report?sid=0a1cdf666b39e00658e91f448195fe2ef20077202ed73f01d17c3b008fd9ba9c&rev=440436