All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=669129

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgu8iWsAsM


Bot(s) for this bug's original alert(s):

android-nexus5X

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994723882862827472

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994723803721403024

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean      Std Dev   N   Good?
chromium@432945  10701103  10995124  27  good
chromium@432982  10166993  3991732   27  bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 669129

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=search.portal.google system_health.memory_mobile
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/search_portal/search_portal_google
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/914
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994723882862827472


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5897341469458432

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994715193178874000

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean      Std Dev  N   Good?
chromium@432161  14738811  5186793  27  good
chromium@432195  15073015  5005354  27  bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 669129

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.tools.drive system_health.memory_mobile
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/load_tools/load_tools_drive
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/918
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994715193178874000


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5888344049844224

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Simon - this metric is bimodal so bisecting isn't going to work. What can I do to help?

=== Auto-CCing suspected CL author ricea@chromium.org ===

Hi ricea@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : WritableStream: minor bug fixes
Author  : ricea
Commit description:
  
* Constructing a WritableStream with a highWaterMark of NaN resulting in
  a TypeError. It should have been a RangeError. Fixed.
* Constructing a WritableStreamDefaultController should fail if the
  WritableStream already had a controller. It did not. Fixed.

BUG= 658144 

Review-Url: https://codereview.chromium.org/2499943002
Cr-Commit-Position: refs/heads/master@{#432131}
Commit  : aff3d02ed430c9039ec2cc8b4f36f28687e2d6ce
Date    : Tue Nov 15 06:29:56 2016


===== TESTED REVISIONS =====
Revision         Mean      Std Dev  N   Good?
chromium@432130  9784960   133795   8   good
chromium@432131  11045683  340308   5   bad    <--
chromium@432132  12042035  320230   5   bad
chromium@432134  11860992  480211   5   bad
chromium@432137  11716403  689921   5   bad
chromium@432144  11709030  190374   5   bad
chromium@432157  11553792  216721   5   bad
chromium@432184  11362509  155016   5   bad
chromium@432237  11117978  244895   5   bad
chromium@432344  10252117  1268443  12  bad
chromium@432556  10229632  1046259  8   bad
chromium@432982  9913958   40278.5  5   bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 669129

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.youtube system_health.memory_mobile
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/browse_media/browse_media_youtube
Relative Change: 1.32%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/915
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994723803721403024


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5841039280046080

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

If the bot runs without --enable-experimental-web-features then it is unbelievably unlikely that https://codereview.chromium.org/2499943002 could have caused the rejection. If it runs with --enable-experimental-web-features then it is merely incredibly unlikely. The change is minor, the changed code isn't executed by the test, and if it has any impact on memory usage at all it should make it smaller.

Re-assigning to picksi@.

+hector
Can we use our local dumb-bisect to find the cause?

The bimodality that occurs just before the consistent regression (https://chromeperf.appspot.com/report?sid=0e12d811edb06282271c5b31af73b22415caebaeb78fac35ab59e8a0dfb924c2&rev=432982) suggests to me that we've had a couple of small changes that have cumulatively pushed something across a threshold (i.e. GC, or cache-clearing, or pool-allocation).

Ok something definitely happened to the N5X.
This smells extremely similar to  Issue 667794 .
In both cases something happened between Nov 11 and Nov 16.

I'm talking about this (top_10_mobile_stress):
https://chromeperf.appspot.com/report?sid=e7e8cef30400d21721427ef01855c1bc514b24f63e988910c6c90726b3e4c93d&start_rev=430554&end_rev=434364&rev=432975

and this (system_health.memory_mobile)
https://chromeperf.appspot.com/report?sid=b9eca0df1a1820017fe72692f2d1a533e1822a90c5a702e6e612e5319c872a07&rev=432165

In both cases bimodaility started around Nov 11 and ended around Nov 16, settling up to higher values.

Let's have a sync all together this afternoon on this.

Hey Primiano, just reading this now. Did you find anything in your sync?

https://github.com/catapult-project/catapult/issues/3050 has the full details.
The results on the dashboard seem out of order, so there is a regression somewhere but we have no way to tell when it happened :/

Closing this as the regressions were caused by a recipe change (testing Chrome 32 rather than 64 bits) and numbers returned to their previous baseline after reverting that recipe change.

Let's follow up in go/catabug/3050 if there is a need to revise the builds being tested on this bot.

 Issue 669448  has been merged into this issue.

 Issue 667794  has been merged into this issue.