All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=667794

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgm5SauwkM


Bot(s) for this bug's original alert(s):

android-nexus5X

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995276196965657232

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@432960  9129567  4083514  27  good
chromium@432975  9416287  4070456  27  bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 667794

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.m.youtube.com.results.q.science memory.top_10_mobile_stress
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/background/after_http_m_youtube_com_results_q_science
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/902
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995276196965657232


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5209925746163712

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995084025620641792

perezju@ I triggered another bisect but looking at the graph the regression seems to be very noisy over time. How do you think we should proceed?

===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@432940  9227567  4095375  27  good
chromium@432985  9418411  4129765  27  bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 667794

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.m.youtube.com.results.q.science memory.top_10_mobile_stress
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/background/after_http_m_youtube_com_results_q_science
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/906
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995084025620641792


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5882888233418752

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Note this is on the _stress version of the benchmark, that runs without closing the browser. Also the delta in the bisect is ~200KB (running only a single story), but on the graphs it's ~5MB, so could be a leak? Or maybe GC is firing less often?

I'll try a bisect without the story filter, let's see if that gives us something.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995029812719357008

=== Auto-CCing suspected CL author yusukes@chromium.org ===

Hi yusukes@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : arc: enable use_new_wrapper_types for video_accelerator.mojom
Author  : yusukes
Commit description:
  
BUG= 662510 
BUG= 624136 
TEST=try

Review-Url: https://codereview.chromium.org/2505733003
Cr-Commit-Position: refs/heads/master@{#432961}
Commit  : fd879031d441f0cf3927dac7762b20630d314c6b
Date    : Thu Nov 17 21:32:05 2016


===== TESTED REVISIONS =====
Revision         Mean      Std Dev  N   Good?
chromium@432960  11407246  5942916  18  good
chromium@432961  14174003  369011   5   bad    <--
chromium@432962  13246336  2473916  8   bad
chromium@432964  13543424  1581088  8   bad
chromium@432968  13781811  2257107  5   bad
chromium@432975  13032448  4493852  18  bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 667794

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile_stress
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/foreground/http_www_baidu_com_s_word_google
Relative Change: 14.25%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/909
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995029812719357008


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5052423020740608

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

yusukes, it appears that your change caused a regression (the bisect says some 2 or 3MB on that story), mostly apparent when we run tests on a browser for a long-ish amount of time.

Could it be a leak introduced by your CL? Or maybe it causes the GC firing patterns to change?

perezju@, the test failing is android_nexus5X_perf_bisect, right? The CL I landed (comment #10) is for OS_CHROME and shouldn't change OS_ANDROID's behavior at all. All files modified are inside is_chromeos:

chrome/gpu/BUILD.gn:

  if (is_chromeos) {
    deps += [ "//components/arc:arc_bindings" ]
    sources += [
      "arc_gpu_video_decode_accelerator.cc",
      "arc_gpu_video_decode_accelerator.h",
      "arc_video_accelerator.h",
      "gpu_arc_video_service.cc",
      "gpu_arc_video_service.h",
    ]
  }

Primiano, I think we need your wisdom to figure this one out:

- a ~3MB regression in java heap on the _stress version of top_10_mobile (no browser restart, no story filter).
- no regression on the regular version of the benchmark (with browser restart).
- clear bisect, pointing to a chrome_os only CL.

how could this be?

Another weird thing about this bug: a blink_perf.svg benchmark regression has been lumped in with this bug, and it has the exact same peaks and valleys as the memory graphs. I don't know if there's anything memory-related to look for in the code: https://cs.chromium.org/chromium/src/third_party/WebKit/PerformanceTests/SVG/SvgHitTesting.html

Any ideas what could be going on there? I checked and:
1) There don't appear to have been device swaps at play
2) memory.top_10_mobile_stress runs on a different device than blink_perf.svg

I've moved the blink_perf.svg regression over to  issue 669448 , let's see if that bisect thinks it is related to the same CL.

Here's my reading of the situation:
1. yusukes CL is definitely unrelated.
2. this time is not even bisect's fault.
The real problem is that *all* the metrics in top_10_mobile_stress stopped being bimodal in the range 432921 - 432975.
It is evident when you look at this link:
https://chromeperf.appspot.com/report?sid=d2e3748ca6ecc4df14bdb05dd6feb20df6cb1053591830684a1bf4d37bddf562&rev=432975

I picked a number of totally unrelated metrics (V8, leveldb, malloc) and all of them exhibit the same pattern (stop of bimodality).

Now, at this point I would have expected a difference in the number of dumps captured. However that doesn't seem to be the case.
The thing that thrills me is:

Let's have a look, for instance, to leveldb memory (it's usually a very stable metric) for the foreground:wikipedia story.
In the run before the aforementioned range we got this [1]: "values": [8208, 8208, 8208, 8208, 8208]
In the run after the aforementioned range we got [2]: "values": [8200, 8200, 8200, 8200, 8200]
So in both cases we got 5 runs and 5 dumps (Good). But before this point the values were consistently oscillating of +- 8 bytes, either always 8208 or 8200 bytes for all runs (bad).

Something very similar happened to v8 where now we get pretty much stably 7 MB, while before we were oscillating between 10 M and 7M. Again it was either always 10 or always 7 for all runs.
The same for all the other metrics. And they happen all to move together:
in some runs all (malloc, v8, leveldb, ...) were either X or Y, after that range they stabilized on Y.

Now the funny part is that this "change of bimodality for the good" happened only on N5X.
So, something change. Very likely something about the lifetime of the processes. But I manually scanned the CLs in the range (and nearby ranges) and none of them seem to possibly have caused that.
Also I still can't explain why only on N5X.


[1] https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__3_%2F2695%2F%2B%2Frecipes%2Fsteps%2Fmemory.top_10_mobile_stress%2F0%2Flogs%2Fjson.output%2F0#
[2] https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__3_%2F2705%2F%2B%2Frecipes%2Fsteps%2Fmemory.top_10_mobile_stress%2F0%2Flogs%2Fjson.output%2F0#

Ok this is some more widespread issue on N5X.
See  crbug.com/669129#c11  which has a similar pattern on system_health benchmarks.
This is not just about top_10_mobile_stress.

-yusukes to stop spamming him about something unrelated with his change.

https://github.com/catapult-project/catapult/issues/3050  :(

Same as  issue 669129 , artificial changes due to recipe change in nexus5X bot, also recovered after fix.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982651855605856176