Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003438993116115584

=== Auto-CCing suspected CL author petrcermak@chromium.org ===

Hi petrcermak@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [system-health] Wait for 10 seconds after each story (except for memory benchmarks)
Author  : petrcermak
Commit description:
  
Non-memory SH benchmarks:
All SH stories will wait for 10 seconds after loading.

Memory SH benchmarks:
All SH stories will re-use action_runner.MeasureMemory().

BUG=589726, 637158 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq

Review-Url: https://codereview.chromium.org/2248693002
Cr-Commit-Position: refs/heads/master@{#413128}
Commit  : 4cd6d6ceb622b79120215cae46df22f8abe68466
Date    : Fri Aug 19 13:58:46 2016


===== TESTED REVISIONS =====
Revision         Mean      Std Dev  N  Good?
chromium@413120  21631795  861142   5  good
chromium@413127  21626061  860762   5  good
chromium@413128  912589    3426.96  5  bad    <--
chromium@413129  913408    4096.0   5  bad
chromium@413131  914227    5340.53  5  bad
chromium@413134  915046    6211.89  5  bad
chromium@413147  910950    4670.16  5  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 640615

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_mobile
Test Metric: load_social-memory:chrome:all_processes:reported_by_os:system_memory:ashmem:private_dirty_size_avg/load_social_facebook
Relative Change: 95.79%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4053
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003438993116115584


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5909101297532928

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

The bisect clearly confirmed my suspicion, so this is a WontFix.

perezju,primiano: FYI this demonstrates the huge impact of flushing caches before measuring memory on ashmem (20.6 MiB → 0.9 MiB).

Nice find. Good to know the effect of flushing caches.

We can see a similar effect (although not as dramatic) on top_10_mobile regular vs stress:
https://chromeperf.appspot.com/report?sid=01841b0e6c3787575d71b7b210732188aa8e495347b1bbd6c9296d687f9480b0&start_rev=410803&end_rev=414034

We should probably spend some time trying to get more acquainted with those effects.

Btw, did other metrics move as much due to your CL?

I don't know. It's quite difficult to figure this out (I'd have to manually go through every single metric). However, the overall private dirty changed quite a lot (-30 MiB on load:media): https://chromeperf.appspot.com/report?sid=9add42bbc7102202d90fe2152ab421d95f2a280990e6eb407196f092c56a259d

Yup ashmem behaves like that by design, we should deal with the fact that ashmem is noisy (do we have any altert there? if so we should drop it).
As you say in #5 the problem is that ashmem also counts in private dirty. I wonder if we should do something smarter there, like counting private-dirty - ashmem.