primiano@, This is mostly due to a change I made last month to move from a 1s to 5s timeout, crbug.com/720146, I had meant to follow up on this, but it had fallen off my radar.

The basic issue is that we have a resource pool in CC which evicts unused resources over a period of N seconds. With N=1, it ~always happened before the memory dump was taken. With N=5, it sometimes misses the dump (hence the noise here and in the other bug).

We can "fix" this by either:
a) reducing the eviction time in our cache, so that it happens before the dump
b) increasing the dump time-out (5s seems to work)

I know of cases where (a) will regress performance, but (b) is sort of sad as it adds more delays to tracing....

Do you have a preference here? The benefits of the 5s time out were real, but we can also live without them.

Hmm no strong opinion honestly, adding +perezju,+erikchen +hjd for thoughts.

There seems to be a general problem with the benchmark w.r.t taking just one dump: no matter about how smart we are, things like this are always going to happen. On the other side we have to be pragmatic, so at some point we have to make a choice.

perezju@ any idea how bad would be for the benchmark to delay the dumps by 1 -> 5 seconds? If is not too bad, we should do this as a short term solution.

Long term, instead, my feeling is that if we want to keep the "one dump benchmark model" (again, not ideal but pragmatically I think is the only thing we can afford right now) we need some sort of more robust signal for quiescence.
Maybe instead of waiting for N seconds we should wait for an idle callback? ericrk@ any idea if this specific thing wouldn't have happened if we were using idle callbacks here?

Benchmarking is hard.

I'll kick off an experiment to measure the effect of a 5 seconds delay.

This sounds very much related to  issue 733897 , where I was investigating noise in memory metrics and found gpu:graphics to be very strictly bimodal.

Results of the experiment are in!

Bumping _DUMP_WAIT_TIME from its current 3 to 5 seconds does indeed reduce noise quite significantly in both gpu and cc memory (on gpu and renderer processes respectively).

According to my calculations this should add ~13 minutes total run-time to system_health.memory_mobile.

+nednguyen, given large reductions in noise, it's probably worth to pay the extra 13 minutes of run-time; what do you think?

== Details ==

I ran a try job with:

$ tools/perf/run_benchmark try android-nexus5 system_health.memory_mobile --story-filter browse:chrome:omnibox --pageset-repeat 100

The first graph shows the stdev (in MiB) for all metrics reported by memory-infra on individual processes. The previously noisiest metrics are now under 0.4MiB after the patch; and a few metrics became slightly noisier but only by a tiny not.

The second graph is a detail of the previously noisiest metrics, comparing the effect without/with patch.

Colab with data to play with: https://colab.corp.google.com/v2/notebook#fileId=0B586tdnIdbhKZV94alc1RjRwRUU

Deleted: metric_stdev.png 17.3 KB

	metric_stdev.png 17.3 KB View Download

Deleted: noisiest_metrics.png 63.2 KB

	noisiest_metrics.png 63.2 KB View Download

fwiw the gpu scheduler CL also improved a lot of memory metrics: https://chromeperf.appspot.com/group_report?rev=479175

I've submitted a CL to verify this on the perfbot but now I'm thinking this might not be a real improvement :/

@Primiano: to confirm, the long term solution is still moving the dumping logic to inside Chrome (event based) instead of keeping it in the benchmark code?

If that's the plan, I am fine with whatever cheapest fix we can do in the short term to reduce the noise.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e3095a72f2b1928064b14e9f1d0119b00b171949

commit e3095a72f2b1928064b14e9f1d0119b00b171949
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Mon Jun 26 15:40:28 2017

Roll src/third_party/catapult/ d4f2d777c..b4eb70c31 (1 commit)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/d4f2d777c7aa..b4eb70c319cb

$ git log d4f2d777c..b4eb70c31 --date=short --no-merges --format='%ad %ae %s'
2017-06-26 perezju [System Health] Extend dump time to 5 seconds

Created with:
  roll-dep src/third_party/catapult
BUG= 734853 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I61dc56b5ed9b543cda55f5133ecf76e9d28f22a1
Reviewed-on: https://chromium-review.googlesource.com/548676
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#482283}
[modify] https://crrev.com/e3095a72f2b1928064b14e9f1d0119b00b171949/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/35a87f9da8aa1268ec4cdf1c0a7e6835ac53b268

commit 35a87f9da8aa1268ec4cdf1c0a7e6835ac53b268
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Sat Jul 08 11:53:54 2017

Roll src/third_party/catapult/ 256098db1..00b0c16c9 (1 commit)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/256098db1e09..00b0c16c9708

$ git log 256098db1..00b0c16c9 --date=short --no-merges --format='%ad %ae %s'
2017-07-08 perezju Revert of [System Health] Extend dump time to 5 seconds (patchset #2 id:20001 of https://codereview.chromium.org/2946013004/ )

Created with:
  roll-dep src/third_party/catapult
BUG= 734853 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I767d74467eecdddcc223be036d7ae68a6eb9c1e6
Reviewed-on: https://chromium-review.googlesource.com/564312
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#485147}
[modify] https://crrev.com/35a87f9da8aa1268ec4cdf1c0a7e6835ac53b268/DEPS

I think this is obsolete by this point. While increasing the delay is desirable, it seems that it may introduce as many headaches as we're trying to fix. This is just something we should keep in mind as we triage memory issues.