Make most memory benchmarks reproducible, and some realistic |
||||
Issue descriptionMain design doc and discussion: https://docs.google.com/a/google.com/document/d/1sa0u_-7P0r6ADPZX5cPh3VCpCMshWNVlwkgyBX5nY3E/edit?usp=sharing Implementation proposal: https://docs.google.com/a/google.com/document/d/1mAXsOik83G5QSG8SKr0w-szkIzwH-6MQ-lZoIkPDzc4/edit?usp=sharing We want to make most of our memory benchmarks *reproducible*, providing measurements with low noise levels. This means in particular: - Switch default behavior of all benchmarks to restart browser between page set repeats (go/catabug/2294); this keeps measurements from different iterations statistically independent of each other. - Tune the timing of events (e.g. waits, forced GCs, cache clearing) as to reduce noise levels. This kind of reproducible benchmarks, however, would leave uncovered the case of memory leaks or crashes that only appear after a prolonged period of interaction for the browser. For this we also want to provide a *realistic* benchmark that: - Runs for a long period of time (e.g. --pageset-repeat 60). - Keeps the browser(s) open for the whole duration of the test. - Does not do any forced GCs or cache clearing.
,
Aug 4 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7ead84a83aec79a7f2bb471a5b1177a932f66fab commit 7ead84a83aec79a7f2bb471a5b1177a932f66fab Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Thu Aug 04 16:27:29 2016 Roll src/third_party/catapult/ ae72aad91..ff42a341b (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/ae72aad918e0..ff42a341b88b $ git log ae72aad91..ff42a341b --date=short --no-merges --format='%ad %ae %s' BUG= 625657 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2209353002 Cr-Commit-Position: refs/heads/master@{#409801} [modify] https://crrev.com/7ead84a83aec79a7f2bb471a5b1177a932f66fab/DEPS
,
Aug 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6f2b6d752f9994f123f8195f844d503cd262ec1b commit 6f2b6d752f9994f123f8195f844d503cd262ec1b Author: perezju <perezju@chromium.org> Date: Mon Aug 08 13:19:14 2016 [tools/perf] Switch memory_top_10_mobile to action_runner.MeasureMemory Switch the MemoryTop10Mobile page set to use action_runner.MeasureMemory rather than rolling its own memory dumping code snippet. Also modifies the benchamarks: - memory.top_10_mobile - same behavior as before - memory.top_10_mobile_stress - run the page set in "realistic" mode Depends on: https://codereview.chromium.org/2123713005/ BUG= 625657 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq Review-Url: https://codereview.chromium.org/2123713006 Cr-Commit-Position: refs/heads/master@{#410337} [modify] https://crrev.com/6f2b6d752f9994f123f8195f844d503cd262ec1b/tools/perf/benchmarks/memory_infra.py [modify] https://crrev.com/6f2b6d752f9994f123f8195f844d503cd262ec1b/tools/perf/page_sets/memory_top_10_mobile.py
,
Oct 5 2016
,
Oct 5 2016
,
Sep 6 2017
I think all of this actually happened. |
||||
►
Sign in to add a comment |
||||
Comment 1 by bugdroid1@chromium.org
, Jul 4 2016