TBMv2 Memory Benchmarks & Value Name Migration |
||||||||
Issue description
This is an overall tracking bug for coordinating TBMv2 memory benchmarks
and memory metric value name migration. The following actions are to be
done:
1. Stop reporting new and drop existing non-V8 memory values in
long-running benchmarks.
2. Drop existing non-average values in memory-infra and system health
benchmarks.
(https://bugs.chromium.org/p/chromium/issues/detail?id=610962#c20)
3. Rename old memory values.
(https://bugs.chromium.org/p/chromium/issues/detail?id=610962#c21)
4. Normalize (newly reported) memory metric value names and
subsequently rename all existing values.
(https://codereview.chromium.org/2018503002/)
5. Re-enable memory system health benchmarks.
(https://codereview.chromium.org/2012743002/)
Full details of the plan: https://docs.google.com/a/google.com/document/d/12erOqBy2Y7ZMFTGpuX2MpK0jK06CQM0V3o8w1_FsCeo/edit?usp=sharing
,
Jun 6 2016
,
Jun 6 2016
,
Jun 6 2016
,
Jun 6 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b85904016ad11c2b2712cad3b545eab6e2e4c693 commit b85904016ad11c2b2712cad3b545eab6e2e4c693 Author: ulan <ulan@chromium.org> Date: Mon Jun 06 17:56:12 2016 Limit tracked values in long running gmail benchmarks to V8 specific values and the overall renderer memory stats. BUG= 617117 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2030343002 Cr-Commit-Position: refs/heads/master@{#398060} [modify] https://crrev.com/b85904016ad11c2b2712cad3b545eab6e2e4c693/tools/perf/benchmarks/memory_infra.py
,
Jun 7 2016
Here's the state of the current plan:
1. Stop reporting non-V8 memory values in long-running benchmarks.
(DONE: https://codereview.chromium.org/2030343002 by ulan@)
2. Drop existing non-V8 memory values in long-running benchmarks.
(TODO: issue 617654 assigned to sullivan@)
3. Drop existing non-average values in memory-infra and system health
benchmarks.
(TODO: issue 617636 assigned to sullivan@)
4. Prepare benchmarks using the memory metric for normalization.
(IN REVIEW: https://codereview.chromium.org/2044853002/ by
petrcermak@)
5. Notify benchmark owners using the memory metric and perf sheriffs
about normalization.
(TODO: petrcermak@)
6. Normalize newly reported memory metric value names.
(BLOCKED on 2+3+4: https://codereview.chromium.org/2018503002/ by
petrcermak@)
7. Transfer alerts of all benchmarks using the memory metric.
(BLOCKED on 5)
8. Rename all old memory values.
(BLOCKED on 5)
9. Re-enable memory system health benchmarks.
(BLOCKED on 5: https://codereview.chromium.org/2012743002/ by
petrcermak@)
I realized it would probably make more sense to perform all renaming
simultaneously once the memory metric is normalized. That's why steps
6 and 8 were swapped.
,
Jun 7 2016
After a chat with sullivan@, I'll go ahead and re-enable memory system health benchmarks again right now. This only means that there will be a few more values to rename in step 8.
,
Jun 8 2016
,
Jun 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b5ab3f3839905899869fa80764428c50642fb560 commit b5ab3f3839905899869fa80764428c50642fb560 Author: petrcermak <petrcermak@chromium.org> Date: Wed Jun 08 18:18:55 2016 [perf] Prepare benchmarks for memory metric normalization This is patch is necessary so that benchmarks using the memory metric wouldn't stop reporting values once https://codereview.chromium.org/2018503002/ lands. BUG= 617117 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2044853002 Cr-Commit-Position: refs/heads/master@{#398620} [modify] https://crrev.com/b5ab3f3839905899869fa80764428c50642fb560/tools/perf/benchmarks/memory_infra.py [modify] https://crrev.com/b5ab3f3839905899869fa80764428c50642fb560/tools/perf/benchmarks/v8.py
,
Jun 9 2016
Update on the progress (re-ordered to reflect reality):
1. Stop reporting non-V8 memory values in long-running benchmarks.
(DONE: https://codereview.chromium.org/2030343002 by ulan@)
2. Drop existing non-average values in memory-infra and system health
benchmarks.
(DONE: issue 617636 assigned to sullivan@)
3. Drop existing non-V8 memory values in long-running benchmarks.
(ALMOST DONE: issue 617654 assigned to sullivan@)
4. Prepare benchmarks using the memory metric for normalization.
(DONE: https://codereview.chromium.org/2044853002/ by petrcermak@)
5. Notify benchmark owners using the memory metric and perf sheriffs
about normalization.
(DONE: petrcermak@)
6. Re-enable memory system health benchmarks.
(DONE: https://codereview.chromium.org/2012743002/ by petrcermak@)
7. Normalize newly reported memory metric value names.
(WAITING FOR FINAL SIGN-OFF:
https://codereview.chromium.org/2018503002/ by petrcermak@)
8. Transfer alerts of all benchmarks using the memory metric.
(BLOCKED on 7)
9. Rename all old memory values.
(BLOCKED on 7)
,
Jun 13 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f5de9f7cef3514f7ec57c554604afd4e9277064f commit f5de9f7cef3514f7ec57c554604afd4e9277064f Author: petrcermak <petrcermak@chromium.org> Date: Mon Jun 13 10:18:05 2016 [perf] Prepare benchmarks for memory metric normalization (vol. 2) This is patch is necessary so that benchmarks using the memory metric wouldn't stop reporting values once https://codereview.chromium.org/2018503002/ lands. First preparation patch: https://codereview.chromium.org/2044853002/ BUG= 617117 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2060443002 Cr-Commit-Position: refs/heads/master@{#399422} [modify] https://crrev.com/f5de9f7cef3514f7ec57c554604afd4e9277064f/tools/perf/benchmarks/memory_infra.py
,
Jun 14 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a831a0c9fde99c4263d3e92f82c691841483db41 commit a831a0c9fde99c4263d3e92f82c691841483db41 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Jun 14 11:27:02 2016 Roll src/third_party/catapult/ e3efccf3f..76449fc4e (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/e3efccf3f35e..76449fc4e65a $ git log e3efccf3f..76449fc4e --date=short --no-merges --format='%ad %ae %s' BUG= 617117 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2062043002 Cr-Commit-Position: refs/heads/master@{#399682} [modify] https://crrev.com/a831a0c9fde99c4263d3e92f82c691841483db41/DEPS
,
Jun 14 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/92809021dabc3fd5036bb69b516ba113aabf0c67 commit 92809021dabc3fd5036bb69b516ba113aabf0c67 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Jun 14 12:42:31 2016 Roll src/third_party/catapult/ 76449fc4e..4aaf1cd42 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/76449fc4e65a..4aaf1cd420f1 $ git log 76449fc4e..4aaf1cd42 --date=short --no-merges --format='%ad %ae %s' BUG= 617117 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2064113002 Cr-Commit-Position: refs/heads/master@{#399687} [modify] https://crrev.com/92809021dabc3fd5036bb69b516ba113aabf0c67/DEPS
,
Jun 14 2016
We're almost there! The normalization finally happened in two Catapult patches which have just rolled into Chromium: https://codereview.chromium.org/2062043002 (r399682) https://codereview.chromium.org/2064113002 (r399687) Once all perf bots pass r399687, we can rename all old values and migrate alerts (I'll assign a bug to sullivan@ with precise instructions when that happens). In the meantime, I wrote a small script for people wondering what happened to their "favorite" memory value (attached): $ ./normalize.py < memory:webview:browser:android_memtrack:gl_avg > memory:webview:browser_process:reported_by_os:gpu_memory:gl:proportional_resident_size_avg
,
Jun 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f5de9f7cef3514f7ec57c554604afd4e9277064f commit f5de9f7cef3514f7ec57c554604afd4e9277064f Author: petrcermak <petrcermak@chromium.org> Date: Mon Jun 13 10:18:05 2016 [perf] Prepare benchmarks for memory metric normalization (vol. 2) This is patch is necessary so that benchmarks using the memory metric wouldn't stop reporting values once https://codereview.chromium.org/2018503002/ lands. First preparation patch: https://codereview.chromium.org/2044853002/ BUG= 617117 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2060443002 Cr-Commit-Position: refs/heads/master@{#399422} [modify] https://crrev.com/f5de9f7cef3514f7ec57c554604afd4e9277064f/tools/perf/benchmarks/memory_infra.py
,
Jun 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a831a0c9fde99c4263d3e92f82c691841483db41 commit a831a0c9fde99c4263d3e92f82c691841483db41 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Jun 14 11:27:02 2016 Roll src/third_party/catapult/ e3efccf3f..76449fc4e (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/e3efccf3f35e..76449fc4e65a $ git log e3efccf3f..76449fc4e --date=short --no-merges --format='%ad %ae %s' BUG= 617117 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2062043002 Cr-Commit-Position: refs/heads/master@{#399682} [modify] https://crrev.com/a831a0c9fde99c4263d3e92f82c691841483db41/DEPS
,
Jun 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/92809021dabc3fd5036bb69b516ba113aabf0c67 commit 92809021dabc3fd5036bb69b516ba113aabf0c67 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Jun 14 12:42:31 2016 Roll src/third_party/catapult/ 76449fc4e..4aaf1cd42 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/76449fc4e65a..4aaf1cd420f1 $ git log 76449fc4e..4aaf1cd42 --date=short --no-merges --format='%ad %ae %s' BUG= 617117 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2064113002 Cr-Commit-Position: refs/heads/master@{#399687} [modify] https://crrev.com/92809021dabc3fd5036bb69b516ba113aabf0c67/DEPS
,
Jun 15 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/clank/internal/apps/+/728b58e149a6da73766ee3d0ed7e2e33b40da002 commit 728b58e149a6da73766ee3d0ed7e2e33b40da002 Author: Juan A. Navarro Perez <perezju@google.com> Date: Tue Jun 14 16:09:54 2016
,
Jun 15 2016
,
Jun 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f52622ed991f0d5e269a1d99085edb115e58d267 commit f52622ed991f0d5e269a1d99085edb115e58d267 Author: petrcermak <petrcermak@chromium.org> Date: Wed Jun 15 17:08:12 2016 [perf] Don't ignore dump and process counts in TBMv2 memory-infra benchmarks The existing code incorrectly filtered memory dump count totals because they end with '_count': memory:<browser>:all_processes:dump_count The filtering is only supposed to apply to statistics of numerics. BUG= 617117 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2069953003 Cr-Commit-Position: refs/heads/master@{#399932} [modify] https://crrev.com/f52622ed991f0d5e269a1d99085edb115e58d267/tools/perf/benchmarks/memory_infra.py [modify] https://crrev.com/f52622ed991f0d5e269a1d99085edb115e58d267/tools/perf/benchmarks/v8.py
,
Jun 27 2016
The migration has just been finished :-) All TBMv2 memory values on the perf dashboard now follow the new schema:
memory:{chrome, webview}:{browser_process, renderer_processes, ..., all_processes}:reported_by_{chrome, os}:...
,
Jan 4 2017
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by petrcermak@chromium.org
, Jun 3 2016