rendering benchmark results is noisy |
|||||
Issue descriptionThe rendering benchmark results is noisy, which means it takes longer to detect regressions, and often the metrics are confusing and not actionable. We need to reduce the noise of the benchmark. There are several options we should consider. From ccameron@ in https://bugs.chromium.org/p/chromium/issues/detail?id=866000#c21: """ A few things I think we should change to doing, based on my experience with power testing (a) wait at least 10 seconds after the browser has started and the page has loaded in order to get reliable measurements. (b) disable all JS that isn't related to the test (e.g, JS doing the animation that I'm testing). Otherwise there's just too much noise (c) load the page directly from disk -- there's no need to add network loading as a variable unless that's a variable that we want to be testing (which, for a pinch-zoom test, we don't) """ I worry that introducing ~10s wait would also make the cycle-times, which is already pretty bad, even worse. I think disabling unneeded javascript is a good idea, although I am not sure how we would do it for non-trivial sites like yahoo, facebook etc., especially if we have to disable only some javascript and keep running the rest. If we can disable all javascript, then this should be easier to do. But then, the test result doesn't necessarily reflect what the end-user is experiencing, and so the value of the metric goes down, IMO. I think in addition to exploring these options, we can take some additional steps, e.g.: (d) We can look at the metric at a higher percentile, instead of (or perhaps in addition to) looking at the average. For example, we could look at 75%ile, 95%ile, 99%ile. From some initial local testing, this looks promising (in terms of the metrics being more stable). (e) I would like to see if we can run native perf-tests [1], which would give us greater control on the various parts of chrome (e.g. it would be much easier to just disable the tab-spinner, or the infobars [2], or wait for specific events to occur before proceeding). This should help with reducing the noise in the metrics too. [1] https://docs.google.com/document/d/1Ar8JZjJH5KU-Zodcz0Rtthhi9cGmi0NKI0ZQVobGZvg/edit [2] https://bugs.chromium.org/p/chromium/issues/detail?id=804324
,
Oct 11
+pasko@ who can also gives great advices about direction of reducing noise :-)
,
Oct 12
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/6190da59c2adbb6c0ba9899ada1543fd53e5d46d commit 6190da59c2adbb6c0ba9899ada1543fd53e5d46d Author: Sadrul Habib Chowdhury <sadrul@chromium.org> Date: Fri Oct 12 13:23:34 2018 rendering: Generate metrics at the 95%ile. The metrics at the 95%ile might be more stable than the default average. So we plan on keeping an eye on the 95%ile metric to see if the data is more stable, but at the same time, if it still catches regressions or not. BUG=chromium:894287 Change-Id: Icb717309087dfc30830880077fd5b7322880efba Reviewed-on: https://chromium-review.googlesource.com/c/1274369 Reviewed-by: Ben Hayden <benjhayden@chromium.org> Commit-Queue: Sadrul Chowdhury <sadrul@chromium.org> [modify] https://crrev.com/6190da59c2adbb6c0ba9899ada1543fd53e5d46d/tracing/tracing/metrics/rendering/frame_time.html [modify] https://crrev.com/6190da59c2adbb6c0ba9899ada1543fd53e5d46d/tracing/tracing/metrics/rendering/pixels.html [modify] https://crrev.com/6190da59c2adbb6c0ba9899ada1543fd53e5d46d/tracing/tracing/metrics/rendering/cpu_utilization.html [modify] https://crrev.com/6190da59c2adbb6c0ba9899ada1543fd53e5d46d/tracing/tracing/metrics/rendering/queueing_duration.html [modify] https://crrev.com/6190da59c2adbb6c0ba9899ada1543fd53e5d46d/tracing/tracing/metrics/rendering/latency.html [modify] https://crrev.com/6190da59c2adbb6c0ba9899ada1543fd53e5d46d/tracing/tracing/metrics/rendering/pipeline.html
,
Oct 12
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bb11cbc66e56a475bdf0f5bf3d962cf80c3afe27 commit bb11cbc66e56a475bdf0f5bf3d962cf80c3afe27 Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Date: Fri Oct 12 22:54:58 2018 Roll src/third_party/catapult c8b97e37ec9c..99b88b0a2d58 (21 commits) https://chromium.googlesource.com/catapult.git/+log/c8b97e37ec9c..99b88b0a2d58 git log c8b97e37ec9c..99b88b0a2d58 --date=short --no-merges --format='%ad %ae %s' 2018-10-12 simonhatch@chromium.org Pinpoint - Merge bugs with same root cause. 2018-10-12 benjhayden@chromium.org Change deprecation warnings in /api/alerts to errors. 2018-10-12 benjhayden@chromium.org Cache report names, test suites, and descriptors in v2spa service worker 2018-10-12 chiniforooshan@chromium.org Telemetry: process all rendering pipeline events 2018-10-12 benjhayden@chromium.org Cache session ids in v2spa service worker. 2018-10-12 benjhayden@chromium.org Fix links in rollback.md 2018-10-12 benjhayden@chromium.org Add KeyValueCacheRequest for v2spa service worker. 2018-10-12 benjhayden@chromium.org Add ts_mon metrics to common/timing.py 2018-10-12 benjhayden@chromium.org Add CacheRequestBase for v2spa service worker. 2018-10-12 nednguyen@google.com [Telemetry] Add --test-filter flag that support exact matching of multiple tests 2018-10-12 sadrul@chromium.org rendering: Generate metrics at the 95%ile. 2018-10-12 pasko@chromium.org androidStartupMetric: re-introduce First Contentful Paint 2018-10-12 chrishtr@chromium.org Clean up categories to match current tracing in Blink. 2018-10-11 jbudorick@chromium.org Revert "Enable orderfile memory optimization in benchmarks" 2018-10-11 benjhayden@chromium.org Add task queue for v2spa service worker. 2018-10-11 benjhayden@chromium.org Remove unnecessary file dashboard/test/index.html 2018-10-11 perezju@chromium.org [soundwave] Allow points with missing r_chromium 2018-10-11 chiniforooshan@chromium.org Telemetry: migrate frame_times & percentage_smooth 2018-10-11 ulan@chromium.org [tracing] Restore the minimum mutator utilization metric. 2018-10-11 seanmccullough@chromium.org [chromeperf] add prefix to wct script so tests get found. 2018-10-11 pasko@chromium.org Enable orderfile memory optimization in benchmarks Created with: gclient setdep -r src/third_party/catapult@99b88b0a2d58 The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel BUG=chromium:840872, chromium:893199 ,chromium:867060,chromium:894261,chromium:894287, chromium:886621 ,chromium:893514,chromium:758566, chromium:879526 , chromium:890757 , chromium:892385 , chromium:877660 ,chromium:758566 TBR=sullivan@chromium.org Change-Id: I469ed20d66790fa12e290ad65e2ea51c16cfdba7 Reviewed-on: https://chromium-review.googlesource.com/c/1279078 Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#599392} [modify] https://crrev.com/bb11cbc66e56a475bdf0f5bf3d962cf80c3afe27/DEPS
,
Oct 15
,
Oct 15
,
Oct 17
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by nedngu...@google.com
, Oct 11