There are way too many smoothness benchmarks |
||||||||
Issue descriptionCurrently, we have way too many smoothness benchmark. It's very confusing which ones are actively maintained & which ones are not. To see all the smoothness benchmarks on perf dashboard, one can go to https://chromeperf.appspot.com/report, then type "smoothness" in "Test suite" field to see.
,
May 18 2016
,
May 19 2016
Add owners of smoothness benchmarks: do you have any opinion which one we can remove here? Removing benchmarks that don't matter is now a P1 bug for us because our perf dashboard system currenlty can't handle the load (https://github.com/catapult-project/catapult/issues/2340)
,
May 19 2016
,
May 19 2016
Sounds good to try to reduce the benchmarks, but I hope this is not considered a long term solution to dashboard scalability.
,
May 19 2016
Yes, it's not. The long term solution to dashboard scalability is switching from datastore to bigtable. However, that gonna take a while to do, and we are already at the tipping point of dashboard's capacity, so this is a P1 bug for us.
,
May 19 2016
I agree with the comment about the dashboard's scalability. I care very much about smoothness.tough_webgl_cases and want it to keep running. In the spreadsheet I note it says "not monitored" -- what does that mean? I am looking periodically at some of the numbers, and assume that automated alerts are in place if the numbers get worse.
,
May 19 2016
Shouldn't we be able to roughly determine which benchmarks are giving us the least value in an automated way? If we sort benchmarks by the number of alerts fired in the past 6 months which weren't marked WontFix, I'd expect the least useful benchmarks to be the ones that fired the fewest alerts. I value all of: thread_times.tough_scrolling_cases smoothness.sync_scroll.key_mobile_sites_smooth smoothness.key_mobile_sites_smooth smoothness.top_25_smooth
,
May 19 2016
I think we can remove "smoothness.gpu_rasterization.key_mobile_sites_smooth". It's currently disabled though, so it isn't adding to the dashboard volume.
,
May 19 2016
,
May 20 2016
Alerts aren't the only value we get: benchmarks can also be used to show improvements. It's a bit hard to quantify this in an automated way though.
,
May 20 2016
You're right. We could eliminate a bunch of candidates from the list of potential removals that way, but we couldn't say that a specific benchmark should be removed.
,
May 24 2016
So here is the list of benchmarks sorted by owners: ajuma@ smoothness.key_silk_cases alancutter@ smoothness.tough_animation_cases bokan@ smoothness.tough_pinch_zoom_cases chrishtr@ smoothness.polymer erick@ smoothness.gpu_rasterization.tough_pinch_zoom_cases junov@ smoothness.tough_canvas_cases kbr@, zmo@, bajones@ smoothness.maps kbr@, zmo@, smoothness.tough_webgl_cases picksi@ smoothness.pathological_mobile_sites senorblanco@ smoothness.gpu_rasterization.tough_filters_cases senorblanco@ smoothness.tough_filters_cases senorblanco@ smoothness.tough_path_rendering_cases skyostil@ smoothness.tough_ad_cases skyostil@ smoothness.tough_scrolling_cases skyostil@ smoothness.tough_webgl_ad_cases ssid@ smoothness.key_desktop_move_cases tdresser@, rbyers@ smoothness.sync_scroll.key_mobile_sites_smooth vmiura@ smoothness.gpu_rasterization.key_mobile_sites_smooth vmiura@ smoothness.gpu_rasterization.polymer vmiura@ smoothness.gpu_rasterization.top_25_smooth vmiura@ smoothness.simple_mobile_sites vmiura@ smoothness.top_25_smooth vmiura@, tdresser@ smoothness.key_mobile_sites_smooth UNOWNED smoothness.bidirectionally_scrolling_tough_ad_cases UNOWNED smoothness.desktop_tough_pinch_zoom_cases UNOWNED smoothness.fling.simple_mobile_sites UNOWNED smoothness.gpu_rasterization_and_decoding.image_decoding_cases UNOWNED smoothness.gpu_rasterization.desktop_tough_pinch_zoom_cases UNOWNED smoothness.gpu_rasterization.tough_path_rendering_cases UNOWNED smoothness.gpu_rasterization.tough_scrolling_cases UNOWNED smoothness.image_decoding_cases UNOWNED smoothness.scrolling_tough_ad_cases UNOWNED smoothness.tough_animated_image_cases UNOWNED smoothness.tough_image_decode_cases UNOWNED smoothness.tough_scrolling_while_zoomed_in_cases UNOWNED smoothness.tough_texture_upload_cases For the starting point, I will delete all the unowned benchmarks, which are: smoothness.bidirectionally_scrolling_tough_ad_cases smoothness.desktop_tough_pinch_zoom_cases smoothness.fling.simple_mobile_sites smoothness.gpu_rasterization_and_decoding.image_decoding_cases smoothness.gpu_rasterization.desktop_tough_pinch_zoom_cases smoothness.gpu_rasterization.tough_path_rendering_cases smoothness.gpu_rasterization.tough_scrolling_cases smoothness.image_decoding_cases smoothness.scrolling_tough_ad_cases smoothness.tough_animated_image_cases smoothness.tough_image_decode_cases smoothness.tough_scrolling_while_zoomed_in_cases smoothness.tough_texture_upload_cases Please let me know you want to keep if any of those benchmarks to be deleted.
,
May 24 2016
I've got a patch deleting smoothness.bidirectionally_scrolling_tough_ad_cases: https://codereview.chromium.org/2003353002/ (Stuck in cq because linux_perf_cq is broken.) Can you sign me up to own smoothness.scrolling_tough_ad_cases? We use that quite a bit.
,
May 24 2016
Let's squash one of smoothness.polymer or smoothness.gpu_rasterization.polymer.
,
May 24 2016
Please make me owner of smoothness.gpu_rasterization.tough_path_rendering_cases (don't delete it). Thanks!
,
May 24 2016
Let's not delete smoothness.desktop_tough_pinch_zoom_cases & smoothness.gpu_rasterization.desktop_tough_pinch_zoom_cases. Those should be owned by ericrk@. Other owners: *tough_path_rendering_cases => senorblanco@ *tough_scrolling_cases => reveman@ *tough_image_decode_cases => cblume@ *image_decoding_cases => cblume@
,
May 24 2016
Let's squash smoothness.desktop_tough_pinch_zoom_cases and smoothness.gpu_rasterization.desktop_tough_pinch_zoom_cases - I should also be the owner of those. I can put together a CL.
,
May 24 2016
Folks, feel free to edit go/perf-owners directly but please add a comment to cc me & sullivan@ on your change.
,
May 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/44e5b5558ef50d30adce9eaae981e0a2cde1a886 commit 44e5b5558ef50d30adce9eaae981e0a2cde1a886 Author: nednguyen <nednguyen@google.com> Date: Thu May 26 16:37:27 2016 [tools/perf] Remove smoothness.fling.simple_mobile_sites BUG= 609523 Review-Url: https://codereview.chromium.org/2012573002 Cr-Commit-Position: refs/heads/master@{#396196} [modify] https://crrev.com/44e5b5558ef50d30adce9eaae981e0a2cde1a886/tools/perf/benchmarks/smoothness.py
,
May 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dae5ac0c4418d454baa570b80d4b5bc23f00149d commit dae5ac0c4418d454baa570b80d4b5bc23f00149d Author: ericrk <ericrk@chromium.org> Date: Fri May 27 00:34:04 2016 With Mac now running with GPU rasterization enabled by default, we can remove Mac from a number of the GPU rasterization benchmarks, and remove some that were only run on mac. BUG= 609523 Review-Url: https://codereview.chromium.org/2016773002 Cr-Commit-Position: refs/heads/master@{#396330} [modify] https://crrev.com/dae5ac0c4418d454baa570b80d4b5bc23f00149d/tools/perf/benchmarks/smoothness.py
,
Jun 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d0d23ec53312500e59cbbe8dd7064e8027cb3465 commit d0d23ec53312500e59cbbe8dd7064e8027cb3465 Author: nednguyen <nednguyen@google.com> Date: Wed Jun 01 22:38:08 2016 [tools/perf] Remove smoothness.tough_animated_image_cases BUG= 609523 Review-Url: https://codereview.chromium.org/2031803002 Cr-Commit-Position: refs/heads/master@{#397272} [modify] https://crrev.com/d0d23ec53312500e59cbbe8dd7064e8027cb3465/tools/perf/benchmarks/smoothness.py [delete] https://crrev.com/bf45205a016725fc61a39985726bd5459beb3899/tools/perf/page_sets/tough_animated_image_cases.py [delete] https://crrev.com/bf45205a016725fc61a39985726bd5459beb3899/tools/perf/page_sets/tough_animated_image_cases/gifs.html
,
Jun 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bba54052dd02dab5950e663085e3aa290cc1373a commit bba54052dd02dab5950e663085e3aa290cc1373a Author: nednguyen <nednguyen@google.com> Date: Wed Jun 01 22:44:49 2016 [tools/perf] Remove smoothness.polymer benchmark Keep smoothness.gpu_rasterization.polymer and remove the other one. BUG= 609523 Review-Url: https://codereview.chromium.org/2027403002 Cr-Commit-Position: refs/heads/master@{#397274} [modify] https://crrev.com/bba54052dd02dab5950e663085e3aa290cc1373a/tools/perf/benchmarks/smoothness.py
,
Jun 2 2016
I wrote smoothness.tough_scrolling_while_zoomed_in_cases It was designed to test checkerboarding. In order to do this, it had to scroll at pretty ridiculous speeds to measure anything. If we regressed raster performance it should show up there. However, it also had a bug -- crbug.com/502027 The way checkerboarding was measured did not correctly include scale when pinch zoomed in. So really, it wasn't a good place to monitor until that bug was fixed, anyway.
,
Jun 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/84bbbdf62fe7190e0b5855fdb44ea85936effb1e commit 84bbbdf62fe7190e0b5855fdb44ea85936effb1e Author: nednguyen <nednguyen@google.com> Date: Thu Jun 02 11:15:39 2016 [tools/perf] Remove smoothness.tough_scrolling_while_zoomed_in_cases benchmark BUG= 609523 Review-Url: https://codereview.chromium.org/2035553002 Cr-Commit-Position: refs/heads/master@{#397362} [modify] https://crrev.com/84bbbdf62fe7190e0b5855fdb44ea85936effb1e/tools/perf/benchmarks/smoothness.py [delete] https://crrev.com/d5eb5fc29849c3ddb1f96d87f84c448cd65286cd/tools/perf/page_sets/tough_scrolling_while_zoomed_in_cases.py
,
Jun 2 2016
,
Jun 7 2016
All unowned benchmarks have been deleted. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by sullivan@chromium.org
, May 5 2016