Overhaul page_cycler_v2 desktop benchmarks |
|||||||
Issue descriptionWe want to overhaul page_cycler_v2 benchmarks. The requirements are: 1) We want to merge all the page_cycler_v2 benchmarks into a single loading.desktop benchmark (on par with the current loading.mobile version). The benefits of merging are described in issue 713327 2) We want to modernize the lists of URLs & page recording to use so they cover up-to-date use cases. 3) We want to make sure the new combined loading.desktop benchmarks has the total cycle of 1.5h instead of 4h. This is because we wants to agressively reduce the cycle time of perf waterfall (issue 713345). Once there are more hardware in the lab, we can re-add more tests to increase coverage.
,
May 6 2017
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bb9147ed072a94397e8d448ba7445c8f214230f3 commit bb9147ed072a94397e8d448ba7445c8f214230f3 Author: rnephew <rnephew@chromium.org> Date: Fri May 12 11:40:15 2017 [Telemetry] Create loading.desktop, a desktop analog for loading.mobile This will be used to merge the page_cycler_v2 benchmarks into a single benchmark. BUG= 713346 Review-Url: https://codereview.chromium.org/2843133002 Cr-Commit-Position: refs/heads/master@{#471274} [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/testing/buildbot/chromium.perf.json [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/benchmark.csv [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/benchmarks/loading.py [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/core/benchmark_sharding_map.json [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/core/desktop_benchmark_avg_times.json [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/data/loading_desktop.json [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/data/loading_desktop_000.wpr.sha1 [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/data/loading_desktop_001.wpr.sha1 [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/data/loading_desktop_002.wpr.sha1 [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/data/loading_desktop_003.wpr.sha1 [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/data/loading_desktop_005.wpr.sha1 [add] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/loading_desktop.py [modify] https://crrev.com/bb9147ed072a94397e8d448ba7445c8f214230f3/tools/perf/page_sets/page_cycler_story.py
,
May 12 2017
Its timing out on the waterfall: https://build.chromium.org/p/chromium.perf/builders/Win%207%20Perf/builds/708 https://build.chromium.org/p/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf/builds/758 https://build.chromium.org/p/chromium.perf/builders/Win%207%20Nvidia%20GPU%20Perf/builds/738 https://build.chromium.org/p/chromium.perf/builders/Mac%20Mini%208GB%2010.12%20Perf/builds/1563 But not everywhere: https://build.chromium.org/p/chromium.perf/builders/Mac%2010.12%20Perf/builds/476 https://build.chromium.org/p/chromium.perf/builders/Mac%20Pro%2010.11%20Perf/builds/956 There are even more bots where is is currently attempting its first run. Any ideas on what to do about the sharding timing out? As you can see, most of them aer still under the 1.5h mark when they are being killed (I assume that means they are sitting in a pending state for most of the timeout duration?)
,
May 12 2017
I think the problem may be with windows taking longer than mac. 8/8 windows runs failed and only 1/6 mac platforms timed out. Only failures other than timeouts I have seen have been from one story on linux. Linux run: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FLinux_Perf%2F668%2F%2B%2Frecipes%2Fsteps%2Floading.desktop_on__102b__GPU_on_Linux%2F0%2Fstdout
,
May 12 2017
Randy: default hard time out for each swarming bot is 2 hour (see https://cs.chromium.org/chromium/src/tools/perf/core/perf_data_generator.py?rcl=fcf91a8eaacdeaedb0de1eba39ff4c24fab34323&l=543). I would suggest extend that to 2.5 hour for now & if it stills time out, we need to kill a few stories in loading.desktop to fit 2.5 hour.
,
May 12 2017
I'm not sure where a good place to suggest this be, but wouldn't it make sense to have 2 separate timeouts for swarming? A 'stuck in pending' timeout and a 'running too long' timeout? These tests are not running for 2 hours before timing out, a large portion of it is spent pending. I'll get a CL up that ups it to 2.5 hours for now though.
,
May 12 2017
Randy: we already have 2 separate timeouts: hard_timeout vs io_timeout. The latter will kill the test if there is no stdout after |io_timeout| seconds, I believe.
,
May 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0ec821f6c03dd714233032bc97661584958afd5e commit 0ec821f6c03dd714233032bc97661584958afd5e Author: rnephew <rnephew@chromium.org> Date: Sat May 13 07:21:12 2017 [Telemetry] Increase sharding timeout to 2.5 hours from 2 hours. Loading.desktop is currently timing out. Temporarily increasing this timeout until we can get it to not time out. BUG= 713346 Review-Url: https://codereview.chromium.org/2879993003 Cr-Commit-Position: refs/heads/master@{#471577} [modify] https://crrev.com/0ec821f6c03dd714233032bc97661584958afd5e/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/0ec821f6c03dd714233032bc97661584958afd5e/testing/buildbot/chromium.perf.json [modify] https://crrev.com/0ec821f6c03dd714233032bc97661584958afd5e/tools/perf/core/perf_data_generator.py [modify] https://crrev.com/0ec821f6c03dd714233032bc97661584958afd5e/tools/perf/core/perf_data_generator_unittest.py
,
May 15 2017
After bumping up the timeout from 2 hours to 2.5 hours, it is running without timing out on all but one bot. https://build.chromium.org/p/chromium.perf/builders/Win%2010%20High-DPI%20Perf
,
May 15 2017
In the long run, I think we should slow machine should have more bot so we can keep the same runtime SLO across all builder. In the short term, what should we do about this benchmark timed out on Win High DPI Perf?
,
May 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5cf77c9e72082bfc4b0a7a9e1481b20569eacac5 commit 5cf77c9e72082bfc4b0a7a9e1481b20569eacac5 Author: rnephew <rnephew@chromium.org> Date: Fri May 19 03:47:51 2017 [Telemetry] Split Oopif page_cycler_v2 benchmarks into their own file. We are currently working on removing the page_cycler_v2 benchmarks, replacing them with the new loading.desktop benchmark. The Oopif page cycler v2 benchmarks are not yet ready for this, so we have to split them into their own benchmarks seperate from page_cycler_v2. The name shall rename the same (including the page_cycler_v2) in order to keep the data in the same place on the dashboard. Eventually Oopif benchmarks will be migrated to the new loading benchmark harness as well. BUG= 713346 Review-Url: https://codereview.chromium.org/2884383006 Cr-Commit-Position: refs/heads/master@{#473067} [add] https://crrev.com/5cf77c9e72082bfc4b0a7a9e1481b20569eacac5/tools/perf/benchmarks/oopif.py [modify] https://crrev.com/5cf77c9e72082bfc4b0a7a9e1481b20569eacac5/tools/perf/benchmarks/page_cycler_v2.py
,
May 22 2017
,
May 25 2017
Ping on the discussion on what we should do about the timeouts. Its only happening persistently on the win high dpi bots, but it happens sporadically on other windows bots as well.
,
May 25 2017
In the long run, we should do sharding per stories, and add more hardwares for slower configs like win high dpi. For now, I suggest just bump up the timeout limit again.
,
May 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/205bf376ac96ed889732bf1172883263213ea0fe commit 205bf376ac96ed889732bf1172883263213ea0fe Author: rnephew <rnephew@chromium.org> Date: Thu May 25 20:36:51 2017 [Telemetry] Increase swarming timeout to 3 hours from 2.5 hours. Loading.desktop is still timing out on win highdpi bots. The long term plan is to shard by story and increase hardware so we do not have timeout issues any longer. This is a temporary measure to allow the test to run now. Bug: 713346 Change-Id: I02404621e4c908515b99dd1259d43058782f1635 Reviewed-on: https://chromium-review.googlesource.com/515585 Commit-Queue: rnephew <rnephew@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#474784} [modify] https://crrev.com/205bf376ac96ed889732bf1172883263213ea0fe/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/205bf376ac96ed889732bf1172883263213ea0fe/testing/buildbot/chromium.perf.json [modify] https://crrev.com/205bf376ac96ed889732bf1172883263213ea0fe/tools/perf/core/perf_data_generator.py [modify] https://crrev.com/205bf376ac96ed889732bf1172883263213ea0fe/tools/perf/core/perf_data_generator_unittest.py
,
Jun 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e0b1b5815b5e272c67bf0f63a96bb65f85814e1b commit e0b1b5815b5e272c67bf0f63a96bb65f85814e1b Author: Juan A. Navarro Perez <perezju@chromium.org> Date: Wed Jun 14 16:56:11 2017 [tools/perf] Cleanup unused code in Top10MobilePageSet Many of the code paths are no longer being used, allowing to simplify the code quite a bit. The only remaining client is page_cycler_v2.top_10_mobile, a cros contrib benchmark. Bug: 713346 Change-Id: I311720c0e8b83710c534bc7767853c27c2f4b3b4 Reviewed-on: https://chromium-review.googlesource.com/522129 Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Reviewed-by: Cheng-Yu Lee <cylee@chromium.org> Cr-Commit-Position: refs/heads/master@{#479423} [modify] https://crrev.com/e0b1b5815b5e272c67bf0f63a96bb65f85814e1b/tools/perf/contrib/cros_benchmarks/page_cycler_v2.py [modify] https://crrev.com/e0b1b5815b5e272c67bf0f63a96bb65f85814e1b/tools/perf/page_sets/top_10_mobile.py
,
Jul 19 2017
,
Mar 26 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by bugdroid1@chromium.org
, Apr 24 2017