Create a Telemetry benchmark to measure Webview startup time |
|||||||||
Issue descriptionSignificant regressions in Webview startup time are a potential release-blocker for Chrome, yet we currently have no Telemetry tests in place that monitor these regressions. We need to get such a test in place. The main blocker in this effort is that we need the ability to use Android atrace events to power Telemetry tests. Once that's possible, we can create a simple TBMv2 metric that'll do what we want. Relevant design doc (google.com only): https://docs.google.com/document/d/1PR-pDWTmG_QV9OW3IgLtRXo3EvIws82ONBkPNH-EBWI/edit#
,
May 20 2016
The blocking Catapult bug for the Telemetry atrace integration is: https://github.com/catapult-project/catapult/issues/2356.
,
May 21 2016
,
May 21 2016
Issue 571629 has been merged into this issue.
,
May 25 2016
+Ben: the startup timing should use the StartupExpectation framework in tracing/
,
Jun 3 2016
,
Jun 4 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bfce663cc58822fb6b7ce17864dbb3698ea8e4bc commit bfce663cc58822fb6b7ce17864dbb3698ea8e4bc Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Jun 04 02:13:16 2016 Roll src/third_party/catapult/ 71d985b3c..be4159a15 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/71d985b3c851..be4159a15b3e $ git log 71d985b3c..be4159a15 --date=short --no-merges --format='%ad %ae %s' BUG= 613706 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2035133004 Cr-Commit-Position: refs/heads/master@{#397890} [modify] https://crrev.com/bfce663cc58822fb6b7ce17864dbb3698ea8e4bc/DEPS
,
Jun 9 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8303e43ab4eea56f6e173e92215e36f5ecfec0eb commit 8303e43ab4eea56f6e173e92215e36f5ecfec0eb Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Thu Jun 09 01:35:10 2016 Roll src/third_party/catapult/ 901f7edd7..8ff23141e (14 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/901f7edd757b..8ff23141e1df $ git log 901f7edd7..8ff23141e --date=short --no-merges --format='%ad %ae %s' BUG=552376,552376, 613706 ,617849, 616483 ,616832 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2048953003 Cr-Commit-Position: refs/heads/master@{#398760} [modify] https://crrev.com/8303e43ab4eea56f6e173e92215e36f5ecfec0eb/DEPS
,
Jun 10 2016
,
Jun 10 2016
Awesome, thanks all for the great work! The benchmark is now running, e.g., at: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20WebView%20Perf%20%282%29/builds/270/steps/system_health.webview_startup And data flows to the dashboards at: https://chromeperf.appspot.com/report?sid=7f416287627f794a5a679c5b4815a71c1730219ea3d49d223848d00472c84776 Although we only have a couple of data points now. I'll try to keep an eye on it to get an idea of noisiness and whether we might want to add a pageset-repeat (it's doing a single blank page once for now).
,
Jun 10 2016
Fantastic! Thanks for sorting this out Alexander (...and everyone else who has helped along the way).
,
Jun 10 2016
Thanks for doing a great job and owning this Alex! Is this bug okay to mark as "fixed"?
,
Jun 10 2016
Yes we can mark this bug as fixed.
,
Jun 13 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/137ca695b63f964ff6788787e19db982754c5f73 commit 137ca695b63f964ff6788787e19db982754c5f73 Author: perezju <perezju@chromium.org> Date: Mon Jun 13 12:19:06 2016 [tools/perf] Tear down state between webview_startup stories This is needed for --page-repat and --pageset-repeat to work correctly. BUG= 613706 Review-Url: https://codereview.chromium.org/2056323002 Cr-Commit-Position: refs/heads/master@{#399435} [modify] https://crrev.com/137ca695b63f964ff6788787e19db982754c5f73/tools/perf/benchmarks/system_health.py
,
Jun 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/137ca695b63f964ff6788787e19db982754c5f73 commit 137ca695b63f964ff6788787e19db982754c5f73 Author: perezju <perezju@chromium.org> Date: Mon Jun 13 12:19:06 2016 [tools/perf] Tear down state between webview_startup stories This is needed for --page-repat and --pageset-repeat to work correctly. BUG= 613706 Review-Url: https://codereview.chromium.org/2056323002 Cr-Commit-Position: refs/heads/master@{#399435} [modify] https://crrev.com/137ca695b63f964ff6788787e19db982754c5f73/tools/perf/benchmarks/system_health.py
,
Jun 20 2016
clicking on the link above in #10 https://chromeperf.appspot.com/report?sid=7f416287627f794a5a679c5b4815a71c1730219ea3d49d223848d00472c84776 I get: Failed to fetch graph data. Note that some data is only available when logged in reopening.
,
Jun 20 2016
Shows up fine for me. Are you logged in with your google or chromium account on the dashboard?
,
Jun 20 2016
Ah yes, signing in problem.
,
Jun 20 2016
Also, that metric you linked seems to no longer be collected. There are others for wall and cpu time that are still being updated though. https://chromeperf.appspot.com/report?sid=ddbb7ab45cc991183324fe98be17765d0236fafcbeec8f5d3578226f351df22c
,
Jul 15 2016
Hi sgurun@, As Randy mentioned, the reason that you were getting an error was that the original metric name (webview_startup_time) wasn't descriptive enough: was that time measured in CPU time or wall time? Because of this, we split the metric into two, webview_startup_cpu_time and webview_startup_wall_time, and migrated all old data to webview_startup_wall_time (because that's what we were actually measuring). |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by nedngu...@google.com
, May 20 2016