Automate WebVR latency testing |
||||
Issue descriptionWe can manually test the WebVR app latency using the Motopho, but this takes time to set up and isn't performed regularly. We have access to the ShakerBot, which allows us to physically move a test device, so we should be able to combine the two to run regular, automated latency tests on ToT builds.
,
Apr 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a15848198deb716d77753d297355bac34cd33b96 commit a15848198deb716d77753d297355bac34cd33b96 Author: bsheedy <bsheedy@chromium.org> Date: Thu Apr 20 21:35:27 2017 Make VR latency results compatible with perf dashboard, refactor into classes Makes the following changes to the automated VR latency testing script: - Adds the functionality to save latency and correlation results in a format compatible with the Chrome performance dashboard - Refactors the code into separate classes and files for improved readability and to better support future Windows latency testing - Runs the test multiple times (default 10) to get an average and standard deviation BUG= 708747 Review-Url: https://codereview.chromium.org/2823883003 Cr-Commit-Position: refs/heads/master@{#466147} [modify] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/BUILD.gn [add] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/BUILD.gn [delete] https://crrev.com/f6cbd1a7c6fc9970840ad5242dad4cfb55753630/chrome/test/vr/perf/latency/BUILD.gn [add] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/latency/__init__.py [add] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/latency/android_webvr_latency_test.py [add] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/latency/motopho_thread.py [add] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/latency/robot_arm.py [modify] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/latency/run_latency_test.py [add] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/chrome/test/vr/perf/latency/webvr_latency_test.py [modify] https://crrev.com/a15848198deb716d77753d297355bac34cd33b96/testing/buildbot/gn_isolate_map.pyl
,
May 11 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/44432510039a7dfcf356a8669134d98aac0dd6f4 commit 44432510039a7dfcf356a8669134d98aac0dd6f4 Author: bsheedy <bsheedy@google.com> Date: Thu May 11 19:43:56 2017
,
May 31 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cfcb849246a5b77e9cac7ae92813f9b386f83b68 commit cfcb849246a5b77e9cac7ae92813f9b386f83b68 Author: bsheedy <bsheedy@chromium.org> Date: Wed May 31 22:18:57 2017 Support multiple URLs for latency testing Adds support for testing multiple URLs per script run for automated VR latency testing. Allows us to get latency results from more realistic scenarios instead of just the synthetic flicker app without having to run the script and all the setup overhead that comes with it multiple times. Also fixes an issue caused by the recent swarming bot config change that resulted in the adb startup message appearing in the device name. BUG= 708747 Review-Url: https://codereview.chromium.org/2904583003 Cr-Commit-Position: refs/heads/master@{#476058} [modify] https://crrev.com/cfcb849246a5b77e9cac7ae92813f9b386f83b68/chrome/test/vr/perf/latency/android_webvr_latency_test.py [modify] https://crrev.com/cfcb849246a5b77e9cac7ae92813f9b386f83b68/chrome/test/vr/perf/latency/robot_arm.py [modify] https://crrev.com/cfcb849246a5b77e9cac7ae92813f9b386f83b68/chrome/test/vr/perf/latency/run_latency_test.py [modify] https://crrev.com/cfcb849246a5b77e9cac7ae92813f9b386f83b68/chrome/test/vr/perf/latency/webvr_latency_test.py
,
Jun 6 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/abc06e58df49f78d04c16c13c7c481db26fed962 commit abc06e58df49f78d04c16c13c7c481db26fed962 Author: bsheedy <bsheedy@google.com> Date: Tue Jun 06 23:11:20 2017
,
Jul 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/b114519baf9bf750022c11477aa36df1b606402a commit b114519baf9bf750022c11477aa36df1b606402a Author: bsheedy <bsheedy@chromium.org> Date: Thu Jul 06 17:01:36 2017 Expose mb_config_path in chromium_tests.main_waterfall_steps Allows an alternate mb_config_path to be set to be used in the main waterfall steps so that the internal Chromium recipe wrapper can use its own mb_config.pyl file. Bug: 708747 Change-Id: I6c6ce427d1a9590f3ba82d98557b4b44555c2d1e Reviewed-on: https://chromium-review.googlesource.com/527532 Commit-Queue: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> [modify] https://crrev.com/b114519baf9bf750022c11477aa36df1b606402a/scripts/slave/recipe_modules/chromium_tests/api.py
,
Jul 12 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/b5d45c166cf1cbc25e620d0b5ad0a9e183c2a1c1 commit b5d45c166cf1cbc25e620d0b5ad0a9e183c2a1c1 Author: bsheedy <bsheedy@google.com> Date: Wed Jul 12 16:48:10 2017
,
Jul 13 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/c118b4eea690ca9d281596a40a37a7297986b896 commit c118b4eea690ca9d281596a40a37a7297986b896 Author: bsheedy <bsheedy@google.com> Date: Thu Jul 13 21:45:22 2017
,
Jul 14 2017
,
Jul 18 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/01c1365ce1405b9ff015063b68e2f5d123d34379 commit 01c1365ce1405b9ff015063b68e2f5d123d34379 Author: bsheedy <bsheedy@google.com> Date: Tue Jul 18 21:24:51 2017
,
Jul 20 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/00302ae47e6644ea6c38860830890e26f5b8f451 commit 00302ae47e6644ea6c38860830890e26f5b8f451 Author: bsheedy <bsheedy@google.com> Date: Thu Jul 20 16:32:37 2017
,
Jul 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/df7a4c59b34a2075f3ddb61042b07ab1704747c6 commit df7a4c59b34a2075f3ddb61042b07ab1704747c6 Author: bsheedy <bsheedy@chromium.org> Date: Mon Jul 24 22:51:54 2017 Make VR perf tests compatible with bots Adds --isolated-script-test-output and --isolated-script-test-chartjson-output to the list of known arguments for non-Telemetry VR perf tests. This is due to the tests being run as isolated scripts on the automated perf bot, which is automatically passed these two arguments. Bug: 708747 Change-Id: Icf4345e6e96a671feeb6206c1959fc1aa7379441 Reviewed-on: https://chromium-review.googlesource.com/580111 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Christopher Grant <cjgrant@chromium.org> Commit-Queue: Brian Sheedy <bsheedy@chromium.org> Cr-Commit-Position: refs/heads/master@{#489114} [modify] https://crrev.com/df7a4c59b34a2075f3ddb61042b07ab1704747c6/chrome/test/vr/perf/latency/webvr_latency_test.py [modify] https://crrev.com/df7a4c59b34a2075f3ddb61042b07ab1704747c6/chrome/test/vr/perf/vr_perf_test.py [modify] https://crrev.com/df7a4c59b34a2075f3ddb61042b07ab1704747c6/chrome/test/vr/perf/vr_test_arg_parser.py [modify] https://crrev.com/df7a4c59b34a2075f3ddb61042b07ab1704747c6/chrome/test/vr/perf/vrcore_fps/vrcore_fps_test.py [modify] https://crrev.com/df7a4c59b34a2075f3ddb61042b07ab1704747c6/testing/buildbot/gn_isolate_map.pyl
,
Jul 26 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/2c09ec2cb6de1aac4052ffa074aab63adf63d577 commit 2c09ec2cb6de1aac4052ffa074aab63adf63d577 Author: bsheedy <bsheedy@google.com> Date: Wed Jul 26 16:33:15 2017
,
Jul 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c51acbbcedd78134a3151188afa688e645b63102 commit c51acbbcedd78134a3151188afa688e645b63102 Author: bsheedy <bsheedy@chromium.org> Date: Fri Jul 28 22:01:36 2017 Make VR latency test more robust against low correlation Makes the VR motion-to-photon latency test more robust against failures due to low correlation by allowing a certain amount of retries per URL. This should prevent the automated bot from failing to get valid results with its current minimum correlation settings. Could also be used to improve result consistency by raising the minimum required correlation and allowing it to retry more times. Bug: 708747 Change-Id: I8593ad4059b6d8b1737cdf242192696f027a3843 Reviewed-on: https://chromium-review.googlesource.com/587687 Reviewed-by: Christopher Grant <cjgrant@chromium.org> Commit-Queue: Brian Sheedy <bsheedy@chromium.org> Cr-Commit-Position: refs/heads/master@{#490522} [modify] https://crrev.com/c51acbbcedd78134a3151188afa688e645b63102/chrome/test/vr/perf/latency/motopho_thread.py [modify] https://crrev.com/c51acbbcedd78134a3151188afa688e645b63102/chrome/test/vr/perf/latency/webvr_latency_test.py
,
Aug 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/50d8f0fd48d236c3b9af4fd3abdcda8f4f6bb4f8 commit 50d8f0fd48d236c3b9af4fd3abdcda8f4f6bb4f8 Author: bsheedy <bsheedy@chromium.org> Date: Tue Aug 15 18:17:59 2017 Bound automated VR latency test results Causes the automated VR latency test to retry an iteration if the latency result is clearly wrong. This is to prevent massive, erroneous spikes in results such as https://chromeperf.appspot.com/report?sid=b834b8c2f343a023056eeb93ab7657054caaec03407ea6fb3a0aea076a85fff9&rev=494118 Also increases the max number of allowed retries to compensate for the potential increase in failed iterations. Bug: 708747 Change-Id: Ibacfa9d050fa4c01e7a2e304097809b085ae2163 Reviewed-on: https://chromium-review.googlesource.com/614364 Commit-Queue: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Christopher Grant <cjgrant@chromium.org> Cr-Commit-Position: refs/heads/master@{#494451} [modify] https://crrev.com/50d8f0fd48d236c3b9af4fd3abdcda8f4f6bb4f8/chrome/test/vr/perf/latency/motopho_thread.py [modify] https://crrev.com/50d8f0fd48d236c3b9af4fd3abdcda8f4f6bb4f8/chrome/test/vr/perf/latency/webvr_latency_test.py
,
Oct 16 2017
I'm going to say that this is fixed at this point, as we have automated latency tests running pretty reliably now. It's still only on one device atm, but additional device coverage will come for free as we expand general perf test device coverage.
,
Jul 4
|
||||
►
Sign in to add a comment |
||||
Comment 1 by bugdroid1@chromium.org
, Apr 13 2017