Make VR Shell Instrumentation Tests Work Properly |
|||||||||
Issue descriptionInstrumentation tests are enabled and running on the FYI bot, but currently failing due to several reasons: * VR Shell is now behind a runtime flag, not just a build flag (already fixed with the @CommandLineFlags.Add annotation) * Bots do not ensure that VR Services (com.google.vr.vrcore) and Daydream home (com.google.android.vr.home) are installed on compatible devices before running tests * Bots do not ensure that the DON flow is skipped and that the first time setup is complete on compatible devices (can be set using ADB)
,
Dec 8 2016
,
Dec 8 2016
,
Dec 8 2016
,
Dec 9 2016
,
Jan 12 2017
I don't know if this should be considered a P1 for M-57 since we aren't shipping VR Shell in M-57 (at least AFAIK). WDYT? We have a plan on how to add support for installing/configuring VRCore and Daydream for continuous tests (https://docs.google.com/document/d/1WU8QkuTWNLrrM_4ZuPU4J22xvpkahpBBxudayUXsa3A/edit) and all the blocking issues have been dealt with AFAIK, so we should just have to take the time to implement it. However, I think getting this in before the branch point will be quite difficult. Getting it in during the beta period should be pretty easily doable, though.
,
Jan 12 2017
Definitely not a P1.
,
Jan 18 2017
Here's a more recent short doc describing the planned work to make automated instrumentation tests work properly https://docs.google.com/document/d/1V6f9bbmDbSUmr0BwEUypmQFRxdvZHiw9RYIAz68c9Hc/edit. It also has a number of questions that need answering. A lot of these are implementation details that are mostly relevant to Lei, Infra, and I, but others are relevant to the team as a whole (e.g. do we want to test multiple versions of VR Services, which adds quite a bit of complexity).
,
Jan 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9850866badcafcfd39db10e436217ca0d5bc8826 commit 9850866badcafcfd39db10e436217ca0d5bc8826 Author: bsheedy <bsheedy@chromium.org> Date: Fri Jan 27 03:14:54 2017 Move VR tests to separate APK Moves VR instrumentation tests (currently just VR Shell, but will include WebVR whenever those finally get added) to a separate test APK. This is in preparation for the work to make VR instrumentation tests run continuously, as the VR tests will need to run multiple times with different additional APKs installed and different settings configured. It was decided in the meeting with jbudorick@ that a separate APK would be better than using chrome_public_test_apk with multiple --test-filter arguments. Shouldn't affect anything other than our FYI bot. BUG= 671373 Review-Url: https://codereview.chromium.org/2651423002 Cr-Commit-Position: refs/heads/master@{#446566} [modify] https://crrev.com/9850866badcafcfd39db10e436217ca0d5bc8826/chrome/android/BUILD.gn [modify] https://crrev.com/9850866badcafcfd39db10e436217ca0d5bc8826/chrome/android/javatests/src/org/chromium/chrome/browser/vr_shell/VrShellTest.java [modify] https://crrev.com/9850866badcafcfd39db10e436217ca0d5bc8826/testing/buildbot/chromium.fyi.json
,
Feb 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6 commit 93075a8bd2d7f9370aa9f3aa19d1a711f76973f6 Author: bsheedy <bsheedy@chromium.org> Date: Tue Feb 07 01:11:02 2017 Add VR Services and Daydream hashes and DEPS entry Adds sha1 files for Daydream Home and VR Services APK versions 1.0 and 1.1 (the current release version). Also adds DEPS entry to automatically download them for Android builds. Also adds *_current.apk.sha1 files to be used for continuous tests on the chromium.android waterfall. We don't want to clog up the bots running the VR tests multiple times with different APK versions, so the chromium.android bots will only test with the most recent release while the other versions can be tested on our FYI bot for compatibility. BUG= 671373 Review-Url: https://codereview.chromium.org/2666213002 Cr-Commit-Position: refs/heads/master@{#448485} [modify] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/README.chromium [add] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/test-apks/daydream_home/daydream_home_1_0.apk.sha1 [add] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/test-apks/daydream_home/daydream_home_1_1.apk.sha1 [add] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/test-apks/daydream_home/daydream_home_current.apk.sha1 [add] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/test-apks/vr_services/vr_services_1_0.apk.sha1 [add] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/test-apks/vr_services/vr_services_1_1.apk.sha1 [add] https://crrev.com/93075a8bd2d7f9370aa9f3aa19d1a711f76973f6/third_party/gvr-android-sdk/test-apks/vr_services/vr_services_current.apk.sha1
,
Feb 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3014682836a4792d50abfa27456fce6a2dceaa0d commit 3014682836a4792d50abfa27456fce6a2dceaa0d Author: bsheedy <bsheedy@chromium.org> Date: Tue Feb 07 02:12:09 2017 Add restriction for running tests only with Daydream View headset Adds an @Restriction option for only running a test if the current VR viewer is set to Daydream View. One example use case of this is for VR tests using the Daydream View NFC scan - since scanning sets the current viewer to Daydream View, we don't want to run an NFC test unless the viewer is already set to Daydream View. Otherwise, the viewer will switch in the middle of running tests, which may affect the results of later tests within the same APK. BUG= 671373 Review-Url: https://codereview.chromium.org/2668003006 Cr-Commit-Position: refs/heads/master@{#448511} [modify] https://crrev.com/3014682836a4792d50abfa27456fce6a2dceaa0d/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/VrClassesWrapperImpl.java [modify] https://crrev.com/3014682836a4792d50abfa27456fce6a2dceaa0d/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/VrDaydreamApiImpl.java [modify] https://crrev.com/3014682836a4792d50abfa27456fce6a2dceaa0d/chrome/android/javatests/src/org/chromium/chrome/browser/vr_shell/VrShellTest.java [modify] https://crrev.com/3014682836a4792d50abfa27456fce6a2dceaa0d/chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeInstrumentationTestRunner.java [modify] https://crrev.com/3014682836a4792d50abfa27456fce6a2dceaa0d/chrome/test/android/javatests/src/org/chromium/chrome/test/util/ChromeRestriction.java
,
Feb 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/32861cfc700aac78ff63eeaef658af5f5c5b2d13 commit 32861cfc700aac78ff63eeaef658af5f5c5b2d13 Author: bsheedy <bsheedy@chromium.org> Date: Mon Feb 13 23:46:54 2017 Add android_vr_apks gclient config to VR FYI bot Makes the VR FYI bot use the gclient confit added in https://chromium-review.googlesource.com/c/437710/ so that the VR APKs are pulled down during the gclient runhooks step. BUG= 671373 Change-Id: I635b0cc1d4971fd40d34367ac4f35da576c77e8b Reviewed-on: https://chromium-review.googlesource.com/439408 Commit-Queue: Brian Sheedy <bsheedy@google.com> Reviewed-by: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/32861cfc700aac78ff63eeaef658af5f5c5b2d13/scripts/slave/recipes/chromium.expected/full_chromium_fyi_Android_VR_Tests.json [modify] https://crrev.com/32861cfc700aac78ff63eeaef658af5f5c5b2d13/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
,
Feb 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/32861cfc700aac78ff63eeaef658af5f5c5b2d13 commit 32861cfc700aac78ff63eeaef658af5f5c5b2d13 Author: bsheedy <bsheedy@chromium.org> Date: Mon Feb 13 23:46:54 2017 Add android_vr_apks gclient config to VR FYI bot Makes the VR FYI bot use the gclient confit added in https://chromium-review.googlesource.com/c/437710/ so that the VR APKs are pulled down during the gclient runhooks step. BUG= 671373 Change-Id: I635b0cc1d4971fd40d34367ac4f35da576c77e8b Reviewed-on: https://chromium-review.googlesource.com/439408 Commit-Queue: Brian Sheedy <bsheedy@google.com> Reviewed-by: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/32861cfc700aac78ff63eeaef658af5f5c5b2d13/scripts/slave/recipes/chromium.expected/full_chromium_fyi_Android_VR_Tests.json [modify] https://crrev.com/32861cfc700aac78ff63eeaef658af5f5c5b2d13/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
,
Feb 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/654a5af38665d42fdcddafaa0a5f85810d5fdacc commit 654a5af38665d42fdcddafaa0a5f85810d5fdacc Author: bsheedy <bsheedy@chromium.org> Date: Thu Feb 16 20:25:01 2017 Add shared preference file modification option to instrumentation test runner Adds an --edit-shared-prefs option to the instrumentation test runner that can be used to edit shared preference files before a test is run. BUG= 671373 Review-Url: https://codereview.chromium.org/2688913003 Cr-Commit-Position: refs/heads/master@{#451084} [modify] https://crrev.com/654a5af38665d42fdcddafaa0a5f85810d5fdacc/build/android/pylib/instrumentation/instrumentation_test_instance.py [modify] https://crrev.com/654a5af38665d42fdcddafaa0a5f85810d5fdacc/build/android/pylib/local/device/local_device_instrumentation_test_run.py [modify] https://crrev.com/654a5af38665d42fdcddafaa0a5f85810d5fdacc/build/android/test_runner.py [modify] https://crrev.com/654a5af38665d42fdcddafaa0a5f85810d5fdacc/build/android/test_runner.pydeps [add] https://crrev.com/654a5af38665d42fdcddafaa0a5f85810d5fdacc/chrome/android/shared_preference_files/test/vr_cardboard_skipdon_setupcomplete.json [add] https://crrev.com/654a5af38665d42fdcddafaa0a5f85810d5fdacc/chrome/android/shared_preference_files/test/vr_ddview_skipdon_setupcomplete.json
,
Feb 16 2017
,
Feb 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/828f55534c88e66ac7d9ae7ed7c6ac3d5ed6172f commit 828f55534c88e66ac7d9ae7ed7c6ac3d5ed6172f Author: bsheedy <bsheedy@chromium.org> Date: Wed Feb 22 20:01:19 2017 Add VR test APKs hooks to DEPS Adds the hooks for the VR test APKs into src/DEPS since it looks like there isn't a consensus on how to add custom hooks to particular bots. BUG= 671373 Review-Url: https://codereview.chromium.org/2712733002 Cr-Commit-Position: refs/heads/master@{#452174} [modify] https://crrev.com/828f55534c88e66ac7d9ae7ed7c6ac3d5ed6172f/DEPS
,
Feb 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8d42011f9c4c3f748efb8bb554a62cbb2ec8b474 commit 8d42011f9c4c3f748efb8bb554a62cbb2ec8b474 Author: bsheedy <bsheedy@chromium.org> Date: Thu Feb 23 21:10:45 2017 Add swarmed version of VR tests to FYI bot Adds a swarmed version of chrome_public_test_vr_apk to Android VR Tests. Currently only runs on a single 5X with N, but will update dimensions over time as more OSes besides N are supported. Instrumentation tests are still kept around since there are no Pixel devices available in swarming, and the builder has a Pixel attached. BUG=642946, 671373 Review-Url: https://codereview.chromium.org/2712613003 Cr-Commit-Position: refs/heads/master@{#452628} [modify] https://crrev.com/8d42011f9c4c3f748efb8bb554a62cbb2ec8b474/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/8d42011f9c4c3f748efb8bb554a62cbb2ec8b474/testing/buildbot/gn_isolate_map.pyl
,
Mar 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0df5baacc99135ecb3402eaf046d960b5d65fbfd commit 0df5baacc99135ecb3402eaf046d960b5d65fbfd Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Wed Mar 08 20:52:35 2017 Roll src/third_party/catapult/ 5366cb026..eaf9a11d2 (3 commits) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/5366cb026ba8..eaf9a11d2821 $ git log 5366cb026..eaf9a11d2 --date=short --no-merges --format='%ad %ae %s' 2017-03-08 bsheedy Generalize the removal of system apps 2017-03-08 rnephew [Telemetry] Output grouping keys during telemetry runs to differentiate stories. 2017-03-08 charliea Allows the CPU tracing agent to handle Windows proc strings with spaces Created with: roll-dep src/third_party/catapult BUG= 671373 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2734393002 Cr-Commit-Position: refs/heads/master@{#455539} [modify] https://crrev.com/0df5baacc99135ecb3402eaf046d960b5d65fbfd/DEPS
,
Mar 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/8b4e13d062698f8baaafdcf6ee0b76910394b173 commit 8b4e13d062698f8baaafdcf6ee0b76910394b173 Author: bsheedy <bsheedy@chromium.org> Date: Wed Mar 08 21:53:26 2017 Add system VRCore removal config option and enable it for VR tester Needs https://codereview.chromium.org/2734023002/ to land and be rolled before this can be committed. BUG= 671373 Change-Id: I498b4f81492783542556d5035e04a8a43b203c75 Reviewed-on: https://chromium-review.googlesource.com/450970 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Brian Sheedy <bsheedy@google.com> [modify] https://crrev.com/8b4e13d062698f8baaafdcf6ee0b76910394b173/scripts/slave/recipe_modules/chromium_android/config.py [modify] https://crrev.com/8b4e13d062698f8baaafdcf6ee0b76910394b173/scripts/slave/recipe_modules/chromium_android/api.py [modify] https://crrev.com/8b4e13d062698f8baaafdcf6ee0b76910394b173/scripts/slave/recipe_modules/chromium_android/example.py [modify] https://crrev.com/8b4e13d062698f8baaafdcf6ee0b76910394b173/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py [add] https://crrev.com/8b4e13d062698f8baaafdcf6ee0b76910394b173/scripts/slave/recipe_modules/chromium_android/example.expected/remove_system_vrcore_basic.json
,
Mar 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/387a3f3fe76f92ef0b02d1743e124df643eb14bf commit 387a3f3fe76f92ef0b02d1743e124df643eb14bf Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Mar 11 02:25:07 2017 Roll src/third_party/catapult/ 5cdfbfc97..4550ebce3 (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/5cdfbfc97459..4550ebce30c0 $ git log 5cdfbfc97..4550ebce3 --date=short --no-merges --format='%ad %ae %s' 2017-03-10 bsheedy Make SharedPrefs apply package's security context on commit Created with: roll-dep src/third_party/catapult BUG= 671373 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2742143002 Cr-Commit-Position: refs/heads/master@{#456261} [modify] https://crrev.com/387a3f3fe76f92ef0b02d1743e124df643eb14bf/DEPS
,
Mar 13 2017
,
Mar 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/70e3edf37a04a60bc3398c65d317de3e51353875 commit 70e3edf37a04a60bc3398c65d317de3e51353875 Author: bsheedy <bsheedy@chromium.org> Date: Tue Mar 14 17:31:19 2017 Fix several issues with VR instrumentation tests Fixes the following issues that were occuring with VR instrumentation tests: - Daydream-readiness checking failing during restriction checks - Strict mode policy failure from VR Shell creation There are still a number of issues preventing the tests from passing consistently that will be addressed in separate CLs: - Strict mode violations from VR Services that are not fixable from Chrome ("--strict-mode off" added for time being to prevent failures) crbug.com/b/35628570 - Null pointer exceptions crbug.com/699220 - General test flakiness (looks like caused by VR Services, but not sure yet) TBR=dtrainor@chromium.org BUG= 671373 Review-Url: https://codereview.chromium.org/2746453003 Cr-Commit-Position: refs/heads/master@{#456752} [modify] https://crrev.com/70e3edf37a04a60bc3398c65d317de3e51353875/chrome/android/java/src/org/chromium/chrome/browser/vr_shell/VrShellDelegate.java [modify] https://crrev.com/70e3edf37a04a60bc3398c65d317de3e51353875/chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeInstrumentationTestRunner.java [modify] https://crrev.com/70e3edf37a04a60bc3398c65d317de3e51353875/testing/buildbot/chromium.fyi.json
,
Mar 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/4e48b5ec0b17fa90512a2abb5c8302556c468cf4 commit 4e48b5ec0b17fa90512a2abb5c8302556c468cf4 Author: bsheedy <bsheedy@chromium.org> Date: Wed Mar 22 22:04:10 2017 Make 'Android Builder (dbg)' download VR test APKs 'Android VR Tests' already has this set, but that only covers instrumentation tests on the locally attached device. The builder that does the compiling needs to download the APKs as well for swarming. BUG= 671373 Change-Id: Idccac16da87ea9c609d3f277bb1d20ed22ea100d Reviewed-on: https://chromium-review.googlesource.com/457782 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Brian Sheedy <bsheedy@google.com> [modify] https://crrev.com/4e48b5ec0b17fa90512a2abb5c8302556c468cf4/scripts/slave/recipes/chromium.expected/full_chromium_fyi_Android_Builder__dbg_.json [modify] https://crrev.com/4e48b5ec0b17fa90512a2abb5c8302556c468cf4/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
,
Apr 3 2017
I'm going to go ahead and (finally) close this, as VR Shell and WebVR instrumentation/e2e tests are running properly now (https://build.chromium.org/p/chromium.fyi/builders/Android%20VR%20Tests). Some tests are still failing, but this is due to actually catching a bug ( Issue 704692 ), not an issue with the tests themselves. Similarly, sometimes there's an infrastructure exception, but this is due to the swarming tests timing out waiting for devices since FYI bots have such a low priority. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by bsheedy@chromium.org
, Dec 5 2016