EarlGrey test for Settings fail if Settings item in Menu is off the viewport |
||||||
Issue description
Steps to reproduce:
(1) Ensure that the display is too short (vertical dimension) for "Settings" to be visible after tapping the "three dots" menu. For example, use the iPhone 6s Plus (iOS 10.2) simulator in landscape mode.
(2) Run any EearlGrey test suite which works on Settings (e.g. testAccessibilityOnSettingsPage).
Expected result:
The EGTest should be able to open Settings.
Actual result:
The EGTest fails with:
Exception Name: NoMatchingElementException
Exception Reason: Cannot find UI element.
Exception with Action: {
"Action Name" : "Tap",
"Element Matcher" : "(respondsToSelector(accessibilityIdentifier) && accessibilityID('kToolsMenuSettingsId'))",
"Recovery Suggestion" : "Check if element exists in the UI, modify assert criteria, or adjust the matcher"
}
Note: This affects many tests -- all of them which use just grey_accessibilityID(kToolsMenuSettingsId) to identify the Settings button. For short screens, the Settings button is initially off-screen. The EG test would need to scroll first, to get to it (see https://github.com/google/EarlGrey/blob/master/docs/api.md#selecting-off-screen-ui-elements).
I found this when testing https://codereview.chromium.org/2846253002/ manually (all CQ bots are green). While using the above linked trick would fix this issue, I wonder if we also need to ensure that some of the bots run in "short-screen" configuration? Or is this a known issue and we ignore it to keep the test complexity down?
I'm Cc-ing baxley@ as the EGTest master, and sdefresne@ and lpromero@ as ios/chrome/browser/ui/settings/OWNERS. I would love to hear your thoughts on this.
,
May 2 2017
Thanks! Once the helper is there, I'll be happy to use it in passwords-related EGTests. There is still the issue that none of our bots apparently runs in the landscape mode, because the bug has not been detected. Should we try to switch some bots to that, or is that a maintenance tradeoff we took to ignore landscape?
,
May 2 2017
I filed a bug to track running them in landscape mode (crbug.com/717548). I think the biggest concern is capacity, and figuring out the tradeoffs of the number of tests and how frequently to run them.
,
May 9 2017
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/888c1d23cef857d6b6f784f5499092485669e3c9 commit 888c1d23cef857d6b6f784f5499092485669e3c9 Author: Mike Baxley <baxley@chromium.org> Date: Fri May 12 20:56:45 2017 Fix AccountCollectionsTestCase to be more EarlGrey-like. Fix bugs in how menus are tapped when elements are not visible, and create shared matchers and utilities to tap menus. Also create shared matchers for sign in buttons. Bug: 698866 , 717163 Change-Id: I7ccdfbb39d359a7a3461a670961e720ed4706fd3 Reviewed-on: https://chromium-review.googlesource.com/505209 Commit-Queue: Mike Baxley <baxley@chromium.org> Reviewed-by: Eugene But <eugenebut@chromium.org> Cr-Commit-Position: refs/heads/master@{#471441} [modify] https://crrev.com/888c1d23cef857d6b6f784f5499092485669e3c9/ios/chrome/browser/ui/settings/accounts_collection_egtest.mm [modify] https://crrev.com/888c1d23cef857d6b6f784f5499092485669e3c9/ios/chrome/test/earl_grey/chrome_earl_grey_ui.h [modify] https://crrev.com/888c1d23cef857d6b6f784f5499092485669e3c9/ios/chrome/test/earl_grey/chrome_earl_grey_ui.mm [modify] https://crrev.com/888c1d23cef857d6b6f784f5499092485669e3c9/ios/chrome/test/earl_grey/chrome_matchers.h [modify] https://crrev.com/888c1d23cef857d6b6f784f5499092485669e3c9/ios/chrome/test/earl_grey/chrome_matchers.mm
,
Jun 29 2017
Several cases have been fixed, but there are still some tests that fail (Autofill and Passwords). A simple (but not perfect) way to test this is to run the settings egtests on the iPhone SE in landscape mode. Flipping to Available. In comment 3 a separate bug was filed to run our tests in landscape mode (at least periodically) to help find UI bugs, similar to the multitasking tests.
,
Jul 16 2017
,
Jul 16 2017
,
Jul 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/857e08190fa0f2b4a031b1ef98c74d4eebb52bc8 commit 857e08190fa0f2b4a031b1ef98c74d4eebb52bc8 Author: Vaclav Brozek <vabr@chromium.org> Date: Wed Jul 19 15:27:06 2017 Passwords settings EG tests: Scroll in short viewports While passwords_settings_egtest.mm already uses scrolling to find some elements, it is not applied consistently. This CL ensures that scrolling is applied everywhere where needed, including tapping the "Settings" button, so that the tests also pass on iPhone SE in landscape mode. Bug: 717163 , 744058 Change-Id: Ifbc3438c78d47441c67a0d538a3895e2ed4a39b5 Reviewed-on: https://chromium-review.googlesource.com/573041 Reviewed-by: Mike Baxley <baxley@chromium.org> Reviewed-by: Sylvain Defresne <sdefresne@chromium.org> Commit-Queue: Vaclav Brozek <vabr@chromium.org> Cr-Commit-Position: refs/heads/master@{#487871} [modify] https://crrev.com/857e08190fa0f2b4a031b1ef98c74d4eebb52bc8/ios/chrome/browser/ui/settings/passwords_settings_egtest.mm [modify] https://crrev.com/857e08190fa0f2b4a031b1ef98c74d4eebb52bc8/ios/chrome/browser/ui/settings/save_passwords_collection_view_controller.mm [modify] https://crrev.com/857e08190fa0f2b4a031b1ef98c74d4eebb52bc8/ios/chrome/test/earl_grey/chrome_matchers.h [modify] https://crrev.com/857e08190fa0f2b4a031b1ef98c74d4eebb52bc8/ios/chrome/test/earl_grey/chrome_matchers.mm
,
Sep 14 2017
I think we can mark this fixed. We'll track running all of the tests separately in 717548 |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by baxley@chromium.org
, May 1 2017Status: Started (was: Unconfirmed)