Floating action bar tests in ContentViewCoreSelectionTest are flaky |
||||||||||
Issue descriptiontestSelectActionBarPasswordPaste is very flaky on Lollipop Phone Tester For example, a fail, pass, fail set of 3 builds: https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/3334 https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/3335 https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/3336
,
Mar 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/df73c0f1ff73cd83009a654afa1d04561ef573d4 commit df73c0f1ff73cd83009a654afa1d04561ef573d4 Author: mvanouwerkerk <mvanouwerkerk@chromium.org> Date: Mon Mar 07 13:44:28 2016 Mark testSelectActionBarPasswordPaste as flaky. BUG= 592428 TBR=cblume NOTRY=true Review URL: https://codereview.chromium.org/1766363002 Cr-Commit-Position: refs/heads/master@{#379548} [modify] https://crrev.com/df73c0f1ff73cd83009a654afa1d04561ef573d4/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java
,
Mar 7 2016
When I worked on that other flaky test, I noticed these had the same problem and so I made a note to come back and fix them as well. They weren't flaky yet so it seemed like a bit of a lower priority. But it should be an easy fix. I'll go through and fix the rest. :)
,
Mar 8 2016
A lot of these selection tests in ContentViewCore fail on several bots: Marshmallow 64 bit Tester Marshmallow Tablet Tester Lollipop Tablet Tester in addition to Lollipop Phone Tester. First failing builds across a few bots: https://uberchromegw.corp.google.com/i/chromium.android/builders/Marshmallow%20Tablet%20Tester/builds/3059 https://uberchromegw.corp.google.com/i/chromium.android/builders/Marshmallow%2064%20bit%20Tester/builds/945 https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/3329 Across all of these, this is the oldest CL: https://chromium.googlesource.com/chromium/src/+log/64cdba3378dc100a9df0e5106cecad505db3634d They were all at least working on: 73513876700397ec4da77edcb929537a235a2a05 Which gives the following range of possible culprits: https://chromium.googlesource.com/chromium/src/+log/73513876700397ec4da77edcb929537a235a2a05~..64cdba3378dc100a9df0e5106cecad505db3634d These are the flaky tests (flaky at different times, more showed up after disabling testSelectActionBarPasswordPaste: testPastePopupClearedOnLongPressingOutsideInput testPastePopupClearedOnTappingEmptyInput testPastePopupClearedOnTappingNonEmptyInput testPastePopupClearedOnTappingOutsideInput testPastePopupDismissedOnDestroy testPastePopupNotShownOnLongPressingDisabledInput testPastePopupNotShownOnLongPressingNonEmptyInput testSelectActionBarPasswordPaste Marking them all as flaky in https://codereview.chromium.org/1766233004
,
Mar 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7ee6b4603f2f9813323c78c66813468d148479bf commit 7ee6b4603f2f9813323c78c66813468d148479bf Author: nyquist <nyquist@chromium.org> Date: Tue Mar 08 08:59:18 2016 Mark many ContentViewCoreSelectionTests flaky These are now marked as flaky: testPastePopupClearedOnLongPressingOutsideInput testPastePopupClearedOnTappingEmptyInput testPastePopupClearedOnTappingNonEmptyInput testPastePopupClearedOnTappingOutsideInput testPastePopupDismissedOnDestroy testPastePopupNotShownOnLongPressingDisabledInput testPastePopupNotShownOnLongPressingNonEmptyInput testSelectActionBarPasswordPaste BUG= 592428 TBR=nyquist NOTRY=true Review URL: https://codereview.chromium.org/1766233004 Cr-Commit-Position: refs/heads/master@{#379791} [modify] https://crrev.com/7ee6b4603f2f9813323c78c66813468d148479bf/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java
,
Mar 10 2016
nyquist@ mentioned a range of possible culprits. In that range I see: f351673 [Android] Add instrumentation test suites to chromium.android. by jbudorick ยท 7 days ago https://chromium.googlesource.com/chromium/src/+/f3516738fbd701a28abd8995ad17d3d143b6034f If I am reading this correctly, it sounds possible that the bots previously were not flaky on these tests because the bots previously were not running these tests. If so, the tests themselves may have potentially been flaky all along. At the end of the day, we would still want to have these tests. So we still need to cause the root of the flakiness. But it isn't necessarily within this range. I spoke to jbudorick@, who confirmed this. He added that the K bots seem stable, and the L bots are new. So it could potentially be either infra- related or specific to L+. It seems like a common theme in these flaky tests is a timeout waiting for a UI state. I'll continue to investigate why. I ran PastePopupNotShownOnLongPressingNonEmptyInput 101 times and had 97 successes and 4 failures. All 4 failures were related to a timeout waiting for UI state.
,
Mar 10 2016
I have a theory, which I'll try to confirm. The failing tests all seem to be failing because of a timeout waiting for a UI state. The process of waiting for the UI state is to poll and call a predicate to see if the situation has been satisfied. The problem (I believe) is in the predicate also returns false when the PopupPasteMenu is null. https://code.google.com/p/chromium/codesearch#chromium/src/content/public/android/java/src/org/chromium/content/browser/ContentViewCore.java&q=isPastePopupShowing&sq=package:chromium&type=cs&l=2563 If the PopupPasteMenu is null, the predicate will appear to never be satisfied. This can happen if we are perhaps under memory pressure and an allocation fails. I increased the timeout significantly to test if it would ever be satisfied and I had roughly the same fail percentage. So I believe the issue is that we have no way of indicating that the predicate cannot be satisfied. Tomorrow, I will investigate if we can check before getting into the predicate.
,
Mar 10 2016
To further add to this theory, there is a 1:1 mapping* of the flaky tests and a call to waitForPastePopupStatus(true); If my theory is correct, the true case will never happen because the element is null and so false is always returned. *except one test...the one I updated a while ago. I may have broken it in a different way.
,
Mar 13 2016
I have confirmed my theory and traced it down to the IPC message pump. I created a bug to get some help from the input team (https://bugs.chromium.org/p/chromium/issues/detail?id=594416). The sequence of events for the non-flaky and the flaky cases are almost the same. The non-flaky case gets 20x calls to InternalSwapCompositorFrame. Then it gets a call to OnShowPastePopup. The flaky case gets 19x calls to InternalSwapCompositorFrame. Then it gets a call to OnShowPastePopup. Then the 20th call to InternalSwapCompositorFrame comes in, which sets the textbox to not be editable, which destroys the insertion handles, which causes the test to not progress and eventually time out and flake. I have confirmed that all these messages are happening on the same thread an in the order they were received. So I suspect that there is a race condition with the messages being posted. This is where I stopped and asked for help from the input team.
,
May 26 2016
Pedro is looking into rewriting the action bar show/hide logic completely, which hopefully will let these tests be more reliable.
,
May 26 2016
Issue 605941 has been merged into this issue.
,
Sep 2 2016
,
Sep 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/611dec311f31ead13d9c8648febc34dad1409c38 commit 611dec311f31ead13d9c8648febc34dad1409c38 Author: aelias <aelias@chromium.org> Date: Thu Sep 22 23:39:49 2016 Disable flaky Android IME and selection tests. These tests (*all* Selection tests, and a few of the IME tests) are extremely flaky to the point that they often cause waterfall redness despite @RetryOnFailure. These areas are actively owned (amaralp@ owns Android selection, and changwan@/yabinh@ own Android IME) and we plan to circle back to reenable without @RetryOnFailure when they have been fully deflaked. The @FlakyTest annotation is deprecated so also replace existing uses of it with @DisabledTest, and secondly don't comment out the other annotations which seems to be unnecessary nowadays. BUG= 649411 , 628964 , 592428 Review-Url: https://codereview.chromium.org/2362013002 Cr-Commit-Position: refs/heads/master@{#420502} [modify] https://crrev.com/611dec311f31ead13d9c8648febc34dad1409c38/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java [modify] https://crrev.com/611dec311f31ead13d9c8648febc34dad1409c38/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java
,
Sep 23 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/59e3a9e0896916dee88888a692d1f2a4c94d3102 commit 59e3a9e0896916dee88888a692d1f2a4c94d3102 Author: aelias <aelias@chromium.org> Date: Fri Sep 23 22:59:37 2016 Actually disable testSelectionPreservedAfterReshown. I made a text editing mistake in http://crrev.com/2362013002 and only removed @RetryOnFailure for this particular test without adding @DisabledTest as intended, thus actually increasing bot redness. TBR=boliu NOTRY=true BUG= 649411 , 592428 Review-Url: https://codereview.chromium.org/2360333005 Cr-Commit-Position: refs/heads/master@{#420767} [modify] https://crrev.com/59e3a9e0896916dee88888a692d1f2a4c94d3102/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java
,
Oct 27 2016
BTW, 5 Ime tests are flaky for the same reason. These tests are: https://cs.chromium.org/search/?q=assertWaitForSelectActionBarStatus+file:imetest.java&sq=package:chromium&type=cs Here is an example: I 150.142s run_tests_on_device(06085ccf006236dd) Failure in testImeStaysOnLongPressingDifferentNonEmptyInputs: I 150.142s run_tests_on_device(06085ccf006236dd) junit.framework.AssertionFailedError: Values did not match. Expected: true, actual: false I 150.142s run_tests_on_device(06085ccf006236dd) at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:74) I 150.142s run_tests_on_device(06085ccf006236dd) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:112) I 150.142s run_tests_on_device(06085ccf006236dd) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:133) I 150.143s run_tests_on_device(06085ccf006236dd) at org.chromium.content.browser.input.ImeTest.assertWaitForSelectActionBarStatus(ImeTest.java:1641) I 150.143s run_tests_on_device(06085ccf006236dd) at org.chromium.content.browser.input.ImeTest.testImeStaysOnLongPressingDifferentNonEmptyInputs(ImeTest.java:758) I 150.143s run_tests_on_device(06085ccf006236dd) at java.lang.reflect.Method.invokeNative(Native Method) I 150.143s run_tests_on_device(06085ccf006236dd) at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) I 150.143s run_tests_on_device(06085ccf006236dd) at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) I 150.143s run_tests_on_device(06085ccf006236dd) at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) I 150.143s run_tests_on_device(06085ccf006236dd) at org.chromium.content_shell_apk.ContentShellTestBase.runTest(ContentShellTestBase.java:233) I 150.145s run_tests_on_device(06085ccf006236dd) at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161) I 150.145s run_tests_on_device(06085ccf006236dd) at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124) I 150.145s run_tests_on_device(06085ccf006236dd) at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191) I 150.145s run_tests_on_device(06085ccf006236dd) at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176) I 150.145s run_tests_on_device(06085ccf006236dd) at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554) I 150.145s run_tests_on_device(06085ccf006236dd) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701) https://build.chromium.org/p/chromium.linux/builders/Android%20Tests/builds/33823
,
Nov 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/abb5f9cb42bdfe9b13c71d02496131d24e96f6f3 commit abb5f9cb42bdfe9b13c71d02496131d24e96f6f3 Author: amaralp <amaralp@chromium.org> Date: Wed Nov 02 02:42:44 2016 Removed OnShowingPastePopup OnShowingPastePopup was a hack that is causing test flakiness. This CL removes it. BUG= 592428 Review-Url: https://codereview.chromium.org/2468043002 Cr-Commit-Position: refs/heads/master@{#429189} [modify] https://crrev.com/abb5f9cb42bdfe9b13c71d02496131d24e96f6f3/content/browser/android/content_view_core_impl.cc [modify] https://crrev.com/abb5f9cb42bdfe9b13c71d02496131d24e96f6f3/content/browser/renderer_host/render_widget_host_view_android.cc [modify] https://crrev.com/abb5f9cb42bdfe9b13c71d02496131d24e96f6f3/content/browser/renderer_host/render_widget_host_view_android.h [modify] https://crrev.com/abb5f9cb42bdfe9b13c71d02496131d24e96f6f3/content/public/android/java/src/org/chromium/content/browser/ContentViewCore.java
,
Nov 2 2016
ImeTest#testPastePopupsShowAndHide is also flaky and was disabled. crbug.com/661572
,
Nov 3 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8baf53f912098f36d71772ed240238520f7a6249 commit 8baf53f912098f36d71772ed240238520f7a6249 Author: amaralp <amaralp@chromium.org> Date: Thu Nov 03 17:46:07 2016 Re-enabling tests that seem to not be flaky These tests seem to not be flaky after crrev.com/2468043002 landed. I've run each of these tests 400+ times locally on my Nexus 6P (Android M) and ~40 times on my Nexus 4 (Android K) and had no failures. BUG= 592428 Review-Url: https://codereview.chromium.org/2471283003 Cr-Commit-Position: refs/heads/master@{#429640} [modify] https://crrev.com/8baf53f912098f36d71772ed240238520f7a6249/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java
,
Dec 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fa17096f47dcc992185f1a09de28034f17d33b5d commit fa17096f47dcc992185f1a09de28034f17d33b5d Author: amaralp <amaralp@chromium.org> Date: Sat Dec 17 02:28:46 2016 Set fields before calling ActionMode.invalidate() Previously mEditable/mIsPassword/mNeedsPrepare were set after calling invalidate(). This means that SelectionPopupController.onPrepareActionMode() sees the old values of those fields which is incorrect. This fix could reduce flakiness in the scenario where the IME thread updates after the action mode is created. BUG= 592428 Review-Url: https://codereview.chromium.org/2584773002 Cr-Commit-Position: refs/heads/master@{#439291} [modify] https://crrev.com/fa17096f47dcc992185f1a09de28034f17d33b5d/content/public/android/java/src/org/chromium/content/browser/SelectionPopupController.java
,
Jan 5 2017
yabinh@, is the problem outlined in Comment 15 still present in the ImeTests? It seems almost all the ImeTests have been deflaked.
,
Jan 6 2017
Checked the flaky dash board (http://test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&testType=content_shell_test_apk&tests=imetest) and (only) found one failure. Same problem with the one mentioned in comment 15. https://build.chromium.org/p/chromium.android/builders/Android%20N5X%20Swarm%20Builder/builds/7926 org.chromium.content.browser.input.ImeTest#testSelectionClearedOnKeyEvent (run #1): junit.framework.AssertionFailedError: Values did not match. Expected: true, actual: false at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:79) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:117) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:138) at org.chromium.content.browser.input.ImeTest.assertWaitForSelectActionBarStatus(ImeTest.java:1455) at org.chromium.content.browser.input.ImeTest.testSelectionClearedOnKeyEvent(ImeTest.java:1159) at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) at org.chromium.content_shell_apk.ContentShellTestBase.runTest(ContentShellTestBase.java:252) at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161) at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124) at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191) at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176) at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879)
,
Jan 18 2017
Found another flaky IME test with similar failure: select action bar doesn't show up after long pressing input node. See issue 682080 .
,
Feb 6 2017
New flake on org.chromium.content.browser.ContentViewCoreSelectionTest#testPastePopupClearedOnLongPressingOutsideInput See: https://build.chromium.org/p/chromium.android/builders/KitKat%20Tablet%20Tester/builds/6744
,
Feb 8 2017
,
Apr 18 2017
Omg all the tests are disabled! How are we doing feature work while there's zero test coverage?
,
Apr 18 2017
We chatted offline and Pedro will work on reenabling the tests soon, since the system should finally not have known race conditions after https://codereview.chromium.org/2785853002 lands (it's always been the plan to reenable after that, but it turned out to be blocked behind a lot of other things).
,
Oct 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dc805e132bdcce1fcbf2ed7aaa354ceeb1cbdaf7 commit dc805e132bdcce1fcbf2ed7aaa354ceeb1cbdaf7 Author: Pedro Amaral <amaralp@chromium.org> Date: Thu Oct 05 18:59:47 2017 Re-enabling ContentViewCore SelectionTests that used to be flaky Changes in how selection and the action mode are triggered seem to have deflaked these tests. Each of the re-enabled tests passed 60 times on a Nexus 6P running Android O and 60 times on a Nexus 4 running Android J. Bug: 592428 Change-Id: If774d474df13e9e20e3d714f06358811a941d075 Reviewed-on: https://chromium-review.googlesource.com/701278 Reviewed-by: Alexandre Elias <aelias@chromium.org> Commit-Queue: Pedro Amaral <amaralp@chromium.org> Cr-Commit-Position: refs/heads/master@{#506805} [modify] https://crrev.com/dc805e132bdcce1fcbf2ed7aaa354ceeb1cbdaf7/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java
,
Oct 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cccb6abbd27f80aeb53cf1897bdc9a18bd177622 commit cccb6abbd27f80aeb53cf1897bdc9a18bd177622 Author: Pedro Amaral <amaralp@chromium.org> Date: Sat Oct 07 00:36:54 2017 Fixing more Selection tests Re-enabling more selection tests. Ran each test 100 times successfully on Android O and Android J. I also removed some lines in |testCursorPositionAfterHidingActionMode| that were causing flakiness. Bug: 592428 Change-Id: Ic320f46cdf56db86bd5596151e24ee4defe632f4 Reviewed-on: https://chromium-review.googlesource.com/706338 Reviewed-by: Alexandre Elias <aelias@chromium.org> Commit-Queue: Pedro Amaral <amaralp@chromium.org> Cr-Commit-Position: refs/heads/master@{#507243} [modify] https://crrev.com/cccb6abbd27f80aeb53cf1897bdc9a18bd177622/content/public/android/javatests/src/org/chromium/content/browser/ContentViewCoreSelectionTest.java
,
Oct 27 2017
,
Nov 6 2017
,
Mar 13 2018
All but 2 ContentViewCoreSelectionTest are now enabled and the resulting 2 aren't flakes. Marking as fixed. |
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by mvanouwe...@chromium.org
, Mar 7 2016Status: Assigned (was: Untriaged)