These were also flaking as late as yesterday on the KK bots:

https://uberchromegw.corp.google.com/i/chromium.android/builders/KitKat%20Tablet%20Tester/builds/3553
https://uberchromegw.corp.google.com/i/chromium.android/builders/KitKat%20Tablet%20Tester/builds/3554

C  256.381s Main  [FAIL] org.chromium.content.browser.input.ImeTest#testPastePopupShowAndHide:
C  256.381s Main  junit.framework.AssertionFailedError: Criteria not met in allotted time.
C  256.381s Main  	at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:74)
C  256.381s Main  	at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:112)
C  256.381s Main  	at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:133)
C  256.381s Main  	at org.chromium.content.browser.input.ImeTest.waitAndVerifyUpdateSelection(ImeTest.java:1116)
C  256.381s Main  	at org.chromium.content.browser.input.ImeTest.testPastePopupShowAndHide(ImeTest.java:981)
C  256.381s Main  	at java.lang.reflect.Method.invokeNative(Native Method)
C  256.381s Main  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
C  256.381s Main  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
C  256.381s Main  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C  256.381s Main  	at org.chromium.content_shell_apk.ContentShellTestBase.runTest(ContentShellTestBase.java:232)
C  256.381s Main  	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
C  256.381s Main  	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
C  256.381s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C  256.381s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C  256.381s Main  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
C  256.381s Main  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)

+changwan +aelias

Just a cursory look at the test, but its setUp call checks for matching a page scale factor of 1.0.  I've had nasty luck with that check failing before.  Changing the test to look for a 1.25 instead of a 1.0 seems to make it a lot less flaky.  Going to package this up and send for review to see if that makes the bots happier.

The first recent failure on Ime(Lollipop)Test is starting around Sun-Mon last week on all builders (summarized below).
Did we have any change around the moment? (My big change landed one more week before so I'm hoping it's not the direct cause... :|)

https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Tablet%20Tester/builds/2329
Mon Mar 21 18:54:50 2016
blame: 382472-382439

https://uberchromegw.corp.google.com/i/chromium.android/builders/KitKat%20Tablet%20Tester/builds/3488
Mon Mar 21 21:35:25 2016
blame: 382504-382474

https://build.chromium.org/p/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/2800
Sun Mar 20 20:17:41 2016
blame: 382244-382240

https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Tablet%20Tester/builds/2329
Mon Mar 21 18:54:50 2016
blame: 382472-382439

Er, might as well continue the discussion here instead of that CL.

The CL I sent out seems to fix a bunch of the tests, but results in 
org.chromium.content.browser.input.ImeTest#testSelectActionBarClearedOnTappingOutsideInput
flaking out.  When it doesn't pass, the radio button moves upwards after a "click", but it doesn't seem like the web page registers that the radio button gets clicked.

This might be a side-effect of looking for a slightly larger version of the page (at 1.1 instead of 1.0).

That said, my sheriff shift is just about up, so I'm not planning on disabling anything.  It'd be really, really nice if this were fixed up, though; it's been extremely rare for the tablet bot to be happy.

It seems to me that this has nothing to do with setup, as some of the failures occur from the middle of tests, and all the flaky tests are related to selection popup window. My initial suspicion is that we changed floating popup window behavior recently and regressed, but it somehow did not get caught by SelectPopupTest. Still flashing my device.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2f60b340b4082340c921c7b3a2fefb56e930aa5b

commit 2f60b340b4082340c921c7b3a2fefb56e930aa5b
Author: changwan <changwan@chromium.org>
Date: Tue Mar 29 08:10:56 2016

Deflake two IME tests

In testSelectionClearedOnKeyEvent and
testKeyboardNotDismissedAfterCopySelection,
we first click on the text, and then long-press the text. This call
sequence seems to cause flakiness depending on the timing.

Since we can trigger context popup menu even when the input form is not
focused, we don't need the first clicks for these tests.

Also change a bit more to make the tests extra robust.

TBR=aelias@chromium.org
BUG= 598482 

Review URL: https://codereview.chromium.org/1842663002

Cr-Commit-Position: refs/heads/master@{#383687}

[modify] https://crrev.com/2f60b340b4082340c921c7b3a2fefb56e930aa5b/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

FYI, I could not reproduce ImeTest.testImeCopy failure, which failed once at
https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Tablet%20Tester/builds/2428 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9d26048c16cf69b760f595e5045469a9117692ab

commit 9d26048c16cf69b760f595e5045469a9117692ab
Author: changwan <changwan@chromium.org>
Date: Tue Mar 29 10:24:20 2016

Deflake another IME test

In testPastePopupShowAndHide test, we don't need to click and long press
again. I suspect that this is causing flakiness on KK flo buildbot.

TBR=aelias@chromium.org
BUG= 598482 

Review URL: https://codereview.chromium.org/1842613003

Cr-Commit-Position: refs/heads/master@{#383697}

[modify] https://crrev.com/9d26048c16cf69b760f595e5045469a9117692ab/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

cblume@ had investigated testPastePopupShowAndHide in http://crbug.com/594416, it sounded like that change wasn't sufficient to remove the flakiness in local testing though.

I also suspected that it is more deep-rooted. My point of suspicion was long pressing twice or long pressing immediately after clicking, so I just tried to avoid them if possible (which do not belong to IME tests anyways), and it seems that most of the test failures on L / KK tablet buildbots cannot be observed any more except testImeCopy and testImeCut (whose repro rate is very low and I could not find the culprit.)

Chris, please feel free to take this over.

Another test failure spotted: ImeTest#testImeSelectAndUnSelectAll:

https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Tablet%20Tester/builds/2457

 Issue 604494  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8b12c85c1af2fcab6b0f04d201d4d45144a34ea9

commit 8b12c85c1af2fcab6b0f04d201d4d45144a34ea9
Author: mikecase <mikecase@chromium.org>
Date: Tue Apr 19 21:05:14 2016

Mark ImeTests SelectAndUnSelectAll and PastePopupShowAndHide as Flaky

Marking two ImeTests that have flakily failed a several bots as
flaky. These tests are ImeTest#testImeSelectAndUnSelectAll and
ImeTest#testPastePopupShowAndHide

BUG= 598482 

Review URL: https://codereview.chromium.org/1902853002

Cr-Commit-Position: refs/heads/master@{#388306}

[modify] https://crrev.com/8b12c85c1af2fcab6b0f04d201d4d45144a34ea9/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

We are also seeing a lot of flakes for ImeTest#testImeCut on linux_android_rel_ng

e.g. 

junit.framework.AssertionFailedError: Criteria not met in allotted time.
	at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:74)
	at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:112)
	at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:133)
	at org.chromium.content.browser.input.ImeTest.waitAndVerifyUpdateSelection(ImeTest.java:1128)
	at org.chromium.content.browser.input.ImeTest.testImeCut(ImeTest.java:542)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
	at org.chromium.content_shell_apk.ContentShellTestBase.runTest(ContentShellTestBase.java:232)
	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)


https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/57903

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/75f1acb7364bcd6faed4bb241ce0e6879910f1b4

commit 75f1acb7364bcd6faed4bb241ce0e6879910f1b4
Author: dgn <dgn@chromium.org>
Date: Mon Apr 25 13:18:15 2016

Mark ImeTest#testImeCut and ImeTest#testImeCopy flaky

Many flakes on those tests have been observed, on the
waterfall and on the trybots.

BUG= 603991 , 598482 
TBR=aelias@chromium.org

Review URL: https://codereview.chromium.org/1916783002

Cr-Commit-Position: refs/heads/master@{#389461}

[modify] https://crrev.com/75f1acb7364bcd6faed4bb241ce0e6879910f1b4/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/75f1acb7364bcd6faed4bb241ce0e6879910f1b4

commit 75f1acb7364bcd6faed4bb241ce0e6879910f1b4
Author: dgn <dgn@chromium.org>
Date: Mon Apr 25 13:18:15 2016

Mark ImeTest#testImeCut and ImeTest#testImeCopy flaky

Many flakes on those tests have been observed, on the
waterfall and on the trybots.

BUG= 603991 , 598482 
TBR=aelias@chromium.org

Review URL: https://codereview.chromium.org/1916783002

Cr-Commit-Position: refs/heads/master@{#389461}

[modify] https://crrev.com/75f1acb7364bcd6faed4bb241ce0e6879910f1b4/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

 Issue 603991  has been merged into this issue.

Detected 4 new flakes for test/step "content_shell_test_apk (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLgsSBUZsYWtlIiNjb250ZW50X3NoZWxsX3Rlc3RfYXBrICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "content_shell_test_apk (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLgsSBUZsYWtlIiNjb250ZW50X3NoZWxsX3Rlc3RfYXBrICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

When we call InputConnection#setSelection and other methods together,
the order of messages sometimes does not get kept. Thus some tests
could be flaky.

We can solve this by enabling IME thread, because when IME thead is on,
the order can be kept. See this cl: https://codereview.chromium.org/2058633004/

Another way is applying the following patch, which deals with the
re-ordering issue: https://codereview.chromium.org/1877073003/. Since this patch has been landed, the flakiness doesn't exist any more.

Let's also remove all the @FlakyTest annotations before closing this.

Detected 3 new flakes for test/step "content_shell_test_apk (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLgsSBUZsYWtlIiNjb250ZW50X3NoZWxsX3Rlc3RfYXBrICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6f23392b1f1f2b91909f646bc5ddf54c2c4003a6

commit 6f23392b1f1f2b91909f646bc5ddf54c2c4003a6
Author: yabinh <yabinh@chromium.org>
Date: Thu Jun 16 07:09:45 2016

There are 6 flaky tests in ImeTest. They are flaky because when we call
InputConnection#setSelection and other methods together, the order of messages
sometimes does not get kept.

We can solve this by enabling IME thread, because when IME thead is on,
the order can be kept.

Another way is applying the following patch, which deals with the re-ordering
issue: https://codereview.chromium.org/1877073003/. Since the patch has been
landed, we can remove the @FlakyTest annotations.

BUG= 598482 ,  603991 

Review-Url: https://codereview.chromium.org/2058633004
Cr-Commit-Position: refs/heads/master@{#400099}

[modify] https://crrev.com/6f23392b1f1f2b91909f646bc5ddf54c2c4003a6/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/186c15876ee97b4d0be6302c0635814c873448f5

commit 186c15876ee97b4d0be6302c0635814c873448f5
Author: grunell <grunell@chromium.org>
Date: Thu Jun 16 14:21:14 2016

Revert of Remove  all the @FlakyTest annotations in ImeTest (patchset #2 id:20001 of https://codereview.chromium.org/2058633004/ )

Reason for revert:
Flaky failure seen in

https://build.chromium.org/p/chromium.linux/builders/Android%20Tests%20%28dbg%29/builds/34878

Original issue's description:
> There are 6 flaky tests in ImeTest. They are flaky because when we call
> InputConnection#setSelection and other methods together, the order of messages
> sometimes does not get kept.
>
> We can solve this by enabling IME thread, because when IME thead is on,
> the order can be kept.
>
> Another way is applying the following patch, which deals with the re-ordering
> issue: https://codereview.chromium.org/1877073003/. Since the patch has been
> landed, we can remove the @FlakyTest annotations.
>
> BUG= 598482 ,  603991 
>
> Committed: https://crrev.com/6f23392b1f1f2b91909f646bc5ddf54c2c4003a6
> Cr-Commit-Position: refs/heads/master@{#400099}

TBR=changwan@chromium.org,yabinh@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 598482 ,  603991 

Review-Url: https://codereview.chromium.org/2070083003
Cr-Commit-Position: refs/heads/master@{#400151}

[modify] https://crrev.com/186c15876ee97b4d0be6302c0635814c873448f5/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

Detected 3 new flakes for test/step "content_shell_test_apk (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLgsSBUZsYWtlIiNjb250ZW50X3NoZWxsX3Rlc3RfYXBrICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f667465a379f9d74f97590399fe7aff97e7a637b

commit f667465a379f9d74f97590399fe7aff97e7a637b
Author: yabinh <yabinh@chromium.org>
Date: Fri Jun 17 07:09:16 2016

Remove all the @FlakyTest annotations in ImeTest & fix another flaky test

The CL
https://codereview.chromium.org/2058633004/
was reverted because another CL is flaky:
https://codereview.chromium.org/2033723003/

There is race condition in the latter CL. Firstly, we copy some text
(running on UI thread), then we copy other text (running on test thread).
At last we paste the text. But we are not sure which text is in the
clipboard because of race condition. We can fix this by not copying the
former text, because it is not necessary originally. After fixing it,
we can remove the @FlakyTest annotations safely.

BUG= 598482 ,  603991 

Review-Url: https://codereview.chromium.org/2076853002
Cr-Commit-Position: refs/heads/master@{#400386}

[modify] https://crrev.com/f667465a379f9d74f97590399fe7aff97e7a637b/content/public/android/javatests/src/org/chromium/content/browser/input/ImeTest.java

Detected 3 new flakes for test/step "content_shell_test_apk (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLgsSBUZsYWtlIiNjb250ZW50X3NoZWxsX3Rlc3RfYXBrICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "content_shell_test_apk (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLgsSBUZsYWtlIiNjb250ZW50X3NoZWxsX3Rlc3RfYXBrICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.