Lots of more failing tests on these bots:
https://uberchromegw.corp.google.com/i/chromium.android/builders/Marshmallow%20Tablet%20Tester
https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester

 Issue 614671  has been merged into this issue.

 Issue 614347  has been merged into this issue.

These are still flaking pretty regularly. Any plans to disable for now?

5 out of the last 150 Android Tests (dbg) runs (that weren't purple) have failed on webview tests with this flag.

The Marshmallow Tablet Tester is pretty much always red on webview test failures.

No. Those bots aren't affect devs on a regular basis. We have bigger problems: crbug.com/604568

We definitely can't unilaterally disable all multiprocess tests. If someone wants to investigate if there is a common root cause, then we can look into fixing that. Maybe it's the same thing as  crbug.com/617173  ?

Let's see if we can fix the individual failing tests in a reasonable time. Disabling all tests should be the last resort.

> If someone wants to investigate if there is a common root cause, then we can look into fixing that.

Nothing obvious. Just random tests timing out :/ Same thing seem to be happening on the trybots after swarming

 Issue 614318  has been merged into this issue.

This makes sheriffing work much more difficult because the webview tests frequently flake. If it affects devs at all (regular basis or not), then that's bad. Are these running on bots in the CQ?

I'm okay with not disabling them yet, but they should be investigated asap and disabled or marked as flaky for now if a solution can't be reached quickly.

Updating title. 

> Are these running on bots in the CQ?

Nope.

> I'm okay with not disabling them yet, but they should be investigated asap and disabled or marked as flaky for now if a solution can't be reached quickly.

But who has the time :p

Ok, more seriously, there are a few thing wrong here.
* cq and main waterfall are clearly higher priority than chromium.android, and I'm actively monitoring those in crbug.com/604568.
** found there are "bad" devices that causes over half of the flakes
** found test retry logi in webview tests is broken
* Most sheriffs don't know what should be done on test failures on chromium.android. And some of this is probably up for debate. Assuming failures are on chromium.android, but are not on cq/main waterfall
** Do not use @DisabledTest. That unnecessarily removes coverage from cq
** Do not use @FlakyTest. That's the same thing as @Disable upstream
** From it's kinda debatable what needs to happen
*** File a bug probably, but search for existing ones first
*** If test flaked once, maybe just ignore it?
*** If test fails consistently, disable it for that (subset) of bot only. There are annotations to disable test based on things like sdk level and things like low memory device etc. cq only runs on n5/kitkat, so just make sure don't include that as well
*** Maybe actually investigate what's wrong. Correlate with other bots etc


So.. I can't find any flakiness dashboard for chromium.android. Looking at the last few builds, clearly AwContentsClientFullScreenTest needs to be disabled; I enabled those last friday because they got disabled everywhere. Other ones it's not so clear. Hopefully they'll clear up after the test retry logic is fixed.

Android Tests and Android Tests (dbg) have flakes too, and they're in the "high priority" bucket in go/clankdash. They're also in the chromium.linux waterfall.

I do agree that the Marshmallow Tablet Tester and Lollipop Tablet Tester are lower priority, but android_webview_test_apk fails pretty consistently on Marshmallow Tablet Tester.

Maybe something good for a sheriff with webview expertise to dig into.

Not having a flakiness dashboard for chromium.android is a problem. I had assumed they'd be on the upstream dashboard since they're public builders, but apparently not. Android Tests and Android Tests(dbg) are in the upstream dashboard:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=android_webview_test_apk

main waterfall and cq covered by crbug.com/604568. Nothing immediately actionable.

Would it make sense to merge this with 604568 or do we suspect that the underlying reasons the tests are flaky on the main waterfall are different from the reasons they're flaky on the tablet testers?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/026c56773f0ba0057f8b2b5a15adfa1f468c6576

commit 026c56773f0ba0057f8b2b5a15adfa1f468c6576
Author: boliu <boliu@chromium.org>
Date: Tue Jun 07 00:06:08 2016

aw: Disable full screen tests on M

Look to just be super flaky. Failure on different test each time.

BUG= 615483 

Review-Url: https://codereview.chromium.org/2046773002
Cr-Commit-Position: refs/heads/master@{#398158}

[modify] https://crrev.com/026c56773f0ba0057f8b2b5a15adfa1f468c6576/android_webview/javatests/src/org/chromium/android_webview/test/AwContentsClientFullScreenTest.java

> Would it make sense to merge this with 604568 or do we suspect that the underlying reasons the tests are flaky on the main waterfall are different from the reasons they're flaky on the tablet testers?

Probably not. I don't think M bot has a bad device

 Issue 623921  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03c0a699233bc0c99d02ad0e7997813c17c90a0d

commit 03c0a699233bc0c99d02ad0e7997813c17c90a0d
Author: Bo Liu <boliu@chromium.org>
Date: Thu Sep 28 18:42:28 2017

aw: Run AwContentsClientFullScreenTest on M

These tests got disabled on M and above due to flakes. However
now webview tests no longer runs on K cq bot, meaning these tests
have no coverage on cq.

Tried running looping the 22 tests that are enabled here 50 times
locally without any failure. So enable them with RetryOnFailure
and see if there are any flakes on bots.

Bug:  615483 
Change-Id: I3bc20a1e1b89f3b925db3943441a5bd00efb9c9b
Reviewed-on: https://chromium-review.googlesource.com/688798
Reviewed-by: Paul Miller <paulmiller@chromium.org>
Commit-Queue: Bo <boliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#505100}
[modify] https://crrev.com/03c0a699233bc0c99d02ad0e7997813c17c90a0d/android_webview/javatests/src/org/chromium/android_webview/test/AwContentsClientFullScreenTest.java

again, some one should check back in a few days/a week to see how flaky these tests really are

Link to dashboard: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=webview_instrumentation_test_apk&tests=org.chromium.android_webview.test.AwContentsClientFullScreenTest

So far, so good. The only red spots are flukes (the entire webview_instrumentation_apk failed). I'll update this bug next time I check back.

These tests don't appear to be flaky any more, based upon Nate's observations on Oct 4, and mine now.

if they don't flake, then should remove RetryOnFailure

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9481a85958eed0e831c94bbe276e16b8c3b2394

commit c9481a85958eed0e831c94bbe276e16b8c3b2394
Author: Nate Fischer <ntfschr@chromium.org>
Date: Thu Nov 09 04:52:51 2017

AW: remove RetryOnFailure from tests

No change to production logic, only tests.

This removes RetryOnFailure from AwContentsClientFullScreenTest. This
test suite no longer appears to be flaky (ran 30 times locally, no
flakes).

Test run_webview_instrumentation_test_apk --break-on-failure --repeat 30 -f AwContentsClientFullScreenTest#*

Bug:  615483 
Change-Id: Ic245b723fbf1db19b6f2ac4f89713f542d2edb5b
Reviewed-on: https://chromium-review.googlesource.com/759560
Reviewed-by: Bo <boliu@chromium.org>
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#515101}
[modify] https://crrev.com/c9481a85958eed0e831c94bbe276e16b8c3b2394/android_webview/javatests/src/org/chromium/android_webview/test/AwContentsClientFullScreenTest.java

Closing this, please ping the bug if we see flakes (feel free to revert my CL).

Bulk edit: marking stale 'fixed' bugs as 'verified' since they don't need verification at this point.