[Build sheriff] I'll proceed to disabling the test, it seems this has flaked before as well.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a3c8726d32e3482d1f56cf5c97ae0d846bf4427d

commit a3c8726d32e3482d1f56cf5c97ae0d846bf4427d
Author: Karan Bhatia <karandeepb@chromium.org>
Date: Fri Mar 16 18:10:49 2018

Mark SafeBrowsingTest#testSafeBrowsingDontProceedNavigatesBackForSubResource as flaky.

TBR=ntfschr@chromium.org
BUG= 822753 

Change-Id: I0ff8b5d2b1a9d5abe930e80d8a21e886ab32c457
Reviewed-on: https://chromium-review.googlesource.com/966866
Reviewed-by: Karan Bhatia <karandeepb@chromium.org>
Commit-Queue: Karan Bhatia <karandeepb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#543762}
[modify] https://crrev.com/a3c8726d32e3482d1f56cf5c97ae0d846bf4427d/android_webview/javatests/src/org/chromium/android_webview/test/SafeBrowsingTest.java

 Issue 831229  has been merged into this issue.

Stack trace looks consistently like:

java.lang.AssertionError: Criteria not met in allotted time.
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:93)
	at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:172)
	at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:194)
	at org.chromium.android_webview.test.SafeBrowsingTest.waitForInterstitialToChangeTitle(SafeBrowsingTest.java:407)
	at org.chromium.android_webview.test.SafeBrowsingTest.testSafeBrowsingDontProceedNavigatesBackForSubResource(SafeBrowsingTest.java:663)
	at java.lang.reflect.Method.invoke(Native Method)

In both test cases, we call loadPathAndWaitForInterstitial() immediately preceding waitForInterstitialToChangeTitle. So, we're definitely on an interstitial page, but the interstitial hasn't changed the page title to what we expect (we search for the string "Security error").

There's a chance the flake is in the page-title-changing code, but this test would fail regardless for devices in non-English locales (the page title would be some translation of "Security error"). We can probably find a better way to verify backward navigation. Some thoughts:

 * AwContents#getUrl() (only works if Safe Browsing interstitials change this value)
 * Wait for visualStateCallback and assert the previous page is showing
 * Check that the interstitial page is detached, and that getUrl() returns the previous page instead of the target page

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d1cc79ae445056fafd7489eb05de213fca5b063

commit 1d1cc79ae445056fafd7489eb05de213fca5b063
Author: Nate Fischer <ntfschr@chromium.org>
Date: Mon Apr 16 23:43:29 2018

AW: deflake Safe Browsing tests

No change to production logic.

This aims to deflake
SafeBrowsingTest#testSafeBrowsingDontProceedNavigatesBackFor*.

These tests previously depended on the interstitial to change the page
title to the string "Security error." This is unreliable, however, as
chromium translates this for non-English locales.

This CL removes the dependency on page title, and asserts that the
previous page is visible after clicking "back to safety."

I couldn't locally reproduce test failures except by changing device
locale, but I expect this method to be more robust than relying on page
title. As mentioned below, I tested this ~200 times without failure, so
I don't expect many flakes.

This also removes the @FlakyTest annotation previously added.

Bug:  822753 
Test: run_webview_instrumentation_test_apk -f SafeBrowsingTest#testSafeBrowsingDontProceedNavigatesBackFor*
Change-Id: I91ded411514302c2c34fdb8e06f7694f58f7e526
Reviewed-on: https://chromium-review.googlesource.com/1013686
Reviewed-by: Bo <boliu@chromium.org>
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551173}
[modify] https://crrev.com/1d1cc79ae445056fafd7489eb05de213fca5b063/android_webview/javatests/src/org/chromium/android_webview/test/SafeBrowsingTest.java

I suspect this will improve flakiness. Previously, this would flake a few times in a week. I'll check again next week to see if flakiness is improved.

https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNydAsSBUZsYWtlImlvcmcuY2hyb21pdW0uYW5kcm9pZF93ZWJ2aWV3LnRlc3QuU2FmZUJyb3dzaW5nVGVzdCN0ZXN0U2FmZUJyb3dzaW5nRG9udFByb2NlZWROYXZpZ2F0ZXNCYWNrRm9yU3ViUmVzb3VyY2UM

https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNycgsSBUZsYWtlImdvcmcuY2hyb21pdW0uYW5kcm9pZF93ZWJ2aWV3LnRlc3QuU2FmZUJyb3dzaW5nVGVzdCN0ZXN0U2FmZUJyb3dzaW5nRG9udFByb2NlZWROYXZpZ2F0ZXNCYWNrRm9yTWFpbkZyYW1lDA

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d1cc79ae445056fafd7489eb05de213fca5b063

commit 1d1cc79ae445056fafd7489eb05de213fca5b063
Author: Nate Fischer <ntfschr@chromium.org>
Date: Mon Apr 16 23:43:29 2018

AW: deflake Safe Browsing tests

No change to production logic.

This aims to deflake
SafeBrowsingTest#testSafeBrowsingDontProceedNavigatesBackFor*.

These tests previously depended on the interstitial to change the page
title to the string "Security error." This is unreliable, however, as
chromium translates this for non-English locales.

This CL removes the dependency on page title, and asserts that the
previous page is visible after clicking "back to safety."

I couldn't locally reproduce test failures except by changing device
locale, but I expect this method to be more robust than relying on page
title. As mentioned below, I tested this ~200 times without failure, so
I don't expect many flakes.

This also removes the @FlakyTest annotation previously added.

Bug:  822753 
Test: run_webview_instrumentation_test_apk -f SafeBrowsingTest#testSafeBrowsingDontProceedNavigatesBackFor*
Change-Id: I91ded411514302c2c34fdb8e06f7694f58f7e526
Reviewed-on: https://chromium-review.googlesource.com/1013686
Reviewed-by: Bo <boliu@chromium.org>
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551173}
[modify] https://crrev.com/1d1cc79ae445056fafd7489eb05de213fca5b063/android_webview/javatests/src/org/chromium/android_webview/test/SafeBrowsingTest.java

The NextAction date has arrived: 2018-04-23

No flakes so far.

Bulk edit: marking stale 'fixed' bugs as 'verified' since they don't need verification at this point.