Gustav, would you be able to take a look at this? It looks like you've dealt with this a bit in https://codereview.chromium.org/1155713005/ and ag/686457.

I can take a look at this first if it's test related, since I was fixing another CTS and set up scripts to run CTS tests multiple time, merge the result and bisect to handle flaky tests

Cool, thanks Yoland!

Still flaking mainly on L and N bot, looking into this,

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b89f0113f539b02c742c7eb1588767282dd0ddb6

commit b89f0113f539b02c742c7eb1588767282dd0ddb6
Author: yolandyan <yolandyan@chromium.org>
Date: Tue Dec 20 17:58:23 2016

Disable failing and flaky CTS tests

Disable WebViewClientTest testShouldOverrideUrlLoadingOnCreateWindow,
which has been flaky on L and N bot, and ServiceWorkerClientTest
testServiceWorkerClientInterceptCallback, which has been failing on N bot

BUG= 659077 , 675809

Review-Url: https://codereview.chromium.org/2590053002
Cr-Commit-Position: refs/heads/master@{#439840}

[modify] https://crrev.com/b89f0113f539b02c742c7eb1588767282dd0ddb6/android_webview/tools/cts_config/expected_failure_on_bot.json

Transition yoland@'s CTS bugs to aluo@.

So, what happened to the investigation here?

Do we have a link/pointer to the scripts that run CTS tests multiple times?

We should investigate whether this is still a problem after PlzNavigate.

> We should investigate whether this is still a problem after PlzNavigate.

Sounds like now is the right time to investigate this. Someone should try re-enabling the test to see if it still flakes.

Also, miscellaneous shouldOverrideUrlLoading improvements: see issue 855733

There's a report in buganiser that this test may now be failing 100% of the time for at least one vendor, instead of just being flaky, so we need to check the status and deal with it ASAP - if it's broken while we've had it disabled for being flaky that may actually be a regression in the real code :(

Looped the M version of this test on a nexus 5 and local trunk release build. Passed 200 times over lunch. Maybe I should flash back to L, according to the old bot. Or maybe it's some bad interaction between tests..

Flashing back to L bricked the device :(

Looking at the test code itself, there doesn't seem to be any guarantee that it works. Test loads frame a, has an iframe b, which opens popup c. It waits for onPageFinished, and then checks that shouldOverrideUrl for popup c is called.

Which doesn't seem like it guarantees being called.

(And then the rest of the test is commented out due to b/12804986. Maybe should fix that too..)

It's been awhile since I've built cts locally...

Fixing b/12804986 too

aluo: is there a bug to keep all the cts tests running, but don't turn the bot red for maybe flaky ones? maybe pull those out into a separate step on the bot that just warns or something? not running disabled tests at all is pretty bad. and maybe this doesn't actually flake anymore, it's just we don't have data to track this

Actually, anyone mind if I just enable this based on #12? What the test is doing (wait for onPageFinished then check url in shoudOverrideUrlLoading) is probably ok, so it's likely that the  issue 2  years ago was inside webview itself.

A lot of has happened since then, including PlzNavigate, and I guess/hope this doesn't happen anymore.

Upoaded that cl here: https://chromium-review.googlesource.com/c/chromium/src/+/1180513

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/47ae66897bb9826cc578cc9bcf99e3d997026069

commit 47ae66897bb9826cc578cc9bcf99e3d997026069
Author: Bo Liu <boliu@chromium.org>
Date: Wed Aug 22 21:12:50 2018

aw: Enable testShouldOverrideUrlLoadingOnCreateWindow

Did not flake locally.

Bug:  659077 
Change-Id: I8770032fb0944b32950987f5674583cf4b4241aa
Reviewed-on: https://chromium-review.googlesource.com/1180513
Reviewed-by: Richard Coles <torne@chromium.org>
Commit-Queue: Bo <boliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#585248}
[modify] https://crrev.com/47ae66897bb9826cc578cc9bcf99e3d997026069/android_webview/tools/cts_config/expected_failure_on_bot.json

bots are pretty happy