The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b89f0113f539b02c742c7eb1588767282dd0ddb6

commit b89f0113f539b02c742c7eb1588767282dd0ddb6
Author: yolandyan <yolandyan@chromium.org>
Date: Tue Dec 20 17:58:23 2016

Disable failing and flaky CTS tests

Disable WebViewClientTest testShouldOverrideUrlLoadingOnCreateWindow,
which has been flaky on L and N bot, and ServiceWorkerClientTest
testServiceWorkerClientInterceptCallback, which has been failing on N bot

BUG= 659077 , 675809

Review-Url: https://codereview.chromium.org/2590053002
Cr-Commit-Position: refs/heads/master@{#439840}

[modify] https://crrev.com/b89f0113f539b02c742c7eb1588767282dd0ddb6/android_webview/tools/cts_config/expected_failure_on_bot.json

There are two things. Looks like service worker callback stopped working starting in M54. The error is junit.framework.AssertionFailedError: Service Worker intercept callbacks not invoked

I also tried m53/m52, and looks there, service worker is working, but looks like the test is just bad and flaky. Most of the time getting this error below, but sometimes passing.
C    5.204s Main  [FAIL] android.webkit.cts.ServiceWorkerClientTest#testServiceWorkerClientInterceptCallback:
C    5.204s Main  junit.framework.ComparisonFailure: expected:<...s://www.example.com/[sw.js]> but was:<...s://www.example.com/[fetch.html]>
C    5.204s Main        at junit.framework.Assert.assertEquals(Assert.java:85)
C    5.204s Main        at junit.framework.Assert.assertEquals(Assert.java:91)
C    5.204s Main        at android.webkit.cts.ServiceWorkerClientTest.testServiceWorkerClientInterceptCallback(ServiceWorkerClientTest.java:166)
C    5.204s Main        at java.lang.reflect.Method.invoke(Native Method)
C    5.204s Main        at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:220)
C    5.204s Main        at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:205)
C    5.204s Main        at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C    5.204s Main        at junit.framework.TestCase.runBare(TestCase.java:134)
C    5.204s Main        at junit.framework.TestResult$1.protect(TestResult.java:115)
C    5.204s Main        at android.support.test.internal.runner.junit3.AndroidTestResult.runProtected(AndroidTestResult.java:77)
C    5.204s Main        at junit.framework.TestResult.run(TestResult.java:118)
C    5.204s Main        at android.support.test.internal.runner.junit3.AndroidTestResult.run(AndroidTestResult.java:55)
C    5.205s Main        at junit.framework.TestCase.run(TestCase.java:124)
C    5.205s Main        at android.support.test.internal.runner.junit3.NonLeakyTestSuite$NonLeakyTest.run(NonLeakyTestSuite.java:63)
C    5.205s Main        at junit.framework.TestSuite.runTest(TestSuite.java:243)
C    5.205s Main        at junit.framework.TestSuite.run(TestSuite.java:238)
C    5.205s Main        at android.support.test.internal.runner.junit3.DelegatingTestSuite.run(DelegatingTestSuite.java:103)
C    5.205s Main        at android.support.test.internal.runner.junit3.AndroidTestSuite.run(AndroidTestSuite.java:68)
C    5.205s Main        at android.support.test.internal.runner.junit3.JUnit38ClassRunner.run(JUnit38ClassRunner.java:103)
C    5.205s Main        at org.junit.runners.Suite.runChild(Suite.java:128)
C    5.205s Main        at org.junit.runners.Suite.runChild(Suite.java:27)
C    5.205s Main        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
C    5.205s Main        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
C    5.205s Main        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
C    5.205s Main        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
C    5.205s Main        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
C    5.205s Main        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
C    5.205s Main        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
C    5.205s Main        at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
C    5.205s Main        at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59)
C    5.205s Main        at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:272)
C    5.205s Main        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932)

my device is being hacked from a remote source using google chrome, tell me what files are needed to check

[Bulk edit]

Greetings from the release team!  This issue is marked as a stable release blocker for M56 on Android.  We're planning to ship our final beta release on Jan 25, so you have less than *two weeks* to fix this issue on trunk and merge the change back to the M56 branch.  Please prioritize working on this ASAP!

Sure this bug shouldn't block the release?  Remove the ReleaseBlock-Stable tag.

Don't think this bug should block the release, but not 100% sure?  CC me to the bug and let me know!

Cheers,
Alex

Is this a test issue or a real issue starting in M54 per comment 2?  In any case, should this still be a 56 stable blocker since it's been happening for a while now?

right looks like there is an issue with callback / flakyness from some time ago, so don't think this is a blocker for now..

I couldn't reproduce this on my O device with latest webview, however I've updated the test to include a proper clean-up with unregistering the service worker upon completion. It doesn't look like they are cleaned up explicitly by the tearDown/cleanUp in the test.. CL here: ag/2336567