From internal bug 71559890, android.webkit.cts.WebViewClientTest#testOnRenderProcessGone is also broken.

I will work on the version bisect and update bug later today.

Version bisect points to this range:

https://chromium.googlesource.com/chromium/src/+log/63.0.3232.0..63.0.3233.0?pretty=fuller&n=10000

trying per-cl bisect next

I'm having problems getting the archived webview apk to work properly as the webview provider on AOSP.  After deleting the system webview, then installing the archived webview.apk, the app shows up under settings->apps, but does not show up as a webview provider.  I see this in logcat when trying to start webview shell browser:

01-13 00:26:24.605  1133  1144 W WebViewUpdater: creating relro file timed out
01-13 00:26:24.606  2846  2846 E WebViewFactory: Chromium WebView package does not exist
01-13 00:26:24.606  2846  2846 E WebViewFactory: android.webkit.WebViewFactory$MissingWebViewPackageException: Failed to load WebView provider: No WebView installed
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.webkit.WebViewFactory.getWebViewContextAndSetProvider(WebViewFactory.java:383)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.webkit.WebViewFactory.getProviderClass(WebViewFactory.java:446)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.webkit.WebViewFactory.getProvider(WebViewFactory.java:257)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.webkit.WebView.getFactory(WebView.java:2608)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.webkit.WebView.setWebContentsDebuggingEnabled(WebView.java:2079)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at org.chromium.webview_shell.WebViewBrowserActivity.onCreate(WebViewBrowserActivity.java:161)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.app.Activity.performCreate(Activity.java:7016)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.app.Activity.performCreate(Activity.java:7007)
01-13 00:26:24.606  2846  2846 E WebViewFactory: 	at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1260)

This happens on both omr1 build as well as the master build of aosp android.  I'm using the archived M63 SystemWebview.apk (not Google webview).  Any ideas?

passing to next week's bugcop

What does the output of "adb shell dumpsys webviewupdate" say when you're trying to use the APK? That's the most important thing to debug any failure to load webview.

The release build archives have the opposite naming scheme to the source tree, however, so if that's the archive you're referring to then probably you're using the wrong one. The "system_webview_apk" target which builds "SystemWebView.apk" is archived as "AndroidWebviewAosp.apk", and the "system_webview_google_apk" target which builds "SystemWebViewGoogle.apk" is archived as "AndroidWebview.apk" (for unfortunate historical reasons)

Thanks torne@, that really helped to debug the issue.  It's because the target sdk version for the archived per-cl webview builds are set to 26 while the system was expecting at least 27.  I tried to use android 26 as the device-under-test, but then the Android 8.0 version of the test was always failing.  I'll do more investigation to see why this is happening.  Maybe able to get around this by build the latest version of the test with a lower min-sdk version, shall see.

The per-cl builds are presumably non-next builds then, so won't work in any situation where the combination of android and chromium version needs a next build, which is a significant proportion of the time :/

That's a problem for issues that only show up on recent OS versions..

Tested locally and I don't think that version bisect is accurate - it fails for me on 63.0.3232.0 as well. The repro rate isn't 100%, you may need to run it multiple times on each version.

From the build archive I got a range of 63.0.3216.3 to 63.0.3217.0.
I'm going to take a look at the commits there and try bisecting by building locally.

thanks torne@!

Getting inconsistent results from bisecting :/

Argh, okay, I should have guessed this is the problem: it's PlzNavigate changing the timing. With --disable-browser-side-navigation it works fine. So, yeah, I think we just need to fix the test to be less timing sensitive.

Unfortunately, my conclusion in comment 14 is only true for android.webkit.cts.WebViewTest#testJavascriptInterfaceCustomPropertiesClearedOnReload - that test is trivially fixed by a one line change to actually wait for the reload to happen instead of continuing immediately (which I'll land in CTS).

For testJavascriptInterfaceForClientPopup, the real problem is masked by the fact that EvaluateJsResultPollingCheck is badly implemented: when waiting for a JS result callback, it waits until *getting the result it wants*, instead of waiting for getting any result. This means that all mismatches between expected and actual values are reported as timeout errors, and don't show the actual value. I'm also correcting this in CTS, but doing so reveals that the problem here is that the JS interface *isn't injected* into the client popup window, and it's not a timing issue as far as I can tell; waiting doesn't help, and debugging it with chrome://inspect clearly shows the interface isn't there. Also, this test doesn't appear to fail on O devices, only on internal releases, which is weird.

testOnRenderProcessGone doesn't appear to be related, and I've unduped the internal bug for this - I'll deal with that on a separate bug.

Issue 782252 has been merged into this issue.

testOnRenderProcessGone is fixed already in M64 though it's not clear what the fix *was*. The test is extremely picky about the callbacks it receives and it was getting one it didn't expect. It passed on M63 if you disable PlzNavigate so I'm going to assume that one of our post-63 PlzNavigate fixes has also fixed this test and move on.

So, this just leaves testJavascriptInterfaceForClientPopup..

OK, I can now reproduce testJavascriptInterfaceForClientPopup on O MR1 as well (haven't tried any other OS versions). It appears to be a reliable failure, not a flake.

I've bisected down to 65.0.3309.0 - 65.0.3310.0 as the breaking range from release builds, and can reproduce this on O MR1 using the O MR1 version of CTS from https://source.android.com/compatibility/cts/downloads, with the command android-cts/tools/cts-tradefed run commandAndExit cts -m CtsWebkitTestCases -t android.webkit.cts.WebViewTest#testJavascriptInterfaceForClientPopup

Can I get a per-cl bisect for this separate failure? Hopefully it won't take too long starting with that range :)

Re c19, let me give it a try

bisected to this cl:
https://chromium-review.googlesource.com/c/chromium/src/+/828000

ahh the always edge case pop up windows.. we should add an instrumentation test to catch this in the future

Now adding a instrumentation test that does the basically same thing as the failing CTS test does..

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5690fe9f3546a1bc34b8e8a6cf4494729324729e

commit 5690fe9f3546a1bc34b8e8a6cf4494729324729e
Author: Jinsuk Kim <jinsukkim@chromium.org>
Date: Sun Jan 28 23:45:36 2018

Fix a bug failing Android webview CTS test

JavascriptInjector instance cached in |AwContents| was left
untouched upon destruction. This is fine for normal webview
but popup window whose WebContents is swapped out on loading
kept using the cached one, therefore caused a problem
restoring the injected Javascript interface objects into
the new WebContents. This CL fixes that by nulling it out,
and adds an instrumentation that does what the failing CTS
test does.

Bug:  800941 
Change-Id: I8fb50998fd461316d6fd4dc34f6a02f95856d24d
Reviewed-on: https://chromium-review.googlesource.com/886122
Reviewed-by: Bo <boliu@chromium.org>
Commit-Queue: Jinsuk Kim <jinsukkim@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532296}
[modify] https://crrev.com/5690fe9f3546a1bc34b8e8a6cf4494729324729e/android_webview/java/src/org/chromium/android_webview/AwContents.java
[modify] https://crrev.com/5690fe9f3546a1bc34b8e8a6cf4494729324729e/android_webview/javatests/src/org/chromium/android_webview/test/AwActivityTestRule.java
[modify] https://crrev.com/5690fe9f3546a1bc34b8e8a6cf4494729324729e/android_webview/javatests/src/org/chromium/android_webview/test/PopupWindowTest.java

Your change meets the bar and is auto-approved for M65. Please go ahead and merge the CL to branch 3325 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/40f96eb81dcbdf77aabc4945b30dfee34f4780cc

commit 40f96eb81dcbdf77aabc4945b30dfee34f4780cc
Author: Jinsuk Kim <jinsukkim@chromium.org>
Date: Tue Jan 30 00:05:03 2018

Fix a bug failing Android webview CTS test

JavascriptInjector instance cached in |AwContents| was left
untouched upon destruction. This is fine for normal webview
but popup window whose WebContents is swapped out on loading
kept using the cached one, therefore caused a problem
restoring the injected Javascript interface objects into
the new WebContents. This CL fixes that by nulling it out,
and adds an instrumentation that does what the failing CTS
test does.

Bug:  800941 
Change-Id: I8fb50998fd461316d6fd4dc34f6a02f95856d24d
Reviewed-on: https://chromium-review.googlesource.com/886122
Reviewed-by: Bo <boliu@chromium.org>
Commit-Queue: Jinsuk Kim <jinsukkim@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#532296}(cherry picked from commit 5690fe9f3546a1bc34b8e8a6cf4494729324729e)
Reviewed-on: https://chromium-review.googlesource.com/892218
Reviewed-by: Jinsuk Kim <jinsukkim@chromium.org>
Cr-Commit-Position: refs/branch-heads/3325@{#162}
Cr-Branched-From: bc084a8b5afa3744a74927344e304c02ae54189f-refs/heads/master@{#530369}
[modify] https://crrev.com/40f96eb81dcbdf77aabc4945b30dfee34f4780cc/android_webview/java/src/org/chromium/android_webview/AwContents.java
[modify] https://crrev.com/40f96eb81dcbdf77aabc4945b30dfee34f4780cc/android_webview/javatests/src/org/chromium/android_webview/test/AwActivityTestRule.java
[modify] https://crrev.com/40f96eb81dcbdf77aabc4945b30dfee34f4780cc/android_webview/javatests/src/org/chromium/android_webview/test/PopupWindowTest.java

Hello，

What should I do for android.webkit.cts.WebViewTest#testJavascriptInterfaceCustomPropertiesClearedOnReload fail ? Should I get waive from Google?