If it wasn't clear, I'm using the WebView control from Android with Chrome as the WebView implementation (default).

@michaelmceuin: Could you please let us know where to on/off javascript? Screencast on reproducing the issue would help in further triaging.

Thanks!

Ok. After some investigation, this is what I know.

First off, we override the 'shouldInterceptRequest()', so we can make the HTTP call ourselves, since we need to attach our SSLContext and TrustManager (actually on this specific call, we use the Android System TrustManager).

With Javascript ON and using our override, the loading is REALLY SLOW or deadlocks. If I turn Javascript OFF, but same override, yahoo.com loads super fast. 

If I don't override 'shouldInterceptRequest()' (return 'null', to indicate WebView should make the call), wether Javascript is ON or OFF, it will load super fast.

So, I need to either have a fix for this scenario or at least a reasonable way to work around it until a fix is available. 

Thanks!

Michael

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

@ michaelmceuin: Could you please respond to comment#3, a screencast would really help in further triaging.

Thanks!

chelamcherla@chromium.org I did respond to comment#3...see comment#4.

To turn on/off javascript, I am calling the following WebSettings API:

webView.getSettings().setJavaScriptEnabled(true);

Once it deadlocks, I no longer receive calls to 'shouldInterceptRequest()'

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

chelamcherla@chromium.org link to screencast:

https://youtu.be/EhMWG9vXXqs

A few thoughts:

1.) It is not surprising that it takes longer with JavaScript enabled, more work needs to be done
2.) Consider using a Chrome Custom Tab instead of a WebView: https://developer.chrome.com/multidevice/android/customtabs.
3.) removing it from the JavaScript queue, because this is an integration/embedder question for WebView

hablich@chromium.org 

1) Yes, but if JS is enabled, but WebView loads it, it is quick. So, it's not specifically related to running JS, but having it ON, triggers some issue when the app loads it's own URL's. This is true, even if the app loads the URL and then tells the web view to load it itself. So, the mere act of making a network call, seems to make it SUPER DUPER SLOW!.

2) I have not looked at this, does it support 'shouldInterceptRequest()'? If not, it would not work for our solution, since we need to send URL's through our own SSLContext and TrustManager.

3) Can you expand on this? I'm not sure what you mean by this.

Thanks!

Michael

3.) is about internal bug routing.

The other two questions are best answered by WebView or CustomTabs people.

Hi,

Can you capture a Chrome trace, as explained here? http://www.chromium.org/developers/how-tos/trace-event-profiling-tool/recording-tracing-runs#TOC-Capture-a-trace-from-Chrome-on-Android-with-DevTools-

This way we can analyze the delays, which may be caused by contention on some thread, from what you're describing.

Thanks!

Sounds like JS might be loading resources, which triggest shouldInterceptRequest. Depending on what you do in that body (you'll need to share a sample app), you might be tying up its background thread with the input streams.

Needs-feedback for the sample app (and if possible, a chrome trace as requested in comment #14).

lizeb@chromium.org 

Here is the trace that you requested.

Additionally, the first time I tried to capture trace, the WebView locked up (could not stop load, did not stop on its own) and the trace tool would not complete. I had to close the WebView app and try it again. This time the page completed (albeit slowly) and the trace attached is the result.

Deleted: trace_subrosa_webview_with_js_enabled.json.gz 6.3 MB

trace_subrosa_webview_with_js_enabled.json.gz 6.3 MB Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

ntfschr@chromium.org

I've attached the trace requested in comment#14 (comment#16).

EVERY load in our app is handled through the 'shouldInterceptRequst()' call, since we need to make the call ourselves, so we can attach our own 'trust manager' and 'key store'.

In our app, we NEVER let WebView do the url/resource load.
 

Thanks for the trace.

Looking at it quickly, it seems quite clear that we spend a *lot* of time in shouldInterceptRequest(). Using the trace tools, I see 44s of wall clock time (it's on thread pool, using 4 threads) for 113 requests, but only 3.6s of CPU time.
Unfortunately we don't have any visibility into what's going on inside these tasks.

A few other observations:
- The renderer seems quite active, is the page CPU intensive?
- Towards the beginning, there is some descheduling of resource fetching for ~600ms. Is the main Android application doing CPU intensive work?


Overall:
- shouldInterceptRequest() can process up to 4 requests at a time, on a thread pool (on this device).
- Slowness seems to come from there, as total time spent in it is 44s

This could be caused by:
- Descheduling
- Waiting for IO in shouldInterceptRequest()

Normal resource fetching in WebView is highly asynchronous, which we somewhat lose with this call.

lizeb@google.com

Is there a way to handle what we do in 'shouldInterceptReqeust()', via a 'shouldInterceptRequestAsync()' or a similar interface? It seems that if WebView offered this, that would solve a lot of issues.

Also, it seems that having JS enabled, exacerbates the slowlness issue and creates deadlocks, as I've mentioned before.

Thanks!

Michael

The long-term solution to your problem would be to stop intercepting requests - nothing we can do there will ever make it perform as well as if you leave the chromium network stack to behave as normal.

The key thing there would be to understand *exactly* why you have to make the requests yourself and what APIs you would need to simply stop doing that. Are you just looking to change SSL connection parameters? Can you show us exactly what you're doing and explain it?

Making shouldInterceptRequest asynchronous is not really going to help a lot - we still have to block the network stack waiting for your response. Request interception is designed to be used when you have a local resource already available as a substitute/cached copy/whatever - using your own network stack is not intended.

torne@chromium.org

So, I did a trace, using the same code, loading the same page (yahoo.com), with the ONLY change being made, is to comment out this (default is false):

webView.getSettings().setJavaScriptEnabled(true);

When I did this, the page loads SUPER quick. This includes me loading ALL the resources through 'shouldInterceptRequest()'....

Hopefully this contrast will reveal the issue...

Thanks!

Michael

Deleted: trace_JS_turned_off (1).json.gz 1.0 MB

trace_JS_turned_off (1).json.gz 1.0 MB Download

Thanks for the other trace. One reason is that pages load way fewer resources without Javascript enabled. Looking at the trace, this is quite clear that there are fewer resources.

Looking at devtools on desktop, for yahoo.com it seems to load ~3x fewer resources. As a consequence, the queuing we see in the other trace is reduced, I suspect this is the main reason for pages loading faster without JS.

Overall, it looks that the root of the issue here is shouldInterceptRequest() taking time, and as torne@ said this is not meant to be used to run a parallel network stack.

Sorry, I know this doesn't really help. I am not a WebView expert though, so I cannot comment on the rest of the issue.

lizeb@chromium.org

My point was/is that calling this function 'setJavaScriptEnabled(true)', triggers the issue of slowness and deadlocks. 

Is there a way to use JavaScript (including javascript interface use, etc.), without the slowness and deadlocks, but way of (at least) a workaround to the issue (which I don't know exactly know what the issue is, except that it's triggered by call 'setJavaScriptEnabled(true)').

BTW, just to note here again, our app MUST do the load in the 'shouldInterceptRequest()' (is there another way to do this?).

Thanks!

Michael

That doesn't appear to be true, though. Without JS enabled you are effectively loading a different web page, with many fewer resources on it. Comparing the two is not really relevant.

You said above that if you don't intercept the requests it works fine even with JS enabled: the problem is caused by intercepting the requests.

The ideal solution as I said in #21 would be to stop intercepting the requests, but since you need to do unusual things with the network traffic that's not an option right now. That's why I asked you if you can explain in detail what you actually need to do, because we are currently trying to design new network request APIs that cover more application use cases specifically to *eliminate* the need for apps to make network requests themselves in shouldInterceptRequest, and we aren't familiar with your use case, so we need more information to make sure that we include this in our plan.

In the meantime: some performance cost is expected if you intercept requests, and there is no workaround for that. However, there shouldn't be any deadlocks, and the slowness you're seeing isn't necessarily *just* the unavoidable overhead of request interception and the resulting queueing.

If you see deadlocks the most likely cause is that your application code (the code you're using to implement shouldInterceptRequest and to implement the InputStream you return) is responsible: we've seen this many times before in many different apps, so this seems much more likely to us than a bug in WebView (though it's possible).

Your interception code also may be causing an *additional* performance cost on top of the overhead on our side that we can't avoid.

We'd need a minimal repro case, with source code, to look into that and either suggest what you're doing wrong, or find any possible bug on our side.

torne@chromium.org

Ok, understood.

In regards to number of resources loaded...I know that having JS off will be slightly quicker, since not loading JS files. But the contrast shouldn't be 'that' large.

I agree that there 'shouldn't' be any deadlocks, but it sometimes happens...usually on a more complicated page (like yahoo.com). Thankfully most of the pages we load, are not 'that' complicated. So, usually we aren't seeing the deadlocks there, but definitely the slowness, when JS is enabled (webView.getSettings().setJavaScriptEnabled(true);)


Here is our use case...we need to run all network requests through our code, so that we can do (mostly) (3) things.

- Provide our own KeyStore
  - This hooks into our software to provide Keys from CAC and PIV smart cards (mostly)
- Provide our own Trust (based on URL/Host)
  - We use our trust for a number of sites (mostly US gov. sites and some special cases)
  - We use the default 'Android' trust for everything else (like yahoo.com, google.com etc)
- Custom SSLContext which hooks to OpenSSL (to provide secure connections with FIPS 140-2)

I'll see what I can do about getting together test code. However, our code base is quite large and involved. It might be hard to duplicate the issue with simpler code, since I don't even know what is causing the issue (super slow and deadlocks)...But, I'll give it a shot :-)

Thanks!

Michael

One more thing, regarding our use case. That would apply to requests coming out of JavaScript as well (otherwise we are NOT FIPS 140-2 compliant).

Having Javascript turned off also means that all resources that are added to the page *by javascript* won't be loaded, which on many modern web pages is the majority of the content. lizeb@ noted above that it loads 3x as many resources with JS enabled, in the trace - that's a huge difference, not just "slightly quicker". Since the threads used to run shouldInterceptRequest are limited to a fixed number, performance may drop off significantly and suddenly if the number of requests that need to be done in parallel is more than that; it's not necessarily linear in the total number of resources.

What I mean by "there shouldn't be any deadlocks" is there isn't really a plausible mechanism I'm aware of for the deadlock to be caused by the WebView code, at all, and so it's much more likely that it's caused by your shouldInterceptRequest/InputStream code, because we've seen that happen many times before. i.e. I don't think this is actually a webview bug. With a repro case we can figure that out, one way or the other.

Separately from this issue I'd like to see a sample of the network request/SSL stuff you are doing, because I don't think we actually have the expertise on our team currently to deduce exactly what's involved just from your description there, but we're aware that this kind of thing is done by other apps as well and we should try to support it.

Incidentally: WebViewClient.shouldInterceptRequest is not actually called for all network requests, even now; websocket connections and some media playback don't use it, and network requests from Service Workers are only interceptable using ServiceWorkerController, not on the WebView itself (and only on later versions of webview/android, as this API was added more recently).