the@: Please provide sample test file/apk to test this issue from our end. Please let us know device details you are seeing this issue on, screencast on reproducing it would help in further triaging.

Thanks!

The WebView's maximum disk cache size is unfortunately hardcoded to 20MB.

I believe there was a change a while ago and now Chromium only allows resources to be cached if they are less than some percentage of the total cache size, to avoid one large resource kicking everything else out of the cache?

I'm not sure about the behaviour of the memory cache though. How are you determining that the file wasn't cached? Just inspecting with devtools? Do you actually see it load the file from the network multiple times, or do you just see the cache failure? (it may still have been loaded from the memory cache anyway)

Sounds like internal b/112255805

Ah, yes, that's the one, and I previously chatted with Maks here in the office about it which is where I was remembering those details from - I'll talk to him again later.

The size limit for individual resource for browser-process disk cache on Android is 1/8th of cache size ATM, which for 20 meg is about... 2.5 meg.
Strongly considering making it 1/4th.

Can't speak of the renderer memory cache. (There can also be a browser memory cache, in case of an incognitto profile --- don't know if webview can get one of those)

No, we don't have incognito profiles.

Raising it to 1/4 will make it less likely I guess, but it's still sort of arbitrary :| Maybe we will have to look at what it would take to calculate a sensible cache size dynamically..

Thanks a lot for looking into this issue! Due to our company policy, I cannot provide the apk file to you and I don't have a test apk yet :(

I have been using the Chrome DevTools to debug the webview issue. I am seeing the ERR_CACHE_WRITE_FAILURE from the Network tab for this large JS file with webview v69. But, I don't see the issue with webview v61 on the same device. Please see the attached two screenshots for the results with both versions on emulator API 27. The results are consistent in testing on multiple Android devices including Samsung S7 and emulator (API 27, 28). 

In addition to DevTools, I am also observing the same error in the WebViewClient callback onReceivedError() on this JS file.

Based on the Charles Proxy tool, I can tell the JS file is loaded only once from the network.

Deleted: chrome-69.png 161 KB

	chrome-69.png 161 KB View Download

Deleted: chrome-61.png 162 KB

	chrome-61.png 162 KB View Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Chatted with morlovich:

- We can raise one or more of various magic numbers, which will solve this particular instance of the issue if we raise them enough (until your JS file gets larger than the *new* limit), but have a bunch of other effects that aren't entirely predictable (increased disk space usage, worse cache behaviour, etc). We're looking at doing this as a mitigation.

- We could force the fetches to happen in series when it's too big for the cache's file size limit. However, this would make it quite likely that you'd end up *actually downloading the file multiple times* since if it's too big for the cache then.. it's not going to get cached, and the next one will have to fetch it again even if the resource was cacheable.

I'm not really an experienced web developer, so if I'm making bad assumptions here please do correct me as any different perspectives here would be valuable, but:

I'm guessing that you have a single large JS file because it contains ~all your app code, and you load the same file into multiple frames because you are assuming that it will be cached, and thus doing one big load of all the code is going to take less total bandwidth than having each frame load the subset of the code it actually uses?

If so then, unfortunately, that's exactly the thing that is going to provoke this problem. The alternative approach would be bundling the code into the right set of separate files such that each frame can load only the code it needs, but code used by multiple frames is still in shared js files. This has the downside that it's more separate network requests, but I generally believe that's less of a problem these days than it used to be (especially if you use HTTP/2) - and of course it may well require a ton of work on the code itself if it's not designed to work that way, which may well be prohibitive :|

We're going to see what we can do here both short and long term to improve things for this kind of case, but it doesn't seem likely that there's going to be a fix that has no downsides and is going to work no matter how large your monolithic JS file gets.

I'm going to take this bug for now and start an internal discussion about whether we can/should increase the overall webview cache size (I think we probably can at this point, but not just by hardcoding a larger number - Android had some suggestions on how to do this better). morlovich is looking at tweaking the per-file limit so that we're less likely to hit this case as well.

One thing I want to clarify re: discussion of changes in versions: I didn't realize this was comparing against 63 (rather than something more recent). While that version would indeed have exact same limit on individual cache entry size, IIRC, it would not try to handle multiple requests for it simultaneously, so it would fetch it multiple times rather than outright fail.

Oh, so if our old behaviour actually *was* to just download the file multiple times then... we should probably have that behaviour back somehow, but also that's a definite reason that app developers should try really hard to avoid this case occurring, no matter what version of webview. :)

The original submitter is from the organization owning the Android app that contains webview, but the web app being served as a 2.6MB .js file is my organization's. We have been working together on the issue so far.

Tome's assumption is pretty much spot on. The bulk of the app is contained in this one .js file, and since it is primarily loaded in non-webview mobile browsers (Chrome for Android and iOS, etc), the file is expected to be loading from cache when multiple instances are used. This is probably not ideal, but the development work to break it up would indeed be prohibitive in the short term.

Looking at the 'chrome-61.png' network trace, I think morlovich in comment 12 is correct. It does appear that the file wasn't loading from cache in version 63 either. While it would great from a performance standpoint if this larger file could be served from cache, the most immediate concern from our end was that multiple loads failed completely in versions 64+. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cf7efa63d334276da2756746e32675dbc947fad0

commit cf7efa63d334276da2756746e32675dbc947fad0
Author: Maks Orlovich <morlovich@chromium.org>
Date: Thu Oct 04 22:07:34 2018

SimpleCache: bump the individual resource limit on very small caches

Android WebView limits cache to 20MiB as of now, which resulted, before
this change, in a 2.5MiB size limit for individual resources... resulting in
parallel fetches to those resource failing due to lack of a cache entry to
back multiple readers.

This bumps the limit to at least 5MiB to mitigate somewhat.

Bug: 890973
Change-Id: Ic6b587414c85d47dbc62411d508446e33cb5cd17
Reviewed-on: https://chromium-review.googlesource.com/c/1262575
Reviewed-by: Asanka Herath <asanka@chromium.org>
Commit-Queue: Maks Orlovich <morlovich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#596878}
[modify] https://crrev.com/cf7efa63d334276da2756746e32675dbc947fad0/net/disk_cache/entry_unittest.cc
[modify] https://crrev.com/cf7efa63d334276da2756746e32675dbc947fad0/net/disk_cache/simple/simple_backend_impl.cc

Yeah, unfortunately I think you should start pursuing breaking up this JS code into smaller chunks in some way, either just simply splitting it into pieces and loading all of them, or bundling it per-page so that it loads only the needed bits, because as I said, even if we change a bunch of the parameters here that will only fix it for as long as your JS doesn't exceed the *new* limit, and even if we return to the pre-64 behaviour where it loads successfully but multiple times, the performance and bandwidth cost implications for users aren't great.

Any comments to also reverting back to the old behavior of re-downloading? Regardless of the page configuration I don't think the webview should not outright fail to load the resource. A more common case of this is multiple references to the same image on the same page.

It should be possible to do it for at least things with a content-length, planning to look at doing that this week...

How do I test this change with Android WebView - https://chromium-review.googlesource.com/c/chromium/src/+/1262575? Can I make canary as provider on my Android device?

Yes, if your device uses Android 7 or later you can install Chrome Canary and go into the Android developer settings to select Chrome Canary as the webview provider. The change is in 71.0.3571.0 so it hasn't gone to dev yet, but should be present in canary.

Is there already a UMA stat for the cache write failure error that we could check? If not, could we add one? (unless you're going to completely avoid this case, which it sounds like we can't in all cases?)

I've filed issue 893318 to track whether we want to increase the webview disk cache size, independently of this, so for now I'll give this to morlovich@ for #18.

SimpleCache.Http.WriteResult2 == "Over Max Size", but note that it only causes trouble in the simultaneous-fetches case, and of course this is for Chrome, not WebView.

UMA logs in net/ should also be being logged for WebView, but I guess what we would actually want is logging for the network error, not the cache write result directly, since as you say it's not always a problem..

Well, at any rate they don't show up on the main uma dashboard.
As for logging the network error... sadly it's not distinguishable, since the disk cache just returns ERR_FAILED for it and a few zillion other things.

Unfortunately WebView currently reports to a separate set of tables for UMA and as such doesn't appear on the main dashboard: https://uma.googleplex.com/p/android_webview/histograms

The stat is there, and over max size is 00.00% so this isn't happening commonly.

If devtools is able to report net::ERR_CACHE_WRITE_FAILURE as the load result then surely it is distinguishable? Or are there just other causes of cache write failures that would be mixed up in that?

The other things in that enum that don't have "success" in the name would also result in net::ERR_CACHE_WRITE_FAILURE

Kinda curious as to why "bad state" is so common, actually, that might indicate some other bug...

@torne@chromium.org - It seems Canary worked and it did resolve the issue reported by OP. I am wondering how I can build chromium and install customer provider to test on Android device. Is it possible?

It seems our app regressed and offline stopped working on latest Chrome's webview. I see there is the 20MB limit - https://cs.chromium.org/chromium/src/android_webview/browser/net/aw_url_request_context_getter.cc?q=aw_url_request_context_getter.cc&dr

I wanted to tweak that see if increasing that limit fixes the issue. Any other suggestions are welcomed too.

I think we may need to tread with caution in this case. (Finch + UMA experiment to check the impact of cache hit ratio?) In the meantime, could salesforce use a workaround? Would splitting JS / compiling JS help?

Custom webview providers can only be used on custom builds of android, or by modifying the system image used by the android emulator - they don't work on regular retail devices, where the packages/signatures that are accepted is a fixed list. See https://www.chromium.org/developers/how-tos/build-instructions-android-webview for some alternatives, though that page is kinda old and some of those might not quite work any more :|

It doesn't sound like your problem is necessarily the same as the one discussed here, so I suggest you file a new bug explaining more detail about exactly what your app is doing and in which version it stopped working.

Yes, just having smaller JS files would avoid the problem, as I suggested in #11/16

We understand the point of having smaller JS files. But as mentioned in #14, salesforce provides the WebView container in our Android app, the actual web app comes from our customer organization. 

As mentioned in #22, it seems there is some implementation difference between Chrome and WebView. Is it the reason why we don't see the issue with the same web app on mobile Chrome browser?

Your customer should make their JS smaller :)

There's no implementation difference - the same cache behaviour exists in both. It's just that Chrome's cache size is typically larger than webview's, so the per-file limit is also larger since it's a fraction of the total cache size. Chrome would have the same problem if the files were bigger.

torne@chromium.org: Sure I'll file a separate issue. It seems our app is ~12MB << 20 MB limit. There was no issue on Android System WebView as a provider but with Chrome 69 app fails to load in offline. 

Hello!  Is this Fix through Canary at this point?  I have a customer that is impacted and they want to get an idea when this will be GA.  Thank you

The limit change is in following versions:
Canary 71 71.0.3571.0
Dev 71 71.0.3574.0

(There is also a follow up one that's out for review..)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/780531da89c747e5f0359de180957bbb143b2133

commit 780531da89c747e5f0359de180957bbb143b2133
Author: Maks Orlovich <morlovich@chromium.org>
Date: Mon Nov 12 19:51:05 2018

HttpCache: make handling of parallel fetches over cache size limit less bad

Previously they would straight up fail. With this cache, such fetches
with specified content-length will "merely" be serialized and re-fetched.
(Which is what we did in <= M63).

Bug: 890973
Change-Id: I1d718cb83caced15d7ea44633a056a3fae0ac936
Reviewed-on: https://chromium-review.googlesource.com/c/1280792
Reviewed-by: Victor Costan <pwnall@chromium.org>
Reviewed-by: Shivani Sharma <shivanisha@chromium.org>
Commit-Queue: Maks Orlovich <morlovich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607309}
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/content/browser/cache_storage/cache_storage_cache_unittest.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/blockfile/backend_impl.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/blockfile/backend_impl.h
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/disk_cache.h
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/memory/mem_backend_impl.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/memory/mem_backend_impl.h
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/simple/simple_backend_impl.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/simple/simple_backend_impl.h
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/disk_cache/simple/simple_entry_impl.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/http/http_cache.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/http/http_cache.h
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/http/http_cache_unittest.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/http/mock_http_cache.cc
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/net/http/mock_http_cache.h
[modify] https://crrev.com/780531da89c747e5f0359de180957bbb143b2133/tools/metrics/histograms/enums.xml

Hello: Is there any visibility your able to provide to when this will be in the "stable" release channel?  My customer is launching in Jan and wants to know if they still will have to face this limitation for their Android users.  Thank you 

The increase in limit to 5MiB is in M71, which should be out in early December per:
https://chromiumdash.appspot.com/schedule
The workaround to serialize the fetches is in M72, which should be out about end of January.

Thank you for the feedback.  I'm not that familiar with the process.  Can you give me an approximate timeline for this to be available in a release that would be in the Play Store?  Thanks again!

The dates Maks mentions in #38 are the approximate timeline for when the changes will be released to regular users via the play store.