Hey Gavin, is this an inherit limit of the API?

Tested in chrome stable #53.0.2785.143 on Linux Ubuntu 14.04, with the provided files. But we are not getting the response. Attaching the screen shot for reference.

@mwills:

Could you please let us know if we have missed anything and if possible, provide us with a screen cast of the issue which would help us triage further.

Thanks in Advance.

Deleted: 653630.png 45.6 KB

	653630.png 45.6 KB View Download

Looks like you've run into an issue with the SimpleCache backend used by CacheStorage. It doesn't allow an entry to be greater than 1/8th the total max cache size. The max cache size for SimpleCache is 2GB since it takes an unsigned int max size, so the max size for an entry is 268.5MB.

See https://cs.chromium.org/chromium/src/net/disk_cache/simple/simple_entry_impl.cc?cl=GROK&q=simple_entry_im&sq=package:chromium&l=418 for the code.

Gavin: is this intended behavior for this API?

At a minimum I would expect the Cache.put promise to be rejected and nothing to be stored. This also applies to Cache.add when the response is too big

We should alter SimpleCache to skip the 1/8th check for APP_CACHE backend types. WDYT gavinp@?

Where does the 2GB limit come from? Is there a reason that couldn't be removed as well?

The backend is initialized with an int32 max size, hence 2GB. It's possible to change that to an int64 or ignore it in APP_CACHE backend instances, but 2GB is roughly as much as the quota manager would allow at this point in most cases anyway.

Can I ask what your use case is?

Ah ok. I'm looking at building a web app that requires downloading of ~1.2GB zip files (android system images). I've been exploring options for actually being able to download and store the file outside of the window's heap and then request chunks of it. Although, the cache API doesn't appear to support this behavior unless I make all of the requests as range requests initially. I'm also exploring caching the response in indexdb, which seems to be working well.

 Issue 840417  has been merged into this issue.

I'm going steal this bug as I'm planning to work on the cache_storage backend to fix a number of functional issues.

Ideally I'd like to change the max_size arguments in disk_cache.h and the various backends to be int64_t.  Origins can now get access to quota sizes much larger than the 2GB permitted by a signed 32-bit integer.

Also, there are an increasing number of use cases on the web that need large amounts of storage.  Consider sites using large WASM binaries or games with large textures.

I don't see anything sensitive in this bug.

Re: sizes --- see also https://bugs.chromium.org/p/chromium/issues/detail?id=632397 (fixed to not be assigned...)

This CL:

https://chromium-review.googlesource.com/c/chromium/src/+/1194508

Fixes the test case from  bug 840417 :

https://jsbin.com/bamumi/edit?js,console

I need to add some tests before submitting for review.

Not sure if I can write a WPT for really large storage sizes.  Filed an issue to ask if its ok:

https://github.com/web-platform-tests/wpt/issues/12763

One thing I should mention that may be painful: increasing individual entry size limit (if you care about it) may increase the foot print of the in-memory index, which is somewhat undesirable... but there may be ways to finesse around it since the size is used for evictions (which CacheStorage won't need), and the size query methods (which CacheStorage might have a different way of performing).

Thanks for the information.  The per-entry size limit is 2GB, correct?

I think 2GB is probably big enough for now.  I've also been told the networking stack may have trouble producing responses greater than 2GB anyway.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4

commit 5a16ed0d987a0b4367c10894662a4ffbd70c3cc4
Author: Ben Kelly <wanderview@chromium.org>
Date: Fri Sep 07 21:28:09 2018

Support 64-bit max cache size in the simple disk_cache backend.

This change modifies the disk_cache API to support 64-bit cache sizes.
Individual entries are still limited to 32-bit sizes.  The blockfile
and memory backends will enforce a 32-bit cache size limit at runtime.
The simple backend is currently the only implementation that fully
supports 64-bit cache sizes.

R=jsbell@chromium.org, morlovich@chromium.org

Bug:  653630 
Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;luci.chromium.try:ios-simulator-full-configs;luci.chromium.try:linux_mojo
Change-Id: I9278b9e14e0a3253a66d20e57cf77026a462b4ac
Reviewed-on: https://chromium-review.googlesource.com/1194508
Commit-Queue: Ben Kelly <wanderview@chromium.org>
Reviewed-by: Eugene But <eugenebut@chromium.org>
Reviewed-by: Joshua Bell <jsbell@chromium.org>
Reviewed-by: Maks Orlovich <morlovich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#589667}
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/components/browsing_data/content/conditional_cache_counting_helper.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/components/browsing_data/content/conditional_cache_counting_helper.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/content/browser/cache_storage/cache_storage_cache.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/content/browser/cache_storage/cache_storage_cache.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/content/browser/cache_storage/cache_storage_cache_unittest.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/ios/chrome/browser/browsing_data/cache_counter.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/ios/chrome/browser/browsing_data/cache_counter.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/backend_unittest.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/blockfile/backend_impl.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/blockfile/backend_impl.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/disk_cache.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/disk_cache.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/disk_cache_test_base.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/disk_cache_test_base.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/memory/mem_backend_impl.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/memory/mem_backend_impl.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/simple/simple_backend_impl.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/disk_cache/simple/simple_backend_impl.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/http/mock_http_cache.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/http/mock_http_cache.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/net/tools/cachetool/cachetool.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/services/network/http_cache_data_counter.cc
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/services/network/http_cache_data_counter.h
[modify] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/third_party/WebKit/LayoutTests/NeverFixTests
[add] https://crrev.com/5a16ed0d987a0b4367c10894662a4ffbd70c3cc4/third_party/WebKit/LayoutTests/http/tests/cachestorage/large-put.html