Relabeled, it would be really handy if each of the caches in the Cache Storage UI showed total bytes :)(

I'm not sure what you are comparing to what :) The usage we are showing is based on the internal representation of data, so it might differ with the actual resources size. But in your case I also suspect you forgot to multiply by two or something.

In that case can you please sanity check what I'm doing before closing? "Internal representation" doesn't matter to users if they can't trust the tools.

I provided repro steps. The only manual part is summing the resources in the 3 caches that chromestatus creates. Hence the request: it would be really handy if each of the caches in the Cache Storage UI showed total bytes" :)

BTW, this report comes an issue reported on https://github.com/GoogleChrome/puppeteer/issues/1596#issuecomment-351841882.

> I provided repro steps.

Your steps are to compare some arbitrary numbers of unknown origin with the usage report from the tool. These numbers have nothing to do with the usage as far as I understand.

> "Internal representation" doesn't matter to users if they can't trust the tools.

The numbers provided by internal representation will be compared against the quota. If you need to know the sizes of blobs that you store, you can sum them up using the web facing API. But those sizes would not matter because of multibyte strings, internal compression, deduping taking place in the browser, etc.

> Hence the request: it would be really handy if each of the caches in the Cache Storage UI showed total bytes" :)

That's not what you ask in the bug, in fact this is the first time you bring it up. You can file it separately, but due to ^^ it is likely to also be not considered for implementation.

Please don't change the status of the bugs the team triages, please instead comment on the bug and we will consider reopening it.

I'm the user from the linked issue re puppeteer who was (and still is) having problems.  I'm still struggling to get the reporting tools to behave as I would expect.  

I've documented some of the issues in the following post on stackoverflow:  https://stackoverflow.com/q/47846914/4070848... in summary:

(a) hitting the Clear Site Data button doesn't move the red Cache Storage amount to zero... except in an Incognito window where it works just fine

(b) the info in the table in the Cache Storage (service worker cache) section is incomplete... size and date not filled in for most items

I'm willing to accept that the issue might lie in my code... the demo (with non-minified code) is linked in the post.  For example, regarding (b), the info in that table seems to be properly populated in another demo I found at https://mdn.github.io/sw-test/ .. not sure why it isn't in my demo.

I'm using Version 63.0.3239.108 (Official Build) (64-bit) on Windows.

I have some steps that are (kinda) reproducible to show what I consider to be incorrect cache reporting:

(a) open an INCOGNITO window (this seems to behave much more cleanly for cache stats reporting... see my comment above)
(b) copy and paste https://cloud3squared.com/files/sw-stackoverflow-demo/index.html into the URL bar and hit enter
(c) hit F5 twice to load all resources fully
(d) note the "using" value: for me it is 11.84 MB at this point (this matches what is reported in DevTools for Cache Storage stat)... see attached Stage1.jpg
(e) open a new tab in the same Incognito window
(f) paste the same URL into the URL bar and hit enter
(g) hit F5 to refresh
(h) for me, the "using" value is now up to 23.88 MB ... basically *double* what it was before... see attached Stage2.jpg
(i) hit F12 to open devtools
(j) Application tab > Clear Storage section > Clear Site Data button (with all options checked)
(k) for me: the pie chart is now blank again, 0B used... see attached Stage3.jpg
(l) hit F5 twice to reload
(m) pie chart fills up again... but at a much lower level

The above may not be exactly what you see.  When I tried again later, it wasn't quite the same again.  But what was the same was that the cache stats were all over the place.  It's really inconsistent.

But mainly what troubles me is the doubling of the cache value when loading in the second window, as observed in step (h).  And this is very similar to what I'm observing in puppeteer (in my real-world application), where I will screenshot a page in a node app via puppeteer (with very similar service worker as per the above demo), so that resources are cached for next time.  It reports that e.g. 4 MB of space is used.  If I reload immediately, it still reports 4 MB.  But if I leave it five minutes, then reload, the "storage used" value goes up to 8 MB.  Leave another 5 mins and reload... gone up to 12 MB.  And so on.  This very soon leads to a "quota exceeded" error on subsequent attempts (storage is limited in my IDE).

Deleted: Stage1.JPG 35.6 KB

	Stage1.JPG 35.6 KB View Download

Deleted: Stage2.JPG 33.0 KB

	Stage2.JPG 33.0 KB View Download

Deleted: Stage3.JPG 10.8 KB

	Stage3.JPG 10.8 KB View Download

More peculiar behaviours (at least from a user perspective... there may or may not be sound internal reasons for this)... the following is again on a desktop (Windows) environment with Chrome 63.0.3239.108:

(a) open a new Incognito window (with devtools CLOSED)
(b) browse to https://cloud3squared.com/files/sw-stackoverflow-demo/index.html
(c) starts showing "using 0 B"
(d) hit F5 repeatedly, noting the usage... for me it's 0 B, 7.91 KB, 12.57 MB, 12.57 MB, 12.57 MB, 1.57 MB, 1.57 MB (etc...)

In other words, it goes up to 12.6 MB, and then DOWN to 1.6 MB after a few refreshes.

Why is that??

Now try the above again, but immediately after opening the Incognito window, hit F12 to open devtools (but no need to do anything in devtools... just open the panel).  Then do the remaining steps above (load page, F5 repeatedly).

Now, for me, the sequence is 0 B, 7.91 KB, 12.57 MB, 12.57 MB, 12.57 MB, 12.57 MB (etc...)

In other words, with devtools open, the value goes up to and then STAYS at 12.6 MB and does *not* drop.

Why is that??

Since this issue seems to be marked as closed, I've opened a new issue which just references the posts above:

https://bugs.chromium.org/p/chromium/issues/detail?id=795582

+jsbell@ for insights on cache storage api internals.

pfeldman@: I reopened b/c multiple users are reporting the same wonkiness and the triage in https://bugs.chromium.org/p/chromium/issues/detail?id=795134#c2 doesn't include details on what was investigated. BTW, always happy to add more info when bugs aren't clear. Just ask or add the Needs-Feedback label. 

drmrbrewer@, thanks for sticking with this. I see the "doubling" that you're talking about on that site. After some additional testing I'm seeing similar unstablness in Chrome 65.

TLDR; 
There's a slightly different repro scenario is below. DevTools increases size of the Cache Storage pie segment every time you navigate to chromestatus.com and the service worker was stopped. So for example, when the user navigates back to the page at a later date, DT reports a higher usage than before. There shouldn't be any increase b/c chromestatus.com only caches requests during `install`.
---

Chrome Version: 65.0.3295.0 
Platform: Mac

Steps to repro:
(1) Open a tab. Go to https://www.chromestatus.com/features
(2) Open DevTools > Application panel > Clear storage in left nav. Note the size that DT reports for "Cache Storage".
(3) Stop the service worker (either in chrome://serviceworker-internals/, or by clicking "stop" next to its status in the Service Workers panel).
(4) Close the tab (important so the page isn't controlled by the sw).
(5) Open a new tab. Load https://www.chromestatus.com/features
(6) Repeat (2). Note that there's a size increase for "Cache Storage".
(7) Run the snippet below verifies the mismatch.

What is the expected result?
The breakdown for Cache Storage usage reflects what's actually stored by sw caching.

You can run this script in the console to verify a site's total cache size:

```
async function getCacheStoragesAssetTotalSize() {
  const cacheNames = await caches.keys();

  let total = 0;

  const sizePromises = cacheNames.map(async cacheName => {
    const cache = await caches.open(cacheName);
    const keys = await cache.keys();
    let cacheSize = 0;

    await Promise.all(keys.map(async key => {
      const response = await cache.match(key);
      const blob = await response.blob();
      total += blob.size;
      cacheSize += blob.size;
    }));

    console.log(`Cache ${cacheName}: ${cacheSize} bytes`);
  });

  await Promise.all(sizePromises);

  return `Total Cache Storage: ${total/1e6} MB`;
}

await getCacheStoragesAssetTotalSize(); 
```

What happens instead?

DT displays an increasing "Cache Storage" total. cache_storage_size.png shows the mismatch after running the repo steps a few times.

Note: for me, clicking the "Clear site data" removes the caches but leaves a lingering Cache Storage label in the UI (see clear_site_data.png screenshot). The other storage breakdowns (IDB, service worker) are removed correctly.

---

If this is WAI, I guess we need to understand a few things and probably document the details for developers:

1. What does the "Cache Storage" pie segment represent (exactly)? e.g. resources on disk, in-memory, multibyte strs, compression, deduping?
2. Why does devtools disagree with the storage and quota APIs?
3. Why does the "Cache Storage" total increase when revisiting chromestatus or opening drmrbrewer@'s page more than once.

Perhaps these are all related. Can anyone help answer these?

Thanks ericbidelman@ for sticking by me in my hour of need!  This has been driving me crazy... I haven't quite been able to pinpoint a simple repro of my real-world scenario, but all of these strange behaviours seem related to it.  I think your chromestatus repro is good, and very close to what I'm seeing in my real-world app (using puppeteer) where the "cache used" amount just rises inexorably (in chunks of e.g. 5MB) with every new load, until eventually I am prevented from saving anything more to service worker with a "quota exceeded" error.

Are you able to comment on the fact that on my page (https://cloud3squared.com/files/sw-stackoverflow-demo/index.html) the service worker cache contents table is not fully populated, whereas with another demo (https://mdn.github.io/sw-test/) it is... all sizes and dates are properly filled in?  See attached.  Possibly it's unrelated, but it's another unsettling thing... particularly when I'm not yet knowledgeable enough to understand whether it's an error I'm making, or whether it's a bug in the devtools page.  That said, I note that the table for the Chrome Status page (https://www.chromestatus.com/features) is similarly not properly populated.

Deleted: my-sw-cache-table.jpg 42.7 KB

	my-sw-cache-table.jpg 42.7 KB View Download

Deleted: sw-sw-cache-table.jpg 46.7 KB

	sw-sw-cache-table.jpg 46.7 KB View Download

Deleted: chromestatus-sw-cache-table.jpg 48.3 KB

	chromestatus-sw-cache-table.jpg 48.3 KB View Download

ericbidelman@ the more I try the repro of your comment #9 the more I'm convinced it demonstrates most closely what I see in my real-world application.  When I load my real page in Chrome desktop, stop the SW, close the tab, load the page in a new tab, etc etc, the storage used goes up consistently each time by the same amount, exactly like I see in puppeteer on node.js.  

I'm willing to accept an explanation that this isn't really "lost" quota, because the resources have been marked as "available if required" or similar, but:

(a) it's really confusing
(b) more importantly it has a real knock-on effect in a real application (mine) when I hit the quota limit and am prevented from saving anything further to the service worker cache

Incidentally, I also note that when I load https://www.chromestatus.com/features in an Incognito window, and continually hit F5 to reload, the Cache Storage value in devtools keeps rising too, on each reload, without even having to stop service worker, close tab,  etc... this doesn't happen in a non-Incognito window.

P.S. ericbidelman@ isn't this issue still marked as closed?  What I'm looking at reads:

Status:	WontFix
Owner:	----
Closed:	Dec 15

Another observation: for the repro in your comment #9, instead of step 3 (stop the service worker) you can just proceed to step 4 (close the tab)... it seems that the service worker for this domain remains "activated and running" for a short time (maybe a couple of minutes), and then its status changes to "activated and stopped".  So it seems to auto-stop after a short time without having any controlled clients.

Again, this corresponds very closely to the behaviour I see in puppeteer (https://github.com/GoogleChrome/puppeteer/issues/1596) with ever-increasing quota used... I find there that if I run the puppeteer script again immediately (to load a page, screenshot it, then close it), the quota used stays the same, but if I leave it a few minutes and then run it, the quota used has increased another notch.  So it seems that the service worker remains "running" for a short time, and when it has auto-stopped, the next page load will result in a higher reported quota used.

So a workaround for my purposes is to load a "dummy" page when initialising puppeteer, which just stays open and does nothing... except ensure that the service worker remains active AND running at all times.  With this workaround, the quota usage remains level (*).  That workaround mimics what you see in desktop Chrome... if you leave a page open that is loaded to https://www.chromestatus.com/features, and then do the repro of comment #9 with the "dummy" page always open in a tab, the Cache Storage value remains constant.

(*) BUT I think I'm still being hit by the "doubling" issue reported in comment #6 (and confirmed by ericbidelman@ in comment #9) because occasionally the storage used will leap up by a chunk, and not come back down.

Hopefully there is enough information in this issue now to prompt some sort of fix.

Other users also seeing Cache Storage weirdness and unexpected Quota Exceeded errors: https://bugs.chromium.org/p/chromium/issues/detail?id=796060#c3

I'm also seeing Cache Storage weirdness and eventually get sw.js:1 Uncaught (in promise) DOMException: Quota exceeded.

Yep and I'm now seeing it in the context of webviews on Android, where I can't really use the workaround of my comment #15 because I can't keep a dummy webview page open indefinitely on Android merely to prevent the service worker cache getting out of control.

I think this needs to be investigated by the Chromium team with some urgency, though at present this issue is still marked as "WontFix... Closed".

Related to comment #17

I'm using SW-Toolbox and it crashes the entire SW on load because it causes the error and doesn't handle it properly. 

It's not the Chromium's responsibility that this "3rd party" (in quotes because Google) library isn't broken but, since I suspect that A LOT of websites use it, this issue should get high prio?

See this bug on sw-toolbox's github:
https://github.com/GoogleChromeLabs/sw-toolbox/issues/277

Additionally: I'm trying to detect the issue (and/or prevent it) and clear the Cache Storage:

caches.delete(key1)
.then(caches.delete(key2))
.then(/* etc */)

but Devtools reports that Cache Storage is still exceeding its limits after a refresh :(

#9: The only increasing storage usage I see when using the page linked to from that comment is IndexedDB usage, not CacheStorage usage. I believe that it's a separate issue, and (if it's a bug) it needs to be filed separately.

For cachestorage, can you first verify that you are not storing any uncacheable cross-origin URLs with opaque responses and intentionally-varying query strings and/or paths (for instance, Google Analytics reporting GIFs) in the cache?

If your cache currently contains such responses, the observed growth pattern without corresponding disk usage is expected. Mitigation steps for a security vulnerability require reported cache usage for such responses to be padded by randomly-varying (multiple-megabyte) amounts. The best solution is likely either to avoid caching these at all, or to automatically prune older ones from your cache.

bsittler@ : for me, the steps of #9 produce an increase in *Cache Storage* of around 1.5 MB for each run through the steps.  Reliably.  Every time.

Are you stopping the service worker, closing the tab, then loading the chromestatus page again in a new tab?

Furthermore, there are no opaque responses in the cache... I'm also following the related  bug 796060  (indeed I submitted that one too), and the output of your code snippet at https://bugs.chromium.org/p/chromium/issues/detail?id=796060#c30 produces *zero* in this example.  The incremental increase of 1.5MB is also less than the 7MB increase you might expect if a new opaque resource was being cached each time.  There is some google analytics network activity (fetching a 1x1 gif with a collect? URL), but these are not being cached in SW.

bsittler@ : even using your modified opaque-count code (https://bugs.chromium.org/p/chromium/issues/detail?id=796060#c31) the count is zero.  I don't think the problem seen in repro #9 relates in any way to caching opaque resources having variable URLs.

#9 still repros for me in Canary 65.0.3316.0 Mac on chromestatus. IDB is going up, but CacheStorage slowly increases by ~1.5mb. Screencast attached.

Using the code snippet in https://bugs.chromium.org/p/chromium/issues/detail?id=796060#c31, there are 0 opaque responses. Also confirming we're not caching resources with varying query strings or Analytics requests.

Deleted: ScreenFlow.mp4 3.2 MB

	ScreenFlow.mp4 3.2 MB View Download

Thanks! Looks like this needs further triage

Correction: this needs another repro attempt, possibly on macOS specifically

Further research is still needed, but it's possible this is a duplicate of  crbug.com/801024 

Tested the issue on mac 10.12.6 using latest chrome version #65.0.3323.0 and chrome reported version #65.0.3294.0.
Attached a screen cast for reference.

Following are the steps followed to reproduce the issue.
------------
(1) Navigated to https://www.chromestatus.com/features
(2) Refreshed the page.
(3) Opened Application > Clear storage in the Devtools.
(4) Observed that "Clear Storage" showed equal usage than what's in Cache Storage using latest chrome version #65.0.3323.0 but was able to reproduce the issue on chrome reported version #65.0.3294.0 very inconsistently.

Hence, marking it as untriaged for further inputs from dev team

Thanks...!!

Deleted: 795134.mp4 3.4 MB

	795134.mp4 3.4 MB View Download

Yeah, this looks exactly like  issue 801024 , especially "(h) for me, the "using" value is now up to 23.88 MB ... basically *double* what it was before..."

Given update in #29 I'm resolving as Duplicate but I'd appreciate the reporters giving Canary a try. 

Verified. I'm no longer seeing the increase in mac 65.0.3325.0! Thanks for tracking this down :)

\o/