Re: incognito - sounds like a bug. We should still report something. Can you file a separate bug on that with a stand-alone repro? (the example here where cache storage is used as well is a distraction)

Re: the increase - the backing store used by Indexed DB lazily compacts data written to the log, so the quota estimations are approximate. Like memory/garbage collection, the usage will grow until the database does a collection pass. This enables higher throughput at the expense of reporting accuracy. At some point - on the order of table sizes, which are in the low megabytes - a threshold will be passed and the data will be restructured.

We've considered work to make this more obvious, so not closing for now.

re incognito... I've submitted a separate bug with cut-down repro at https://bugs.chromium.org/p/chromium/issues/detail?id=795927

re the increase... OK I can confirm this.  I stored (and re-stored) something a bit bigger to make the effect greater, then refreshed lots of times.  The size of indexeddb grows, and after a few MB it drops back again.  As you say, it would be great if this could be made more obvious somehow... some way of showing *actual* indexeddb usage, not including anything discarded but not yet cleared out completely.

#4: For whatever it's worth, the number you're seeing _is_ how much your IndexedDB data space uses on the user's drive. In all modern databases I know of, having a steady stream of writes will result in disk usage above the space used to store your data.

Elaborating on what jsbell@ said, our backing store (LevelDB) is a log-structured merge tree (LSM), so some patterns of overwrites can result in quite a bit of storage overhead. That being said, if you look at other browsers, you'll find that they use SQLite. When tuned for high-performance, SQLite uses a write-ahead log ( WAL - https://www.sqlite.org/wal.html ), which gets checkpointed every few megabytes.

In other words, in order to give you atomic and durable transactions while achieving high throughput, IndexedDB's backing store has to use extra space while you're writing.

We need to charge you (quota) for how much disk space you actually use, because we don't want well-intended but buggy sites to be able to fill up the user's drive. So, you need to be aware of how much actual disk space you end up using, given your particular data access pattern.

I think we should do a better job of documenting what's happening (link / information bubble from the disk usage page?), and I don't think we should implement (and display) an accounting system that deviates from actual disk usage.

pwnall@ whilst what you describe is of interest to both you and me, it seems that it is more of interest to you (browser developer) than me (web developer).  Ideally there would be a checkbox to allow a choice between showing the storage stats both with and without the effects you describe... best of both worlds.

Some further thoughts on this issue... let's assume for the moment that the demo I have provided is completely bug-free and is as efficiently coded as it could be.  It uses up a single space in the IDB, and doesn't ask for any more.  I have asked for a single space, and am only using a single space... I'm doing my bit as a web developer, writing pure and beautiful code.  Therefore, I would not expect to be charged for more than one space, just because the *browser* finds it most efficient to use up more than one space on my behalf.  Although highly unlikely because we're only talking a few MB, in theory it is possible that, despite behaving myself and writing great code that demands very little, I end up going over my quota.  If I only ever use one space in the IDB, I'd expect the IDB storage value to reflect that, remaining constant.

It's similar to what I would expect to see for Cache Storage... if I store a 1 GB file in cache, then remove it and store another 1 GB file, I'd expect my quota usage to still be 1 GB, particularly if my limit is 1.5 GB.  In the background, the old file might still be kicking around, but that doesn't mean it should still be showing in the Cache Storage value... it wouldn't be fair to be charging me for 2 GB when I'm still only using 1 GB.

In other words, I think that storage space used purely for internal database/cache management should not be included in what is reported to (or charged to) the web developer.  Yes it might be interesting to understand things from the browser perspective, but I think there should be a choice regarding what is displayed in devtools, and certainly for quota management purposes the web developer shouldn't be charged for what they are not actually using.

Keep in mind there are also good things happening in leveldb, like compression.

What purpose would this number serve? Why do you want it? Do you want to make sure that your database shrinks when you delete things? What is the specific use cases here that makes you want to check this number? It sounds like there is a feature request in here but I want to make sure I fully understand what you want.

When you use IndexedDB in Chrome you're using (at least indirectly) a mature log-based database platform (leveldb) that guarantees nice things like data consistency, even when crazy things like power outages and crashes happen. It amortizes the costs of disk operations by doing stuff like mutations and cleanups periodically instead of every write. Sometimes deletes don't reflect right away because those deletes haven't been compacted into the database (which is part of the amortization). This makes the leveldb very fast, but it also means that the disk usage fluctuates a bit.

We should definitely do a better job of explaining the disk usage number from IndexedDB so people aren't confused though.

It sounds like you're worried about quota. I wonder what would happen with IndexedDB with a 1GB value, as we would save that to a blob first. It should allow you to delete and then save the new value w/o hurting quota.

Maybe we can do a better job with quota and non-full disks and indexeddb - maintaining a different custom storage size # would be pretty difficult though.

Another good thing here is that compaction is triggered pretty frequently and it's definitely triggered when the log gets large, so any danger here would be with small values near the edge of the quota.

I don't think there is a perfect solution here unfortunately.

Yep I guess I'm most worried about quota, though it's also reassuring to see a flat "storage used" response over time (when you expect it to be) because it is an indicator that you're doing things right and not leaking stuff out.

Regarding quota, I've already experienced hitting the limit... this is what got me here in the first place: https://github.com/GoogleChrome/puppeteer/issues/1596

In the above, I initially blamed it on IDB (sorry IDB), but then noted that it was in fact a parallel write to SW cache that was causing the quota to be exceeded... though I'm pretty convinced there is a bug in the way quota is being handled.

This has all been discussed in relation to Cache Store at https://bugs.chromium.org/p/chromium/issues/detail?id=795134 ... though that bug report was closed prematurely IMHO without proper investigation and it's not clear whether it is being re-opened... there's definitely something fishy going on with the quota being reported, and it is causing me a real problem... i.e. Chrome telling me I'm going over quota when in fact I'm really not caching much resource at all.

I kinda suspect that the linked issue above will be explained away as "well that is the disk space that is being used internally, even if you think you aren't using that much, so that is what we charge you", which is kinda the conclusion jumped to in Comment #2 of that issue.

Going back to IDB specifically, you write:

> it should allow you to delete and then save
> the new value w/o hurting quota

Are you suggesting that it is better to delete an entry before writing a new value to the same key?  Rather than simply (over)writing a new value to the same key?  Surely these two alternatives are equivalent and should really result in the same quota behaviour?

Ah sorry, to clarify - 

Most transactional database system needs to be hold the <size of database> + <size of changes>. This is because transactions need to be abort-able and atomic. So if you're overwriting a value, then the database will temporarily need to hold both the new and the old value. So this, I believe, is unavoidable for transactional databases.

If you want to avoid this happening, then you would need to delete the value in one transaction, wait for it to complete, and then insert the new value in a new transaction.

However - this would introduce a race case in your database, where the webpage or system could be stopped in between deleting the old value and inserting the new value.

> needs to hold the <size of database> + <size of changes>

It would be very useful and informative to be able to show both A+B and A in devtools (or have a choice between the two) to make this absolutely apparent to the developer.  But I think you say this is difficult.  Personally I feel that Quota used should be based on just A, not A+B, but I can understand the argument for using A+B too... particularly if very large items are being stored so that this will have a very real effect on system resources.  At the very least, some more information (in whatever form) in DevTools would be great.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

As has been stated, the quota reporting for indexeddb described above is the expected behaviour due to the particulars of leveldb. We do not intend to change how we report used disk space for idb in devtools, and so this bug is being closed.

Reopening this. We need a bug to track the fact that the way IndexedDB uses quota is not intuitive to developers, and this one has a bunch of good content in it.

 Issue 792426  has been merged into this issue.