New issue
Advanced search Search tips

Issue 702787 link

Starred by 6 users

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug


Participants' hotlists:
IDB-Stability


Sign in to add a comment

IndexedDB eats a lot of memory when opening many cursors to different leveldb blocks that store large values

Project Member Reported by pwnall@chromium.org, Mar 17 2017

Issue description

Each cursor holds a leveldb iterator, which holds onto a leveldb block. Blocks are generally small (32kb), but can be large if large keys / values are stored in them, because leveldb doesn't break keys/values across blocks.
 

Comment 1 by jsb...@chromium.org, Mar 20 2017

Cc: jsb...@chromium.org cmumford@chromium.org dmu...@chromium.org
Labels: -Pri-3 Pri-1
Owner: pwnall@chromium.org
Status: Assigned (was: Untriaged)
Bumping Pri

Comment 2 by jsb...@chromium.org, Mar 21 2017

Cc: pwnall@chromium.org
Owner: dmu...@chromium.org
Status: Started (was: Assigned)
Project Member

Comment 4 by bugdroid1@chromium.org, Mar 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec764054b12a8e21e1fe958c3f1e4d47df671066

commit ec764054b12a8e21e1fe958c3f1e4d47df671066
Author: dmurph <dmurph@chromium.org>
Date: Mon Mar 27 23:12:47 2017

[IndexedDB] Pool and evict leveldb iterators, to save memory

Caps the total number of open cursors per database. Ideally the cap
would be on the # of bytes held by the cursors, but this data is deeply
hidden, and this is good enough for an initial mitigation.

Note: This could potentially cause worse memory churn if the number of
cursors allowed is too low.

R=jsbell,pwnall,cmumford
BUG= 696055 , 702787 

Review-Url: https://codereview.chromium.org/2760163002
Cr-Commit-Position: refs/heads/master@{#459927}

[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/BUILD.gn
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_backing_store.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_backing_store_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_class_factory.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_class_factory.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_cleanup_on_io_error_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_database.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_database.h
[add] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator_impl.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator_impl.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_transaction.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_transaction.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_transaction_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/mock_browsertest_indexed_db_class_factory.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/mock_browsertest_indexed_db_class_factory.h

Comment 6 by dmu...@chromium.org, Mar 29 2017

For clarification, push was on 2017/03/27 at 8pm, and the percent went from 0.58% on Monday to 0.16% on Tuesday.

Comment 7 by roy...@google.com, Mar 30 2017

Do we need to merge this back to older versions ?
I know we addressed the drive offline issue, but we may need to worry about other apps as well which may hit it.

Comment 8 by pwnall@chromium.org, Mar 30 2017

#7: I would prefer not to merge this, unless the crash rate goes back up. While the fix CL is much smaller than I could have hoped for (props to dmurph@!), it still changes our performance profile in a non-trivial way, so it'd be nice to have it bake through the normal channel progression.

Comment 9 by dmu...@chromium.org, Mar 30 2017

I agree - I think it's safer to let the bake, and the crash reports are pretty low (I think?) - around 200 or less every day.
(some of which are probably legitimate OOM). Previously w/ docs hitting this the crash rate was in the mid thousands.
Also, keep in mind that the CL above wouldn't fix all the IndexedDB OOM crashes. That crash stack points to a site where we allocate (comparatively) larger blocks of memory, so an OOM is more likely to hit there than in most other callsites for memory allocators.

Sign in to add a comment