Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 25 users
Status: Fixed
Owner: ----
Closed: Yesterday
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature

Blocked on:
issue 148757


Show other hotlists

Hotlists containing this issue:
IDB-Performance


Sign in to add a comment
IndexedDB blocks DOM rendering
Reported by nolan.la...@gmail.com, Sep 27 2015 Back to list
UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/41.0

Steps to reproduce the problem:
1. Do a long-running IndexedDB operation in a transaction
2. Meanwhile, run an animated GIF in an <img/> tag or a JavaScript animation

What is the expected behavior?
The animation should continue running smoothly even while IndexedDB operations are in progress.

What went wrong?
The animation is paused during the IDB transaction.

Did this work before? N/A 

Chrome version:  47.0.2520.0 canary (64-bit)  Channel: canary
OS Version: OS X 10.10
Flash Version: 

See a live demo here: http://nolanlawson.github.io/database-comparison/

Steps:
1. Choose "IndexedDB"
2. Choose 10000 or 100000
3. Click "Insert docs"
4. Notice that Kirby stops dancing while the documents are being inserted.

The DOM is not affected by IndexedDB when run in a web worker. Also I note that this bug also exists in Firefox, but doesn't exist in Safari or Edge.
 
database-comparison.zip
190 KB Download
Issue filed on Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1208840
Comment 2 by tkent@chromium.org, Sep 28 2015
Labels: -Cr-Blink-JavaScript Cr-Blink-Storage-IndexedDB
Comment 3 by jsb...@chromium.org, Sep 28 2015
Are you sure Safari/Edge behave differently if it's a JS animation? I believe both of those browsers handle GIF animations off the main thread, so it's not quite an apples-to-apples comparison.

And yes... event delivery happens on the main thread, so the IDB event dispatch (of the successful puts) hammers the main thread.

We've discussed a couple of possibilities here, including a "write only" transaction that doesn't bother with individual success events and implicitly commits immediately, or a "putAll" batch operation.
Comment 4 by jsb...@chromium.org, Sep 28 2015
Labels: -OS-Mac Performance
Status: Available
Thanks for the insights. Maybe it would be more apples-to-apples if I used a JavaScript animation instead of a GIF? (E.g. changing a CSS value in a setInterval().) I had always assumed GIF animations were a good stand-in for any kind of DOM operation, but I guess not.

As for the fix, putAll() seems like it would help a lot, but given that I'm not attaching onsuccess/onerror handlers to the put()s, shouldn't it effectively be a "write only" transaction, like what you describe? Why does it hammer the main thread so much?
Comment 6 by jsb...@chromium.org, Sep 28 2015
Yeah, JS-based animation would be apples-to-apples.

There are some thoughts here: https://gist.github.com/inexorabletash/d55a6669a040e92e47c6

Basically there are two reasons we might consider "write-only" transactions.

(1) every 'success' event has an internal overhead even if there's no handler. Even if measured in microseconds, if you're doing 100k put calls that can add up to a noticeable impact on the main thread.

(2) the transaction can't attempt to commit until all of the 'success' events have been delivered since the last one might attempt to queue more work in the tx; this adds measurable overhead to each transaction.

A "write-only" transaction that doesn't deliver success/fail events on individual operations would avoid both of those. putAll() would address #1 by coalescing e.g. 100k events into just one event, but wouldn't help with #2.

We may be able to be more clever about reducing the overhead when there's no listener attached. (It's subtle, though - when exactly do we check to see if an event handler is attached?)

Anyway, definitely keeping this open.
I tried adding a JavaScript animation as well (https://github.com/nolanlawson/database-comparison/tree/with-js-anim), but it had the same characteristics as the animated GIF in every browser I tested - Chrome, Firefox, Edge, and Safari.  So apparently it is a pretty good proxy for measuring DOM blocking.

FWIW, when I test in a web worker, there's absolutely zero DOM blocking in either Firefox or Chrome. Although it might be due to the lack of overhead of structured cloning; in my test, the objects are created within the worker itself.
OK, apparently the structured cloning is not the culprit. @calvinmetcalf added another web worker test using cloned objects (i.e. not creating the objects within the worker itself), and it still blocks less than straight IDB: http://nolanlawson.github.io/database-comparison/
Labels: Cr-Blink-Scheduling
Thanks.

Looking at nolan's perf numbers for Safari/Edge (which don't block) the transactions take an order of magnitude more time to commit. I think they're likely deprioritizing the database event delivery, something we've talked about.

We've had a "to-do" for a while to explore integrating with the blink scheduler to defer IDB event processing - let's track that here.
I should note that it's not just the events that cause overhead - the put() calls themselves must do a little synchronous processing (serializing the data, etc) but that's more obvious and less avoidable.
Blockedon: chromium:148757
Cc: adamk@chromium.org
Labels: -Via-Wizard -Cr-Blink-Scheduling
I was wrong - the event overhead is negligible. The bulk of the time that Kirby pauses is due to the put() calls. After making some speculative changes to event processing (with no benefits) I added some debug spew and realized it was all in the puts.

I used chrome://tracing to view this (nolan - have you tried that out? if not, ping me offline and I'll walk you through it). I also added some deeper tracing to the phases of the put and it looks like the time breaks down as follows:

For 1000 records (to keep the tracing results sane):

Page reports 141ms end-to-end. The script function that makes the put calls has a wall-clock duration of 64ms; the back-end transaction commit and flush to disk takes 53ms and there's a little bit lost in IPC. Even with more records the commit time remains small, so it ends up being the put() calls dominating.

Of the 64ms, the put()s consume 57ms of that. Breaking that down:
* 15ms is spent serializing 
* 15ms is spent extracting the inline key, 9ms of which is deserializing
* 15ms is spent setting up the async IPC call into the backend

The remaining time (~10ms) is split into little operations (validating inputs, checking to see if index keys need to be computed, computing the wire format, creating the result IDBRequest, etc).

Unfortunately, all of those things need to occur synchronously with respect to script to enforce constraints (i.e. they may throw), so we can't defer them. We need to look into optimizing the individual steps.

We know that serialization/deserialization is slow. It's actually faster to JSON.stringify() then postMessage() a string than to postMessage() an object. :(

Nope, never used chrome://tracing. If you have a good tutorial or something you'd recommend, I'd love to try it. :)

That's really odd that serialization eats up so much. With WorkerPouch, I ran tests [1] comparing serializing exactly the same object and then posting to a worker (no JSON.parse or stringify) and it definitely blocks the DOM less. Maybe due to the fact that PouchDB'S internal IDB representation of objects is larger than the external one? 

That database-comparison app also has a test with serializing the IDB objects themselves ("worker with cloning"), and it's still less blocking. If the answer is to just use workers, I'm fine with that, but it'd be really surprising to me if just cloning added up to so much time.

[1]: https://github.com/nolanlawson/worker-pouch/blob/master/README.md#performance-benefits
If you offload to a worker you're still doing one serialization on the main thread (for posting the data) but offloading the key extraction and setting up the IPC call. So That'd be a 2/3 reduction, which is a lot.

the other thing is to split the the put() calls into batches, which you can do within one tx. Just wait on the last put's success event from each batch and do the next one. Its still doing work on the main thread, but shouldn't starve the DOM as much.
Project Member Comment 14 by sheriffbot@chromium.org, Oct 12 2016
Labels: Hotlist-Recharge-Cold
Status: Untriaged
This issue has been available for more than 365 days, and should be re-evaluated. Please re-triage this issue.
The Hotlist-Recharge-Cold label is applied for tracking purposes, and should not be removed after re-triaging the issue.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -Type-Bug -Pri-2 Pri-3 Type-Feature
Status: Available
The work to speed up serialization/deserialization is ongoing (see dependent bug). That's where the bulk of the possible performance improvement is going to happen here.

I don't know if we're expecting any measurable speedup from converting IPC to Mojo.
Comment 16 by mr.a...@gmail.com, Feb 8 2017
Thanks for looking into this.

What honestly surprises me is, that in both Chrome 56 and Chromium 58 using WebSQL does not block while the same test using Indexed DB does block the main thread... any further ideas? 
It'll be interesting to see the speedup once the serialization changes are in.

I note that in nolan's test the IndexedDB operation uses a keypath which incurs a deserialization on each put (to avoid side-effects), whereas the websql one has explicit keys. WebSQL values are also only numbers are numbers or strings, so you're not paying the serialization cost there either. An apples-to-apples comparison would be nice - strings values and explicit keys on both sides.
Comment 18 by kgr@chromium.org, Feb 9 2017
We have a simple chat client PWA that has offline sync support and stores chat history in IDB. When it's running, it can block my typing of emails or IM's in Gmail by a second or more, the time it takes to type two or three words; which is really annoying. Normally, when your tab is using IDB, you're waiting for the data anyway, so you don't really notice the delay, but when some other tab or process (like a PWA SW) is doing IO, unbeknownst to you, you don't expect your window to completely block. I don't know if this is being made worse by the Service Worker being killed and restarted periodically.

I think the priority of this issue should be raised.
kgr@ - do you have a repro?

That sounds extremely unusual. Note the user-noticable delays this issue is about are when 10,000 or 100,000 requests are made in one synchronous loop from the main thread; a similar delay would occur if there were a comparable number of postMessage() calls.

I highly doubt you're doing that in your chat client so you may be seeing something completely different. Can you share a reduced repro?
Comment 20 by kgr@chromium.org, Feb 9 2017
I've send repro information in a private email.

There doesn't need to be any traffic for this to happen, which is why I think it might be related to IDB startup rather than during actual IO, since the service worker will periodically shutdown.

FYI, re: the discussion around #c11

On comparable hardware (I think it's the same...) in 57 I see around a 2x performance improvement.

* The overall time reported by the page for 1000 puts to Indexed DB is down from 141ms to 110ms (but for only 1000 records there's a fair bit of overhead which is not amortized). 
* The script function that makes the call is down from 64ms to 27ms. 
* The put() calls are down from 57ms to 18ms

I did not break down the time in put() but the speedup are likely due to: (1) faster serialization of records, (2) faster deserialization of records (when extracting the key), (3) switch to Mojo which reduces IPC overhead.

Since Chrome updated to m56, my issue went away.
Comment 23 by benhenry@chromium.org, Yesterday (44 hours ago)
Labels: -Performance Performance-Responsiveness
Status: Fixed
Dare we say fixed?
Sign in to add a comment