Handle big data in chrome.storage.local by implementing new methods
Reported by
woxxom@gmail.com,
Nov 26
|
|
Issue descriptionThis is a feature request for the extensions API. ================================================================ Proposing to extend chrome.storage.local (and maybe .sync too for uniformity) by adding the following methods: * getAllKeys() produces an array of key names just like IndexedDB * getAsJson(key or arrayOfKeys or objectWithKeys or null) produces a single JSON string for the specified keys or all keys for "null" advantages: it's superfast; it doesn't pollute V8 memory since it's created natively ================================================================ Currently reading a value from chrome.storage.local spikes memory consumption approximately 10 times the amount of data read in case all the string values are unique and cannot by deduplicated by V8. Depending on the data structure reading 10-100MB could spike to anywhere between 100MB and 3.5GB and beyond, which crashes the extension process. v8::internal::Heap::FatalProcessOutOfMemory v8::internal::Heap::AllocateRawWithRetryOrFail v8::internal::Factory::NewFixedArrayWithFiller v8::internal::Factory::NewFixedArrayWithMap v8::internal::HashTable v8::internal::StringTable::LookupKey v8::internal::StringTable::LookupString v8::internal::LookupIterator::PropertyOrElement v8::internal::JSReceiver::CreateDataProperty v8::Object::CreateDataProperty content::V8ValueConverterImpl::ToV8Object content::V8ValueConverterImpl::ToV8ValueImpl content::V8ValueConverterImpl::ToV8Object content::V8ValueConverterImpl::ToV8ValueImpl content::V8ValueConverterImpl::ToV8Value extensions::APIRequestHandler::ArgumentAdapter::GetArguments extensions::APIRequestHandler::CompleteRequestImpl extensions::APIRequestHandler::CompleteRequest extensions::NativeExtensionBindingsSystem::HandleResponse IPC::MessageT An extension with "unlimitedStorage" permission can shoot itself in the leg by incrementally storing more than V8 can handle in one call when reading the entire storage via chrome.storage.local.get(null, ...) which will crash the extension process as demonstrated by the attached extension (it can take several minutes so be patient). Although it's possible to implement workarounds in JavaScript, but that would be slow as it requires adding a virtual layer to the storage by using a dedicated "index" key to store the actual key names, and write the actual data to uniformly named chunks.
,
Nov 27
+1 - let's do it! > Extensions would need to have protected storage (not evicted by the browser) Eviction "should" only occur for web schemes. But we should verify. Re: proxying over extension messaging - if this were to switch from JSON to structured cloning then in theory we'd have full fidelity with what can be stored in Indexed DB (ignoring transferrable-but-not-serializable). Otherwise, a userspace proxying library seems feasible...?
,
Nov 27
> proxying over extension messaging - if this were to switch from JSON to structured cloning then in theory we'd have full fidelity with what can be stored in Indexed DB (ignoring transferrable-but-not-serializable). I don't have a good idea of how difficult this would be. With extension messaging, we can pass messages between otherwise-unrelated renderers (e.g., the extension's background page and a content script injected into a web page). Does anything outside of blink currently use structured cloning? If so, I think it'd be a great improvement (for multiple reasons). > Otherwise, a userspace proxying library seems feasible...? Yeah, and it would be relatively straightforward. The downside is that a) it's much slower (because of the extra process hops and serialization), and content scripts sometimes need to act fast, and b) it means content scripts that want to use large storage objects are out of luck (because we have similar limitations/inefficiencies in the messaging code). Of course, I don't have a good impression of how frequently content scripts need to access excessively-large amounts of data from storage - that might be something we can grab UMA on. |
|
►
Sign in to add a comment |
|
Comment 1 by rdevlin....@chromium.org
, Nov 26Status: Available (was: Unconfirmed)