New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

the total blobs' size cannot exceed about 500MiB

Reported by jiangzid...@gmail.com, May 20 2014

Issue description

UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36

Steps to reproduce the problem:
1. using the FileSaver.js to download the blob. https://github.com/eligrey/FileSaver.js/
2. call the function:

function blobtest()
{
	for (var i=0;i<100;i++)
	{
	console.log(""+i);
		var myarray=[];
		for (var j=0;j<1000;j++)
		{
		myarray.push(new Array(10001).toString());//it has 10000 commas,which is 10000 Bytes
		}
	var blob = new Blob(myarray, {type : "text/plain;charset=utf-8"});
	saveAs(blob, "filename" + i + ".txt");
	blob=null;
	}
};

What is the expected behavior?
download 100 files (filename0.txt~filename99.txt), each size is 10,000,000 Bytes,9.76MiB

What went wrong?
after it downloads the 52nd file (filename51.txt), the following downloading fails. it shows "Failed, No File".

I can change each file's size, but when the total downloading files' size (blob size) exceeds about 500MiB, it goes wrong.

Did this work before? No 

Chrome version: 34.0.1847.137  Channel: stable
OS Version: 6.1 (Windows 7, Windows Server 2008 R2)
Flash Version: Shockwave Flash 13.0 r0

I also on my ubuntu 12.04 host with chrome M34. It also fail and the max size is also about 500MiB.
 
FileSaver.js
7.6 KB View Download
Showing comments 15 - 114 of 114 Older
I created a workaround!  It's not performant at all, but it works:
https://github.com/dmurph/sandbox/blob/master/blobRepro.js

Main idea:  force blobs to be file-backed by saving them to indexeddb and then grabbing them again.  Then create a blob that concatenates all of these, and then use that as your >500MB blob.
Project Member

Comment 16 by bugdroid1@chromium.org, Dec 20 2014

The following revision refers to this bug:
  http://src.chromium.org/viewvc/blink?view=rev&rev=187574

------------------------------------------------------------------
r187574 | dmurph@chromium.org | 2014-12-19T23:57:40.258850Z

Changed paths:
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/main/Main.js?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/resources/indexedDBViews.css?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/sdk/WorkerTargetManager.js?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/resources/IndexedDBViews.js?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/resources/resourcesPanel.css?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/resources/serviceWorkerCacheViews.css?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/resources/ResourcesPanel.js?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/resources/ServiceWorkerCacheViews.js?r1=187574&r2=187573&pathrev=187574
   M http://src.chromium.org/viewvc/blink/trunk/Source/devtools/front_end/sdk/Target.js?r1=187574&r2=187573&pathrev=187574

[ServiceWorker] Cache inspector out from flag, style & icon changes:

* Added class to indexeddb & SW cache views to prevent double border
* Changed icon class of SW cache tree element to be the 'table' icon from indexeddb object store
* Changed the 'settings' key for the serviceworker tree element, which was mistakenly left the same as indexed db.
* Added 'border-bottom' style to the last element in both indexed db & SW cache grid view.

BUG= 375297 

Review URL: https://codereview.chromium.org/813173002
-----------------------------------------------------------------
whoops, wrong bug. Sorry!
Project Member

Comment 18 by bugdroid1@chromium.org, Jan 7 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/69e28f6f37f80b9a472a41ba8198d5818508100f

commit 69e28f6f37f80b9a472a41ba8198d5818508100f
Author: dmurph <dmurph@chromium.org>
Date: Wed Jan 07 21:07:59 2015

[Storage] Added histograms for blob storage

For finding:
* Size distribution of blob storage
* Size distribution of blob items
* Number of cases when we are exceeding internal memory for blobs.
See here for info: https://bit.ly/AutoBlobToDisk

BUG= 375297 

Review URL: https://codereview.chromium.org/812843005

Cr-Commit-Position: refs/heads/master@{#310367}

[modify] http://crrev.com/69e28f6f37f80b9a472a41ba8198d5818508100f/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/69e28f6f37f80b9a472a41ba8198d5818508100f/tools/metrics/histograms/histograms.xml

Project Member

Comment 19 by bugdroid1@chromium.org, Jan 9 2015

Issue 230858 has been merged into this issue.
Labels: TE-Verified-41.0.2272.3 TE-Verified-M41
Tested the same on mac 10.9 chrome version 41.0.2272.3 using the files given in comment #2 - Fix works as expected

Please find the screenshot
Screen Shot 2015-01-12 at 10.21.58 PM.png
60.6 KB View Download

Comment 22 by den...@gmail.com, Jan 13 2015

Here is the sample file from issue 230858 - This seems more like a memory leak than the 500mb issue though. 
blobappend.html
459 bytes View Download
Labels: -M-41 MovedFrom-41
This issue has already been moved once and is lower than Priority 1,therefore removing mstone.

Comment 24 by kbr@chromium.org, Jan 21 2015

Could we make some short-term progress on this bug? I'm not sure that the data that will be gathered will be that useful since developers will already have worked around Chrome's limitations, and the data gathering process will take weeks. In the short term could we just raise the blob limit to 2 GB or so for 64-bit Chrome?

A more intelligent management system for blobs in the browser process sounds good and should be pursued. I'm not sure users expect blobs to be that fast -- I think they'll use typed arrays for in-memory access of reasonably sized data sets -- so if they were streamed to disk more quickly to allow more robust behavior that sounds good.

Progress is ongoing, here is the design doc:
https://bit.ly/AutoBlobToDisk

and current CLs:
https://codereview.chromium.org/810403004/
https://codereview.chromium.org/821913004/

Apologies, I should have appended that doc to this issue. We're hoping to land the fixes by the next release.

Whoops, sorry guys, for some reason I can't make that doc public.  Here is the public version:
https://bit.ly/BlobStorageRefactor
Project Member

Comment 28 by bugdroid1@chromium.org, Jan 23 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0f82e12a3934b87edc51c800dfc9f27b9131da19

commit 0f82e12a3934b87edc51c800dfc9f27b9131da19
Author: kochi <kochi@chromium.org>
Date: Fri Jan 23 05:57:09 2015

Revert of [Storage] Consoliation of BlobItems with small size (patchset #8 id:140001 of https://codereview.chromium.org/821913004/)

Reason for revert:
This caused regression on some layout tests.
e.g.
http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.6%20%28deps%29/builds/35407
fast/files/read-blob-async.html
fast/files/workers/worker-read-blob-async.html
fast/files/workers/worker-read-blob-sync.html
fast/files/workers/worker-read-file-constructor-async.html
http/tests/local/blob/send-data-blob.html
http/tests/navigation/beacon-same-origin.html
http/tests/websocket/bufferedAmount-after-send.html

Original issue's description:
> [Storage] Consolidation of BlobItems with small size
> https://bit.ly/AutoBlobToDisk
>
> R=michaeln, piman
> BUG= 375297 
>
> Committed: https://crrev.com/4e45f41babddb10985c8c0afb9b0b6b9e8356094
> Cr-Commit-Position: refs/heads/master@{#312770}

TBR=michaeln@chromium.org,piman@chromium.org,dmurph@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 375297 

Review URL: https://codereview.chromium.org/869763003

Cr-Commit-Position: refs/heads/master@{#312782}

[modify] http://crrev.com/0f82e12a3934b87edc51c800dfc9f27b9131da19/content/child/webblobregistry_impl.cc
[modify] http://crrev.com/0f82e12a3934b87edc51c800dfc9f27b9131da19/content/child/webblobregistry_impl.h
[modify] http://crrev.com/0f82e12a3934b87edc51c800dfc9f27b9131da19/storage/common/data_element.h

Project Member

Comment 29 by bugdroid1@chromium.org, Jan 23 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bff2e53fb3e210be45d264797036ec59005ebdd6

commit bff2e53fb3e210be45d264797036ec59005ebdd6
Author: dmurph <dmurph@chromium.org>
Date: Fri Jan 23 09:18:56 2015

[Storage] Blob Storage Refactoring pt 1:
* Renaming classes to be more descriptive.
* Changing smart pointers to reflect strict ownership model.
* Adding pointers to facilitate future resource swapping.
* Remove renderer-side dependency on blob_data.h

This patch makes all of the far-reaching changes that effect
everyone that uses the blob storage context.  Subsequent
changes should only effect the blob infrastructure.

https://bit.ly/AutoBlobToDisk

BUG= 375297 

Review URL: https://codereview.chromium.org/810403004

Cr-Commit-Position: refs/heads/master@{#312800}

[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/chromeos/drive/fileapi/async_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/chromeos/drive/fileapi/fileapi_worker.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/chromeos/file_manager/snapshot_manager.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/chromeos/file_system_provider/fileapi/provider_async_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/chromeos/file_system_provider/fileapi/provider_async_file_util_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/extensions/api/developer_private/developer_private_api.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/extensions/api/page_capture/page_capture_api.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/media_galleries/fileapi/device_media_async_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/media_galleries/fileapi/device_media_async_file_util.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/media_galleries/fileapi/iphoto_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/media_galleries/fileapi/itunes_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/media_galleries/fileapi/native_media_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/media_galleries/fileapi/picasa_file_util_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/drive_backend/drive_backend_util.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/drive_backend/sync_engine.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/local/canned_syncable_file_system.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/local/local_file_sync_context.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/local/local_file_sync_context_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/local/local_file_sync_service.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/chrome/browser/sync_file_system/local/syncable_file_system_operation.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/blob_storage_context_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/blob_storage_host.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/blob_storage_host.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/blob_url_request_job_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/chrome_blob_storage_context.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/copy_or_move_file_validator_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/file_system_operation_impl_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/file_system_operation_impl_write_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/fileapi_message_filter.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/fileapi_message_filter.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/fileapi_message_filter_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/fileapi/transient_file_util_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/indexed_db/indexed_db_active_blob_registry.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/indexed_db/indexed_db_blob_info.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/indexed_db/indexed_db_callbacks.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/indexed_db/indexed_db_dispatcher_host.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/redirect_to_file_resource_handler.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/resource_dispatcher_host_impl.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/resource_dispatcher_host_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/resource_loader_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/temporary_file_stream.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/temporary_file_stream_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/upload_data_stream_builder.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/loader/upload_data_stream_builder_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/media/android/media_resource_getter_impl.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/service_worker/service_worker_browsertest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/service_worker/service_worker_cache.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/service_worker/service_worker_cache.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/service_worker/service_worker_cache_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/service_worker/service_worker_url_request_job.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/service_worker/service_worker_url_request_job_unittest.cc
[rename] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/shareable_file_reference_unittest.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/browser/storage_partition_impl_map.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/child/webblobregistry_impl.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/common/fileapi/webblob_messages.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/content_tests.gypi
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/content/public/test/mock_blob_url_request_context.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/BUILD.gn
[add] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_builder.cc
[add] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_builder.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_handle.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_handle.h
[add] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_item.cc
[add] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_item.h
[add] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_snapshot.cc
[add] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_data_snapshot.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_storage_context.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_url_request_job.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_url_request_job.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_url_request_job_factory.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/blob_url_request_job_factory.h
[rename] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/scoped_file.cc
[rename] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/scoped_file.h
[rename] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/shareable_file_reference.cc
[rename] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/shareable_file_reference.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/view_blob_internals_job.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/blob/view_blob_internals_job.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/async_file_util_adapter.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/copy_or_move_operation_delegate.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/dragged_file_util.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/file_system_file_stream_reader.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/file_system_file_util.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/file_system_operation_impl.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/file_system_operation_impl.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/file_system_operation_runner.cc
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/obfuscated_file_util.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/browser/fileapi/sandbox_file_stream_writer.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/common/BUILD.gn
[delete] http://crrev.com/98d7ef56b969695d1f185881e3eed845a3221136/storage/common/blob/blob_data.cc
[delete] http://crrev.com/98d7ef56b969695d1f185881e3eed845a3221136/storage/common/blob/blob_data.h
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/storage_browser.gyp
[modify] http://crrev.com/bff2e53fb3e210be45d264797036ec59005ebdd6/storage/storage_common.gyp

Project Member

Comment 31 by bugdroid1@chromium.org, Feb 11 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8acaec3721c8f1b76004194be81612ee5ff08ecf

commit 8acaec3721c8f1b76004194be81612ee5ff08ecf
Author: dmurph <dmurph@chromium.org>
Date: Wed Feb 11 01:23:25 2015

[Storage] Blob items are now shared between blobs. Ready for disk swap.
  * Added ShareableBlobDataItem for sharing items between blobs, and
    implemented blob item sharing.
  * Added InternalBlobData for the internal storage of blobs.
  * Removed CreateSnapshot ability from the old Builder, which is now
    exclusively used to tell the context to create a blob.
  * Fixed copying of the content_disposition header when
    programmatically creating a blob.
  * Fixed UMA histogram for files that have uint64max length value
    (which signifies 'end-of-file' without knowing the size), and
    added a histogram to track how often this happens.
  * Added more documentation and histograms for knowing the state of
    blobs on the web.

Design Doc: https://bit.ly/AutoBlobToDisk

R=michaeln
BUG= 375297 

Review URL: https://codereview.chromium.org/895933007

Cr-Commit-Position: refs/heads/master@{#315702}

[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/fileapi/blob_storage_context_unittest.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/fileapi/blob_url_request_job_unittest.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/fileapi/chrome_blob_storage_context.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/indexed_db/indexed_db_dispatcher_host.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/loader/upload_data_stream_builder_unittest.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/service_worker/service_worker_cache.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/service_worker/service_worker_cache_unittest.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/service_worker/service_worker_url_request_job.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/browser/service_worker/service_worker_url_request_job_unittest.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/child/webblobregistry_impl.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/content/public/test/mock_blob_url_request_context.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/BUILD.gn
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_data_builder.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_data_builder.h
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_data_item.h
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_data_snapshot.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_data_snapshot.h
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/blob_storage_context.h
[add] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/internal_blob_data.cc
[add] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/internal_blob_data.h
[add] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/shareable_blob_data_item.cc
[add] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/shareable_blob_data_item.h
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/view_blob_internals_job.cc
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/browser/blob/view_blob_internals_job.h
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/storage/storage_browser.gyp
[modify] http://crrev.com/8acaec3721c8f1b76004194be81612ee5ff08ecf/tools/metrics/histograms/histograms.xml

Project Member

Comment 32 by bugdroid1@chromium.org, Mar 3 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f6629637feb3bbf186de75da8fe4028a5244389f

commit f6629637feb3bbf186de75da8fe4028a5244389f
Author: mtomasz <mtomasz@chromium.org>
Date: Tue Mar 03 00:20:06 2015

Allow offsets in blobs larger than 2GB on 32 bit Chromium builds.

The recent patch crrev.com/895933007 introduced a regression by using size_t
for offsets, which on 32bit builds is sizeof(size_t) = 4 which was causing
broken offset values due to assigning a uint64_t variable to such variables.

This CL fixes that by simply converting the types to uint64_t.

TEST=*FileSystemProvider*BigFile* on a 32 bit build Chromium build with
    chromeos=1.
BUG= 375297 , 458122

Review URL: https://codereview.chromium.org/959113002

Cr-Commit-Position: refs/heads/master@{#318808}

[modify] http://crrev.com/f6629637feb3bbf186de75da8fe4028a5244389f/chrome/browser/chromeos/extensions/file_system_provider/file_system_provider_apitest.cc
[modify] http://crrev.com/f6629637feb3bbf186de75da8fe4028a5244389f/content/browser/fileapi/blob_storage_context_unittest.cc
[modify] http://crrev.com/f6629637feb3bbf186de75da8fe4028a5244389f/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/f6629637feb3bbf186de75da8fe4028a5244389f/storage/browser/blob/blob_storage_context.h

Project Member

Comment 33 by bugdroid1@chromium.org, Mar 3 2015

Labels: merge-merged-2311
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/35feed79c8c72b9ab539f7750a35d4c6001c01b7

commit 35feed79c8c72b9ab539f7750a35d4c6001c01b7
Author: Tomasz Mikolajewski <mtomasz@chromium.org>
Date: Tue Mar 03 01:44:13 2015

Merge: Allow offsets in blobs larger than 2GB on 32 bit Chromium builds.

The recent patch crrev.com/895933007 introduced a regression by using size_t
for offsets, which on 32bit builds is sizeof(size_t) = 4 which was causing
broken offset values due to assigning a uint64_t variable to such variables.

This CL fixes that by simply converting the types to uint64_t.

TBR=michaeln@chromium.org  # For merge.
TEST=*FileSystemProvider*BigFile* on a 32 bit build Chromium build with
    chromeos=1.
BUG= 375297 , 458122

Review URL: https://codereview.chromium.org/959113002

Cr-Commit-Position: refs/heads/master@{#318808}
(cherry picked from commit f6629637feb3bbf186de75da8fe4028a5244389f)

Review URL: https://codereview.chromium.org/973723002

Cr-Commit-Position: refs/branch-heads/2311@{#92}
Cr-Branched-From: 09b7de5dd7254947cd4306de907274fa63373d48-refs/heads/master@{#317474}

[modify] http://crrev.com/35feed79c8c72b9ab539f7750a35d4c6001c01b7/content/browser/fileapi/blob_storage_context_unittest.cc
[modify] http://crrev.com/35feed79c8c72b9ab539f7750a35d4c6001c01b7/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/35feed79c8c72b9ab539f7750a35d4c6001c01b7/storage/browser/blob/blob_storage_context.h

Cc: nhiroki@chromium.org kinuko@chromium.org ricea@chromium.org
 Issue 464699  has been merged into this issue.
Cc: dmu...@chromium.org
 Issue 467886  has been merged into this issue.
Cc: hashimoto@chromium.org tzik@chromium.org hirosh...@chromium.org
 Issue 167111  has been merged into this issue.

Comment 38 by fer...@gmail.com, May 19 2015

Is there an ETA on this issue being resolved?

It's currently blocking files larger than 500MB from being seeded on https://instant.io. GitHub issue here: https://github.com/feross/webtorrent/issues/319
I'm planning on this quarter (by July 1st)

More info:
bit.ly/BlobRefactorPreso
bit.ly/BlobRefactorPreso2
bit.ly/BlobPlan

Comment 40 by fer...@gmail.com, May 19 2015

Excellent. Thanks for sharing your plans, Daniel.
Project Member

Comment 41 by bugdroid1@chromium.org, Jun 15 2015

The following revision refers to this bug:
  http://src.chromium.org/viewvc/blink?view=rev&rev=197142

------------------------------------------------------------------
r197142 | dmurph@chromium.org | 2015-06-15T22:12:29.775696Z

Changed paths:
   M http://src.chromium.org/viewvc/blink/trunk/Source/platform/exported/WebThreadSafeData.cpp?r1=197142&r2=197141&pathrev=197142
   M http://src.chromium.org/viewvc/blink/trunk/public/platform/WebBlobRegistry.h?r1=197142&r2=197141&pathrev=197142
   M http://src.chromium.org/viewvc/blink/trunk/public/platform/WebBlobData.h?r1=197142&r2=197141&pathrev=197142
   M http://src.chromium.org/viewvc/blink/trunk/public/platform/WebThreadSafeData.h?r1=197142&r2=197141&pathrev=197142

[Blob] Dependencies for blob storage testing

* Added a way to populate the memory in a WebThreadSafeData
  for testing support in content/
* Added new blob registration method using the Builder pattern in the WebBlobRegistry class.

BUG= 375297 

Review URL: https://codereview.chromium.org/1162773003
-----------------------------------------------------------------
Project Member

Comment 42 by bugdroid1@chromium.org, Jun 29 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8a2ebfa1cb92700c36390a6ce9e83e8810655a62

commit 8a2ebfa1cb92700c36390a6ce9e83e8810655a62
Author: dmurph <dmurph@chromium.org>
Date: Mon Jun 29 21:30:14 2015

[Blob] Blob memory consolidation & blobregistry migration.

The blob consolidation class will hold the resources for the new asynchronous protocol, where it will keep the memory around while we wait for the browser to request it.

This is a multi-part patch.  The next steps for Builder migration are:
 * Change other users of WebBlobRegistry to use the Builder pattern
   (mock_web_blob_registry.cc)
 * Change blink to use the builder pattern and stop using registerBlob
 * Remove WebBlobData and registerBlob method

The next steps for asynchronous blob transport:
 * Add BlobStorageDispatcher with new IPC messages, and add asynchronous
   callbacks in BlobStorageContext

See:
https://bit.ly/BlobStorageRefactor

This depends on blink patch here:
https://codereview.chromium.org/1162773003

BUG= 375297 

Review URL: https://codereview.chromium.org/1183713003

Cr-Commit-Position: refs/heads/master@{#336648}

[add] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/child/blob_storage/blob_consolidation.cc
[add] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/child/blob_storage/blob_consolidation.h
[add] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/child/blob_storage/blob_consolidation_unittest.cc
[modify] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/child/webblobregistry_impl.cc
[modify] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/child/webblobregistry_impl.h
[modify] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/content_child.gypi
[modify] http://crrev.com/8a2ebfa1cb92700c36390a6ce9e83e8810655a62/content/content_tests.gypi

Any progress on this issue? It's been a while since there were any updates, and this bug is quite the blocker.

Comment 44 by fer...@gmail.com, Jul 20 2015

Much as I dislike +1 comments, I want to second the sentiment in Eric's comment. This bug continues to cause a sub-par user experience for Chrome users who use my app.
Implementation work is ongoing. You can see the "bugdroid" comments above showing incremental work landing.

@jsbell, as far as I can tell, the last [modify] to content_tests.gypi was from 3 weeks ago, and it's hard to tell if it's really relevant to this bug or it's some how auto linked. 

The commit message is " [Blob] Blob memory consolidation & blobregistry migration."

I would really love some news on the actual status of the merges. 

I tried to dig through commits and look at code reviews etc, but I am not familiar enough to know what is happening in terms of getting this fixed.
big patches are in the works. It's still being actively worked on :)

Comment 48 by Deleted ...@, Aug 17 2015

Ping, I have a pretty big work project hinging on this. Was wondering if there were any updated ETAs?
Blocking: chromium:418743
 Issue 505219  has been merged into this issue.
Hi, joining the party...

I have been using blobs for a while but things started to work very recently this this refactor. I'm using blobs to write out big files with the FileWriter API from an extension. It all used to work fine even if I had to put work arounds for 512MB limit but now it's barely working. 

Writing a file returns me an InvalidStateError after a while and I can't write files anymore. It looks the problem is coming from failing at getting a new blob (which I create from arraybuffers). 

I managed to get the point of failure in the debugger and I'm still investigating and this is where it fails:

void BlobURLRequestJob::DidStart() {
  current_file_chunk_number_ = 0;
  error_ = false;

  // We only support GET request per the spec.
  if (request()->method() != "GET") {
    NotifyFailure(net::ERR_METHOD_NOT_SUPPORTED);
    return;
  }

  // If the blob data is not present, bail out.
  if (!blob_data_) {
    NotifyFailure(net::ERR_FILE_NOT_FOUND);  <---------------------------------------- FAIL HERE
    return;
  }

  CountSize();
}

It's a huge problem for us as we use this extension to transfer big files and it has become a big part of our pipeline. Can anyone look at this?

Status: Started
As noted above, this is actively being worked on.

Good news. Is there any ETA and/or a work around until this is fixed?
Labels: Hotlist-Recharge
This issue likely requires triage.  The current issue owner maybe inactive (i.e. hasn't fixed an issue in the last 30 days).  Thanks for helping out!

-Anthony
Project Member

Comment 55 by bugdroid1@chromium.org, Sep 30 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/02561552a57ed8792a8ebb2676bad485e1d99605

commit 02561552a57ed8792a8ebb2676bad485e1d99605
Author: dmurph <dmurph@chromium.org>
Date: Wed Sep 30 02:03:31 2015

[Blob] BlobReader class & tests, and removal of all redundant reading.

* New BlobReader class & tests
* Removal of other reading code, which now uses the BlobReader
* Removal of unnecessary UploadDiskCacheEntryElementReader
* Removal of blob expansion logic in UploadDataStreamBuilder

Now it's very easy for anyone in browserland to read blobs instead of
having to do url requests, and it's also easy for anyone to add new
blob backing storage.

This is a prerequisite for the new blob async transportation, see here:
goto/BlobPaging

BUG= 138051 , 375297 

Review URL: https://codereview.chromium.org/1337153002

Cr-Commit-Position: refs/heads/master@{#351470}

[add] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/content/browser/fileapi/blob_reader_unittest.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/content/browser/fileapi/blob_url_request_job_unittest.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/content/browser/loader/upload_data_stream_builder.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/content/browser/loader/upload_data_stream_builder.h
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/content/browser/loader/upload_data_stream_builder_unittest.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/content/content_tests.gypi
[delete] http://crrev.com/112b86500a3a4e4969bd9f9584ee88490e16937e/net/base/upload_disk_cache_entry_element_reader.cc
[delete] http://crrev.com/112b86500a3a4e4969bd9f9584ee88490e16937e/net/base/upload_disk_cache_entry_element_reader.h
[delete] http://crrev.com/112b86500a3a4e4969bd9f9584ee88490e16937e/net/base/upload_disk_cache_entry_element_reader_unittest.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/net/base/upload_element_reader.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/net/base/upload_element_reader.h
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/net/net.gypi
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/BUILD.gn
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_data_handle.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_data_handle.h
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_data_snapshot.h
[add] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_reader.cc
[add] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_reader.h
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_url_request_job.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_url_request_job.h
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_url_request_job_factory.cc
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/blob_url_request_job_factory.h
[add] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/upload_blob_element_reader.cc
[add] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/browser/blob/upload_blob_element_reader.h
[modify] http://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605/storage/storage_browser.gyp

Project Member

Comment 56 by bugdroid1@chromium.org, Sep 30 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539

commit 929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539
Author: kinuko <kinuko@chromium.org>
Date: Wed Sep 30 03:56:42 2015

Revert of [Blob] BlobReader class & tests, and removal of all redundant reading. (patchset #15 id:280001 of https://codereview.chromium.org/1337153002/ )

Reason for revert:
BlobReaderTest.StateErrors failing on multiple bots.

../../content/browser/fileapi/blob_reader_unittest.cc:985: Failure
Death test: reader_->Read(buffer.get(), 10, &bytes_read, base::Bind(&SetValue<int>, &async_bytes_read))
    Result: failed to die.
 Error msg:
[  DEATH   ]

Original issue's description:
> [Blob] BlobReader class & tests, and removal of all redundant reading.
>
> * New BlobReader class & tests
> * Removal of other reading code, which now uses the BlobReader
> * Removal of unnecessary UploadDiskCacheEntryElementReader
> * Removal of blob expansion logic in UploadDataStreamBuilder
>
> Now it's very easy for anyone in browserland to read blobs instead of
> having to do url requests, and it's also easy for anyone to add new
> blob backing storage.
>
> This is a prerequisite for the new blob async transportation, see here:
> goto/BlobPaging
>
> BUG= 138051 , 375297 
>
> Committed: https://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605
> Cr-Commit-Position: refs/heads/master@{#351470}

TBR=michaeln@chromium.org,mmenke@chromium.org,dmurph@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 138051 , 375297 

Review URL: https://codereview.chromium.org/1376123002

Cr-Commit-Position: refs/heads/master@{#351485}

[delete] http://crrev.com/b9ecdc94bc359c7d16b9c5a93492fcb1e532ef7a/content/browser/fileapi/blob_reader_unittest.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/content/browser/fileapi/blob_url_request_job_unittest.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/content/browser/loader/upload_data_stream_builder.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/content/browser/loader/upload_data_stream_builder.h
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/content/browser/loader/upload_data_stream_builder_unittest.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/content/content_tests.gypi
[add] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/net/base/upload_disk_cache_entry_element_reader.cc
[add] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/net/base/upload_disk_cache_entry_element_reader.h
[add] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/net/base/upload_disk_cache_entry_element_reader_unittest.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/net/base/upload_element_reader.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/net/base/upload_element_reader.h
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/net/net.gypi
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/BUILD.gn
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_data_handle.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_data_handle.h
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_data_snapshot.h
[delete] http://crrev.com/b9ecdc94bc359c7d16b9c5a93492fcb1e532ef7a/storage/browser/blob/blob_reader.cc
[delete] http://crrev.com/b9ecdc94bc359c7d16b9c5a93492fcb1e532ef7a/storage/browser/blob/blob_reader.h
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_url_request_job.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_url_request_job.h
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_url_request_job_factory.cc
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/browser/blob/blob_url_request_job_factory.h
[delete] http://crrev.com/b9ecdc94bc359c7d16b9c5a93492fcb1e532ef7a/storage/browser/blob/upload_blob_element_reader.cc
[delete] http://crrev.com/b9ecdc94bc359c7d16b9c5a93492fcb1e532ef7a/storage/browser/blob/upload_blob_element_reader.h
[modify] http://crrev.com/929e1ea7fa9fe4f5c76d2506bc4a9cc1d7a4c539/storage/storage_browser.gyp

Project Member

Comment 57 by bugdroid1@chromium.org, Sep 30 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/164f0b76c3a8156b95c36f17c95c452e8f3b9765

commit 164f0b76c3a8156b95c36f17c95c452e8f3b9765
Author: dmurph <dmurph@chromium.org>
Date: Wed Sep 30 18:49:01 2015

[Blob] BlobReader class & tests, and removal of all redundant reading.

* New BlobReader class & tests
* Removal of other reading code, which now uses the BlobReader
* Removal of unnecessary UploadDiskCacheEntryElementReader
* Removal of blob expansion logic in UploadDataStreamBuilder

Now it's very easy for anyone in browserland to read blobs instead of
having to do url requests, and it's also easy for anyone to add new
blob backing storage.

This is a prerequisite for the new blob async transportation, see here:
goto/BlobPaging

BUG= 138051 , 375297 

Committed: https://crrev.com/02561552a57ed8792a8ebb2676bad485e1d99605
Cr-Commit-Position: refs/heads/master@{#351470}

Review URL: https://codereview.chromium.org/1337153002

Cr-Commit-Position: refs/heads/master@{#351611}

[add] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/content/browser/fileapi/blob_reader_unittest.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/content/browser/fileapi/blob_url_request_job_unittest.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/content/browser/loader/upload_data_stream_builder.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/content/browser/loader/upload_data_stream_builder.h
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/content/browser/loader/upload_data_stream_builder_unittest.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/content/content_tests.gypi
[delete] http://crrev.com/72869406e9af49285a992c008a81bdfc99ce9f5b/net/base/upload_disk_cache_entry_element_reader.cc
[delete] http://crrev.com/72869406e9af49285a992c008a81bdfc99ce9f5b/net/base/upload_disk_cache_entry_element_reader.h
[delete] http://crrev.com/72869406e9af49285a992c008a81bdfc99ce9f5b/net/base/upload_disk_cache_entry_element_reader_unittest.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/net/base/upload_element_reader.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/net/base/upload_element_reader.h
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/net/net.gypi
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/BUILD.gn
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_data_handle.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_data_handle.h
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_data_snapshot.h
[add] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_reader.cc
[add] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_reader.h
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_url_request_job.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_url_request_job.h
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_url_request_job_factory.cc
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/blob_url_request_job_factory.h
[add] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/upload_blob_element_reader.cc
[add] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/browser/blob/upload_blob_element_reader.h
[modify] http://crrev.com/164f0b76c3a8156b95c36f17c95c452e8f3b9765/storage/storage_browser.gyp

Issue 265670 has been merged into this issue.
Cc: ajha@chromium.org
 Issue 382415  has been merged into this issue.
Project Member

Comment 60 by bugdroid1@chromium.org, Oct 14 2015

Project Member

Comment 61 by bugdroid1@chromium.org, Oct 31 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0140342861a6aa998ffbf44ecfdc360305f560bc

commit 0140342861a6aa998ffbf44ecfdc360305f560bc
Author: dmurph <dmurph@chromium.org>
Date: Sat Oct 31 01:04:52 2015

[BlobAsync] Patch 2: Common Constants
This CL contains constants for the
Blob Async Transport refactor.

Note: This CL is currently a no-op, hookup is in the 'Hookup' patch.

Patches:
1: https://codereview.chromium.org/1287303002 (Committed!)
2: https://codereview.chromium.org/1288373002
3: https://codereview.chromium.org/1292523002
4: https://codereview.chromium.org/1098853003
Hookup: https://codereview.chromium.org/1234813004

BUG= 375297 

Review URL: https://codereview.chromium.org/1288373002

Cr-Commit-Position: refs/heads/master@{#357248}

[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/content/browser/loader/upload_data_stream_builder.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/content/common/resource_messages.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/browser/blob/blob_data_builder.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/browser/blob/blob_data_builder.h
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/browser/blob/blob_data_item.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/browser/blob/blob_data_item.h
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/browser/blob/blob_reader.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/browser/blob/view_blob_internals_job.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/BUILD.gn
[add] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/blob_storage/blob_item_bytes_request.cc
[add] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/blob_storage/blob_item_bytes_request.h
[add] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/blob_storage/blob_item_bytes_response.cc
[add] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/blob_storage/blob_item_bytes_response.h
[add] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/blob_storage/blob_storage_constants.h
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/data_element.cc
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/common/data_element.h
[modify] http://crrev.com/0140342861a6aa998ffbf44ecfdc360305f560bc/storage/storage_common.gyp

Project Member

Comment 62 by bugdroid1@chromium.org, Oct 31 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/910d0ee4c7534c7366f3b1548c50102c67e56007

commit 910d0ee4c7534c7366f3b1548c50102c67e56007
Author: dmurph <dmurph@chromium.org>
Date: Sat Oct 31 02:10:22 2015

[BlobAsync] Patch 3: Renderer Classes & Logic
Note: This patch is a no-op.

Patches:
1: https://codereview.chromium.org/1287303002
2: https://codereview.chromium.org/1288373002
3: https://codereview.chromium.org/1292523002
4: https://codereview.chromium.org/1098853003
Hookup: https://codereview.chromium.org/1234813004

BUG= 375297 

Review URL: https://codereview.chromium.org/1292523002

Cr-Commit-Position: refs/heads/master@{#357258}

[add] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/content/child/blob_storage/blob_transport_controller.cc
[add] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/content/child/blob_storage/blob_transport_controller.h
[add] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/content/child/blob_storage/blob_transport_controller_unittest.cc
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/content/content_child.gypi
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/content/content_tests.gypi
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/browser/blob/blob_data_item.cc
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/browser/blob/blob_data_item.h
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/common/blob_storage/blob_item_bytes_request.cc
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/common/blob_storage/blob_item_bytes_request.h
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/common/blob_storage/blob_item_bytes_response.cc
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/common/blob_storage/blob_item_bytes_response.h
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/common/data_element.cc
[modify] http://crrev.com/910d0ee4c7534c7366f3b1548c50102c67e56007/storage/common/data_element.h

So, whats going on with this? Lots of commits, no idea whats happening?
It seems that it related to  issue #554217 . The BLOB creation method checks for the 500MB limit for BLOBs without making sure the garbage collector has been run. I guess the proper fix is to first run the garbage collector when the 500MB limit is reached. This would dispose the unused BLOBs and make space for the newly allocated BLOBs.
Blocking: chromium:554019
 Issue 559375  has been merged into this issue.
Project Member

Comment 67 by bugdroid1@chromium.org, Dec 2 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/233c81baddf00838335bb712a3f8cd67fd327e71

commit 233c81baddf00838335bb712a3f8cd67fd327e71
Author: dmurph <dmurph@chromium.org>
Date: Wed Dec 02 21:51:17 2015

[BlobAsync] Patch 4: Browser Classes & Logic.
This patch encompasses all browser classes and logic for blob async transportation. This is a no-op.

Patches:
1: https://codereview.chromium.org/1287303002 (committed)
2: https://codereview.chromium.org/1288373002 (committed)
3: https://codereview.chromium.org/1292523002 (committed)
4: https://codereview.chromium.org/1098853003
Hookup: https://codereview.chromium.org/1234813004

See https://bit.ly/BlobStorageRefactor

BUG= 375297 

Review URL: https://codereview.chromium.org/1098853003

Cr-Commit-Position: refs/heads/master@{#362802}

[add] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/content/browser/blob_storage/blob_async_builder_host_unittest.cc
[add] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/content/browser/blob_storage/blob_async_transport_strategy_unittest.cc
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/content/content_tests.gypi
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/BUILD.gn
[add] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_async_builder_host.cc
[add] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_async_builder_host.h
[add] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_async_transport_strategy.cc
[add] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_async_transport_strategy.h
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_data_builder.cc
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_data_builder.h
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_storage_context.cc
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/browser/blob/blob_storage_context.h
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/common/data_element.h
[modify] http://crrev.com/233c81baddf00838335bb712a3f8cd67fd327e71/storage/storage_browser.gyp

Comment 68 by fer...@gmail.com, Feb 28 2016

Can we get an update on this issue? It's blocking lots of powerful browser applications from working correctly in Chrome.
I'm working on this almost daily :)
This is actually a pretty big change.
Is there any way to track progress on this? We're pretty eager to see some work on this...

Comment 71 by a...@rigo.sk, Mar 15 2016

Yes, you can track it at https://codereview.chromium.org/1234813004 . As you can see it it's very close.

Comment 72 by ise...@gmail.com, Mar 28 2016

Is there any effort to change the standard to fix the useless revocation API? I don't know when it's safe to revoke a blob, since there's nothing like a "downloadfinished" event. If there was, I would be able to use code like this for revocation: https://github.com/eligrey/FileSaver.js/blob/62d219a0fac54b94cd4f230e7bfc55aa3f8dcfa4/FileSaver.js#L53-L68
Project Member

Comment 73 by bugdroid1@chromium.org, Mar 30 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1fb98480c61c563587cbb01de87cfad180fec942

commit 1fb98480c61c563587cbb01de87cfad180fec942
Author: dmurph <dmurph@chromium.org>
Date: Wed Mar 30 21:14:26 2016

[BlobAsync] Asynchronous Blob Construction Final Patch
This is the final patch that hooks up the blob storage system to the new
async protocol. In this patch we:
 * Hook up the new classes and modules we created in the previous patches.
 * Remove the old way of creating blobs.
 * Create a few classes like blob_message_filter and blob_dispatcher_host
   for handing IPC messages.

The result of this change makes renderer-initiated blob construction
asynchronous instead of synchronous. So constructing new blobs should be
faster, but the time from construction to read should be the same, as
the reading still has to wait for the blob to be transferred to the
browser.

Because we let the renderer continue before we've sent all of the data,
we use ChildProcess::AddRefProcess() system to keep the renderer alive
while we transfer data.

Patches:
1: https://codereview.chromium.org/1287303002 (committed!)
2: https://codereview.chromium.org/1288373002 (committed!)
3: https://codereview.chromium.org/1292523002 (committed!)
4: https://codereview.chromium.org/1098853003 (committed!)
Hookup: https://codereview.chromium.org/1234813004

BUG= 375297 

Review URL: https://codereview.chromium.org/1234813004

Cr-Commit-Position: refs/heads/master@{#384093}

[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/bad_message.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/blob_storage/blob_async_builder_host_unittest.cc
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/blob_storage/blob_async_transport_request_builder_unittest.cc
[delete] https://crrev.com/44af6ef24f6a9bc2822e4f294919ae71f7288015/content/browser/blob_storage/blob_async_transport_strategy_unittest.cc
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/blob_storage/blob_dispatcher_host.cc
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/blob_storage/blob_dispatcher_host.h
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/blob_storage/blob_dispatcher_host_unittest.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/blob_storage/blob_storage_registry_unittest.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/fileapi/blob_reader_unittest.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/fileapi/blob_storage_context_unittest.cc
[delete] https://crrev.com/44af6ef24f6a9bc2822e4f294919ae71f7288015/content/browser/fileapi/blob_storage_host.cc
[delete] https://crrev.com/44af6ef24f6a9bc2822e4f294919ae71f7288015/content/browser/fileapi/blob_storage_host.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/fileapi/fileapi_message_filter.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/fileapi/fileapi_message_filter.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/browser/renderer_host/render_process_host_impl.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_consolidation.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_consolidation.h
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_message_filter.cc
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_message_filter.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_transport_controller.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/blob_storage/blob_transport_controller_unittest.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/webblobregistry_impl.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/child/webblobregistry_impl.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/common/fileapi/webblob_messages.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/content_browser.gypi
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/content_child.gypi
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/content_common.gypi
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/content_tests.gypi
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/public/common/content_switches.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/renderer/render_thread_impl.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/renderer/render_thread_impl.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/content/renderer/renderer_blink_platform_impl.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/BUILD.gn
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_async_builder_host.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_async_builder_host.h
[rename] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_async_transport_request_builder.cc
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_async_transport_request_builder.h
[delete] https://crrev.com/44af6ef24f6a9bc2822e4f294919ae71f7288015/storage/browser/blob/blob_async_transport_strategy.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_data_builder.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_data_handle.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_data_handle.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_data_snapshot.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_data_snapshot.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_reader.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_reader.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_storage_context.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_storage_context.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_storage_registry.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_storage_registry.h
[add] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/blob_transport_result.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/internal_blob_data.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/internal_blob_data.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/shareable_blob_data_item.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/view_blob_internals_job.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/browser/blob/view_blob_internals_job.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/common/blob_storage/blob_item_bytes_request.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/common/blob_storage/blob_item_bytes_request.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/common/blob_storage/blob_item_bytes_response.cc
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/common/blob_storage/blob_item_bytes_response.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/common/blob_storage/blob_storage_constants.h
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/storage/storage_browser.gyp
[modify] https://crrev.com/1fb98480c61c563587cbb01de87cfad180fec942/tools/metrics/histograms/histograms.xml

Note: The last patch is just the async transportation. Paging to disk/ removing the limit is in the next patch.
Project Member

Comment 75 by bugdroid1@chromium.org, Apr 6 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/90774e39bb24b1c531e9b3aba16c77ab96675fb3

commit 90774e39bb24b1c531e9b3aba16c77ab96675fb3
Author: dmurph <dmurph@chromium.org>
Date: Wed Apr 06 16:20:43 2016

[BlobAsync] Faster shortcuttin, make renderer controller leaky & alive.

This should fix strange memory access crashes observed in the
following bugs by doing the following things:
1. We make the controller a leaky lazy instance so it isn't destructed.
2. We make one less message loop hop before we call AddProcessRef, so
   hopefully our process won't be shutting down in the middle of a
   transfer.
3. We send the 'descriptions' as early as possible if we're below the
   IPC threshold, which should help some of the speed regressions.
4. We prevent sudden shutdowns (like tab bar closing) by using
   blink::Platform::current()->suddenTerminationChanged(false);

BUG=600462,600435, 599490 , 599416 , 375297 

Review URL: https://codereview.chromium.org/1853333003

Cr-Commit-Position: refs/heads/master@{#385472}

[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/browser/blob_storage/blob_async_builder_host_unittest.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/browser/blob_storage/blob_dispatcher_host.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/browser/blob_storage/blob_dispatcher_host.h
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/browser/blob_storage/blob_dispatcher_host_unittest.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/child/blob_storage/blob_transport_controller.h
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/child/blob_storage/blob_transport_controller_unittest.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/child/webblobregistry_impl.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/content/child/webblobregistry_impl.h
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/storage/browser/blob/blob_async_builder_host.cc
[modify] https://crrev.com/90774e39bb24b1c531e9b3aba16c77ab96675fb3/storage/browser/blob/blob_async_builder_host.h

Project Member

Comment 76 by bugdroid1@chromium.org, May 20 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3

commit c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3
Author: dmurph <dmurph@chromium.org>
Date: Fri May 20 03:41:15 2016

[BlobAsync] Renderer support for blob file writing.
This implements the logic where the renderer is writing blob pieces
directly to disk, instead of sending the data to the browser.browser

R=kinuko, michaeln
BUG= 375297 

Review-Url: https://codereview.chromium.org/1414123002
Cr-Commit-Position: refs/heads/master@{#394985}

[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_consolidation.cc
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_consolidation.h
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_consolidation_unittest.cc
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_message_filter.cc
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_message_filter.h
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_transport_controller.h
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/blob_transport_controller_unittest.cc
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/child/blob_storage/webblobregistry_impl.h
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/content/renderer/render_thread_impl.cc
[modify] https://crrev.com/c27bc134ef590aa1435f9ccd6bcb35219c9b3cf3/tools/metrics/histograms/histograms.xml

For those who want to follow along, I'm making the final patch here:
https://codereview.chromium.org/1528233004
Components: Blink>Storage

Comment 79 by a...@rigo.sk, Jul 11 2016

I think the new link is:
https://codereview.chromium.org/2055053003/
 Issue 460736  has been merged into this issue.
Hi,there,

Are there any update or plan on this issue? Thanks.
Blocking: 625934
Cc: u...@chromium.org adamk@chromium.org hpayer@chromium.org kavvaru@chromium.org fmalita@chromium.org
 Issue 468718  has been merged into this issue.
Cc: domenic@chromium.org
 Issue 554217  has been merged into this issue.
 Issue 554019  has been merged into this issue.
Cc: yangguo@chromium.org jochen@chromium.org
 Issue 625934  has been merged into this issue.
Labels: dmurph-project-tracking

Comment 88 by a...@scirra.com, Oct 19 2016

Any news on this?
It seems to be fixed with Chrome v55 (will be released early December). You can already beta test this version.
Thanks, I'll try it with v55.

https://bugs.chromium.org/p/chromium/issues/detail?id=625934

Comment 91 by a...@scirra.com, Oct 19 2016

Doesn't look like it's fixed to me, the blob-test.html file posted earlier still fails here in Canary 56.0.2895.0. Besides, the bug isn't marked fixed yet.
Our online crc32 test also receives an out-of-memory exception (100 iterations).
http://www.backload.org/lab/crc32/run.html

Firefox and Edge are completing the test with no exceptions.

ref: https://bugs.chromium.org/p/chromium/issues/detail?id=625934
Labels: Restrict-AddIssueComment-EditIssue
#88,91,92: The bug status says it's started, so an engineer is working on it. The code changes required are complex and will take a while to get reviewed. I'm going to restrict comments here until we ship the changes so folks can focus on development.
Project Member

Comment 94 by bugdroid1@chromium.org, Oct 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9

commit 0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9
Author: dmurph <dmurph@chromium.org>
Date: Fri Oct 21 21:51:20 2016

[BlobStorage] BlobMemoryController & tests

This is a split off of https://codereview.chromium.org/2055053003/.

This adds the BlobMemoryController & tests, as well as any other small
required changes. I avoided doing too much refactoring (like the move
to BlobStatus everywhere, and removing BlobStorageRegistry::Entry for
InternalBlobData) to just focus on the BlobMemoryController API.

BUG= 375297 
R=michaeln@chromium.org,kinuko@chromium.org,mek@chromium.org

Review-Url: https://chromiumcodereview.appspot.com/2339933004
Cr-Commit-Position: refs/heads/master@{#426901}

[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/content/browser/blob_storage/blob_async_transport_request_builder_unittest.cc
[add] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/content/browser/blob_storage/blob_data_builder_unittest.cc
[add] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/content/browser/blob_storage/blob_memory_controller_unittest.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/content/test/BUILD.gn
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/BUILD.gn
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_async_transport_request_builder.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_data_builder.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_data_builder.h
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_data_item.h
[add] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_memory_controller.cc
[add] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_memory_controller.h
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/blob_storage_context.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/internal_blob_data.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/shareable_blob_data_item.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/browser/blob/shareable_blob_data_item.h
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/common/BUILD.gn
[add] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/common/blob_storage/blob_storage_constants.cc
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/storage/common/blob_storage/blob_storage_constants.h
[modify] https://crrev.com/0b0e36d9a68b386ab2e8e6c921122bb0f6e7e0e9/tools/metrics/histograms/histograms.xml

Alrighty! That last changed basically added the quota system for blob storage, including file paging and general file management. We still need to refactor the current storage system to immediately build blobs even if there are pending parents (taking this out of the transport part), simplifying the transport part now that it doesn't have to wait for pending parent blobs, and add file support to both transport and the system.
Project Member

Comment 96 by bugdroid1@chromium.org, Oct 22 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0c79c5f5b228452450aa542ff33d0b49209ee429

commit 0c79c5f5b228452450aa542ff33d0b49209ee429
Author: reillyg <reillyg@chromium.org>
Date: Sat Oct 22 03:49:30 2016

Fix std::memset usage in blob_memory_controller_unittest.cc.

This change exchanges the 2nd and 3rd parameters to std::memset() in a
number of places where the value to set and the length of the buffer
were inverted. The resulting incompletely initialized buffers triggered
MSan failures.

BUG= 375297 
TBR=dmurph@chromium.org

Review-Url: https://chromiumcodereview.appspot.com/2447473002
Cr-Commit-Position: refs/heads/master@{#426966}

[modify] https://crrev.com/0c79c5f5b228452450aa542ff33d0b49209ee429/content/browser/blob_storage/blob_memory_controller_unittest.cc

Currently working on:
https://codereview.chromium.org/2448353002

Comment 98 by adamk@chromium.org, Oct 31 2016

Cc: -adamk@chromium.org
Project Member

Comment 99 by bugdroid1@chromium.org, Nov 17 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2

commit cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2
Author: dmurph <dmurph@chromium.org>
Date: Thu Nov 17 21:25:07 2016

[BlobAsync] Moving async handling into BlobStorageContext & quota out.

This change moves async building logic, including waiting on referenced
blobs, out of the BlobAsyncBuilderHost into the BlobStorageContext. It
also moves the quota handling out of BlobStorageContext and into
BlobMemoryController.

This change does not enable paging to files (or backing blobs by files
right away) yet. It's just doing the required system and state
refactoring.

BUG= 375297 , 612261 , 600528 
R=michaeln@chromium.org,kinuko@chromium.org,pwnall@chromium.org,mek@chromium.org

Review-Url: https://codereview.chromium.org/2448353002
Cr-Commit-Position: refs/heads/master@{#432960}

[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/chrome/browser/chrome_security_exploit_browsertest.cc
[delete] https://crrev.com/028c9afaa41a29d47e7e40202082b89a667c445f/content/browser/blob_storage/blob_async_builder_host_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_dispatcher_host.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_dispatcher_host.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_dispatcher_host_unittest.cc
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_flattener_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_memory_controller_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_reader_unittest.cc
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_slice_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_storage_context_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_storage_registry_unittest.cc
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_transport_host_unittest.cc
[rename] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/blob_storage/blob_transport_request_builder_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/service_worker/service_worker_url_request_job.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/browser/service_worker/service_worker_url_request_job.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/blob_consolidation.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/blob_message_filter.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/blob_message_filter.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/blob_transport_controller.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/blob_transport_controller_unittest.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/webblobregistry_impl.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/child/blob_storage/webblobregistry_impl.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/common/fileapi/webblob_messages.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/content/test/BUILD.gn
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/net/log/net_log_event_type_list.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/BUILD.gn
[delete] https://crrev.com/028c9afaa41a29d47e7e40202082b89a667c445f/storage/browser/blob/blob_async_builder_host.cc
[delete] https://crrev.com/028c9afaa41a29d47e7e40202082b89a667c445f/storage/browser/blob/blob_async_builder_host.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_data_builder.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_data_builder.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_data_handle.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_data_handle.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_data_item.h
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_entry.cc
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_entry.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_memory_controller.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_memory_controller.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_reader.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_reader.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_storage_context.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_storage_context.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_storage_registry.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_storage_registry.h
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_transport_host.cc
[add] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_transport_host.h
[rename] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_transport_request_builder.cc
[rename] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/blob_transport_request_builder.h
[delete] https://crrev.com/028c9afaa41a29d47e7e40202082b89a667c445f/storage/browser/blob/blob_transport_result.h
[delete] https://crrev.com/028c9afaa41a29d47e7e40202082b89a667c445f/storage/browser/blob/internal_blob_data.cc
[delete] https://crrev.com/028c9afaa41a29d47e7e40202082b89a667c445f/storage/browser/blob/internal_blob_data.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/shareable_blob_data_item.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/shareable_blob_data_item.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/view_blob_internals_job.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/browser/blob/view_blob_internals_job.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/common/blob_storage/blob_storage_constants.cc
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/common/blob_storage/blob_storage_constants.h
[modify] https://crrev.com/cf848a6957fb402bdffeb5a2ce51e9f0cbbb98f2/storage/common/data_element.h

Patch that enables disk paging is here:
https://codereview.chromium.org/2516713002/

For those of you waiting for this feature, I'm very interested in your use-case. What are you building that doesn't work yet? Please email me at dmurph-at-chromium.org. Thanks!
Project Member

Comment 101 by bugdroid1@chromium.org, Dec 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/89bdfe0bd9499962f9f05c30a72d46fc6510d45a

commit 89bdfe0bd9499962f9f05c30a72d46fc6510d45a
Author: dmurph <dmurph@chromium.org>
Date: Mon Dec 05 21:33:50 2016

[BlobStorage] Enabling disk storage and paging.

Adds support for saving blobs to disk by either paging them to disk
or sending files directly to the renderer.

Note: Disk is turned off by default (limit=0). So other than tests
that explicitly set a higher disk limit, this patch should be a NOOP.
The next patch will configure this limit (and the current RAM limit)
by the OS, disk size, and available RAM.

R=michaeln,kinuko
BUG= 375297 

Review-Url: https://codereview.chromium.org/2516713002
Cr-Commit-Position: refs/heads/master@{#436415}

[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/blob_flattener_unittest.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/blob_memory_controller_unittest.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/blob_slice_unittest.cc
[add] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/blob_storage_browsertest.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/blob_storage_context_unittest.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/chrome_blob_storage_context.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/browser/blob_storage/chrome_blob_storage_context.h
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/test/BUILD.gn
[add] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/test/data/blob_storage/blob_creation_and_slicing.html
[add] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/content/test/data/blob_storage/common.js
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_entry.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_entry.h
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_memory_controller.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_memory_controller.h
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_reader.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_storage_context.cc
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/browser/blob/blob_storage_context.h
[modify] https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a/storage/common/blob_storage/blob_storage_constants.h

Project Member

Comment 102 by bugdroid1@chromium.org, Dec 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1108e1fbaf71225ba4b729185868e9e2c0e23976

commit 1108e1fbaf71225ba4b729185868e9e2c0e23976
Author: carlosk <carlosk@chromium.org>
Date: Mon Dec 05 23:18:29 2016

Revert of [BlobStorage] Implementing disk. (patchset #18 id:340001 of https://codereview.chromium.org/2516713002/ )

Reason for revert:
Seems to be breaking BlobStorageBrowserTest.BlobCombinations
on "Mac10.9 Tests (dbg)".

Original issue's description:
> [BlobStorage] Enabling disk storage and paging.
>
> Adds support for saving blobs to disk by either paging them to disk
> or sending files directly to the renderer.
>
> Note: Disk is turned off by default (limit=0). So other than tests
> that explicitly set a higher disk limit, this patch should be a NOOP.
> The next patch will configure this limit (and the current RAM limit)
> by the OS, disk size, and available RAM.
>
> R=michaeln,kinuko
> BUG= 375297 
>
> Committed: https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a
> Cr-Commit-Position: refs/heads/master@{#436415}

TBR=mek@chromium.org,kinuko@chromium.org,michaeln@chromium.org,pwnall@chromium.org,dmurph@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 375297 

Review-Url: https://codereview.chromium.org/2550113003
Cr-Commit-Position: refs/heads/master@{#436443}

[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/browser/blob_storage/blob_flattener_unittest.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/browser/blob_storage/blob_memory_controller_unittest.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/browser/blob_storage/blob_slice_unittest.cc
[delete] https://crrev.com/9c958fd2553179870953b398506678a5e1319714/content/browser/blob_storage/blob_storage_browsertest.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/browser/blob_storage/blob_storage_context_unittest.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/browser/blob_storage/chrome_blob_storage_context.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/browser/blob_storage/chrome_blob_storage_context.h
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/content/test/BUILD.gn
[delete] https://crrev.com/9c958fd2553179870953b398506678a5e1319714/content/test/data/blob_storage/blob_creation_and_slicing.html
[delete] https://crrev.com/9c958fd2553179870953b398506678a5e1319714/content/test/data/blob_storage/common.js
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_entry.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_entry.h
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_memory_controller.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_memory_controller.h
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_reader.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_storage_context.cc
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/browser/blob/blob_storage_context.h
[modify] https://crrev.com/1108e1fbaf71225ba4b729185868e9e2c0e23976/storage/common/blob_storage/blob_storage_constants.h

 Issue 671404  has been merged into this issue.
Project Member

Comment 104 by bugdroid1@chromium.org, Dec 6 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/07725e0f2bd75b9649ff9b4a10096a281e56d998

commit 07725e0f2bd75b9649ff9b4a10096a281e56d998
Author: dmurph <dmurph@chromium.org>
Date: Tue Dec 06 02:28:12 2016

[BlobStorage] Enabling disk storage and paging.

Adds support for saving blobs to disk by either paging them to disk
or sending files directly to the renderer.

Note: Disk is turned off by default (limit=0). So other than tests
that explicitly set a higher disk limit, this patch should be a NOOP.
The next patch will configure this limit (and the current RAM limit)
by the OS, disk size, and available RAM.

R=michaeln,kinuko
BUG= 375297 

Committed: https://crrev.com/89bdfe0bd9499962f9f05c30a72d46fc6510d45a
Review-Url: https://codereview.chromium.org/2516713002
Cr-Original-Commit-Position: refs/heads/master@{#436415}
Cr-Commit-Position: refs/heads/master@{#436485}

[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/blob_flattener_unittest.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/blob_memory_controller_unittest.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/blob_slice_unittest.cc
[add] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/blob_storage_browsertest.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/blob_storage_context_unittest.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/chrome_blob_storage_context.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/browser/blob_storage/chrome_blob_storage_context.h
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/child/blob_storage/blob_transport_controller.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/test/BUILD.gn
[add] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/test/data/blob_storage/blob_creation_and_slicing.html
[add] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/content/test/data/blob_storage/common.js
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_entry.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_entry.h
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_memory_controller.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_memory_controller.h
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_reader.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_storage_context.cc
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/browser/blob/blob_storage_context.h
[modify] https://crrev.com/07725e0f2bd75b9649ff9b4a10096a281e56d998/storage/common/blob_storage/blob_storage_constants.h

Labels: officialtest
Test failures(duped in C#103  Issue 671404 ) are seen on the Precise 64 trunk of official.desktop builder.


Project Member

Comment 106 by bugdroid1@chromium.org, Jan 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bf104bd8baf5e73b71c82ba60249cb778b8db099

commit bf104bd8baf5e73b71c82ba60249cb778b8db099
Author: dmurph <dmurph@chromium.org>
Date: Thu Jan 12 03:50:37 2017

[BlobStorage] Enabling disk paging and direct storage.

Adds disk configuration calculations that are slightly specialized for
Android, ChromeOS, and general desktop. The max memory space and disk
spaced are set based on the total RAM and disk of the device.

BUG= 375297 

Review-Url: https://codereview.chromium.org/2552153002
Cr-Commit-Position: refs/heads/master@{#443139}

[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/blob_dispatcher_host_unittest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/blob_flattener_unittest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/blob_memory_controller_unittest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/blob_storage_browsertest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/blob_storage_context_unittest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/blob_transport_host_unittest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/blob_storage/chrome_blob_storage_context.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/content/browser/web_contents/web_contents_impl_unittest.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/storage/browser/blob/blob_memory_controller.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/storage/browser/blob/blob_memory_controller.h
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/storage/browser/blob/blob_storage_context.h
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/storage/common/blob_storage/blob_storage_constants.cc
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/storage/common/blob_storage/blob_storage_constants.h
[modify] https://crrev.com/bf104bd8baf5e73b71c82ba60249cb778b8db099/tools/metrics/histograms/histograms.xml

Labels: -Restrict-AddIssueComment-EditIssue
The patch should be in the next canary (1/13).

Please note that there is still a maximum blob quota for all systems, which includes both an in-memory and disk quota (usage of either should be opaque to developers/users, but you can see the backing type in chrome://blob-internals).

In-memory quota:
* 2GB if system is x64 and NOT ChromeOS or Android
* Total RAM amount / 5;

Disk quota:
* Disk size / 2 if ChromeOS (user partition disk size)
* Disk size / 20 if Android
* Disk size / 10 otherwise.

Also, if disk is almost full, we try to keep at least (in-memory quota)*2 disk space available, and we limit the disk quota accordingly.

This will all by typed up into a blob system explainer doc, as well as a best practices post for blobs.

I've verified with the following code that blobs > 500MB can be created on appropriate systems.

var b = new Blob([new Uint8Array(700*1024*1024)], {type: 'application/octet-string'}));
var reader = new FileReader();
reader.onload = function(event) {
  console.log("got load event", event);
}
reader.onerror = function(event) {
    console.log("got error", event);
}
reader.readAsArrayBuffer(b);


Removing the comment restriction.

Status: Fixed (was: Started)
and..... marking as Fixed. It's been a long journey, folks. Here's to a 2 year refactoring project *clink*

Now just hoping it sticks :)
In which version of Chrome can we expect this fix?

Comment 110 by a...@rigo.sk, Jan 12 2017

If I'm right it should be in Chrome 57 as it works in the current Canary. The ETA of the stable Chrome 57 is March 14th. Thank you all the hard work!
 Issue 682188  has been merged into this issue.
Cc: krajshree@chromium.org
 Issue 682185  has been merged into this issue.
Components: Blink>Storage>FileAPI
Components: -Blink>FileAPI
Showing comments 15 - 114 of 114 Older

Sign in to add a comment