New issue
Advanced search Search tips

Issue 728854 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug



Sign in to add a comment

blob: urls triggering net::URLRequestContext::AssertNoURLRequests crashes

Project Member Reported by mattm@chromium.org, Jun 1 2017

Issue description

I looked through a bunch of recent minidumps for net::URLRequestContext::AssertNoURLRequests. The majority were for blob:uuid/... urls, one for blob://see_user_data/, and a few there were unrelated.

Crash URL (note, this is all leaked URLRequests, so you need to check the minidump for the url_buf value in the AssertNoURLRequests stack frame to be sure which cause applies to a given crash report):
https://crash.corp.google.com/browse?q=product.name%20CONTAINS%20%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27browser%27%20AND%20%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27net%3A%3AURLRequestContext%3A%3AAssertNoURLRequests%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

I saw blob: urls in win, mac, and linux minidumps at least.

Assigning based on git blame, please reassign if there's a better person to handle this.

 Issue 701851  also looks relevant, insofar as stopping using URLFetcher should remove the issue.
 

Comment 1 by mattm@chromium.org, Jun 1 2017

(Note regarding last comment: "blob:" urls are created in multiple places, so  Issue 701851  may or may not affect the issue, depending where the leaks are actually coming from.)
Project Member

Comment 2 by sheriffbot@chromium.org, Jun 4 2017

Labels: Fracas FoundIn-M-60
Users experienced this crash on the following builds:

Mac Dev 60.0.3112.10 -  0.29 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 3 by sheriffbot@chromium.org, Jun 13 2017

Labels: FoundIn-M-61
Users experienced this crash on the following builds:

Mac Canary 61.0.3128.0 -  0.25 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)
Linux Beta 60.0.3112.24 -  0.64 CPM, 4 reports, 2 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 4 by sheriffbot@chromium.org, Oct 19 2017

Labels: FoundIn-M-63
Users experienced this crash on the following builds:

Ios Dev 63.0.3231.0 -  10.08 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 5 by sheriffbot@chromium.org, Oct 24 2017

Labels: FoundIn-M-62
Users experienced this crash on the following builds:

Ios Beta 62.0.3202.52 -  7.21 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Owner: ----
Status: Untriaged (was: Assigned)
We're still seeing these, though now the URLs the requests are for tend to look like "blob:uuid/<GUID>".

AssertNoURLRequests is now the #6 crasher on Windows Dev, but I can't confirm if it's also because of these blob URLs.  On stable, 3 of 4 randomly sampled crashers were because of outstanding blob requests, but none of the 400 Windows dev crashers in AssertNoURLRequests have stack traces, so the recent jump may well be due to something else.

Comment 7 by mek@chromium.org, Jan 8 2018

Can you explain what exactly the bug here is? What is this assert asserting, and what kind of situations could be triggering it?

Comment 8 by mek@chromium.org, Jan 8 2018

General changes in this area, in M64 the javascript FileReader API no longer uses blob URLs, so that would explain a reduction of URL requests of the form blob:<origin>/<GUID>. It doesn't sound like that shape of blob URL ever triggered this in the first place, though? I'm also not sure I understand what you're trying to say about hat changed for these URLs? In the original report it already seems to say most of the URLs are "blob:uuid/<GUID>" style URLs?
The issue is that URLRequests are still alive when the URLRequestContext is destroyed (And presumably when ProfileIOData is destroyed as well).  The network stack has never supported URLRequests outliving the URLRequestContext.  When this happens, the network stack forces a crash, and records the URL of a URLRequest that lived too long.  Sample crash:  https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27browser%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27net%3A%3AURLRequestContext%3A%3AAssertNoURLRequests%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.component%3D%27src%2Fnet%2Furl_request%27%20AND%20NOT%20upload_info.discard_minidump&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=9bc897ed5c4c5fa1&index=3#0

We've run into this issue before in objects that directly consume URLRequests (i.e., not requests that go through the renderer's standard request path, but instead things that live in the browser process, and interact directly with their own URLRequests).

In Chrome 63, most of the requests where this happens are for blob URLs.  This was also the case when this bug was reported, back in Chrome 60 (?).  I believe there's some path of make a URLRequest direct through blob infrastructure in Chrome, and I suspect that some consumer of this path is the culprit for these crashes.

This sort of crash is a top crasher in Chrome 65, however, I have not been able to confirm if requests for blob URLs are still the main culprit in Chrome 65, since no Windows Chrome 65 Canary crashes here have memory dumps.  The recent jump may well be because of Mojo loading / network service / PlzNavigate changes, which all significantly refactor consumers of URLRequests.
I intend to dig deeper into the recent regression, to figure out what's going on there, but to the extent of my knowledge, regardless of whether it's related to the recent regression or not, some consumer of blob URLRequests is outliving ProfileIOData without cancelling its URLRequests, and that consumer (Or those consumers) should be identified and fixed.

Comment 11 by mek@chromium.org, Jan 8 2018

Ah, I see. The two main paths I can think of where Blob's are read from te browser process, but through blob URLs are indeed the extensions::BlobReader code-path you mentioned earlier, but also IndexedDB, whenever a blob needs to be written to disk (i.e. roughly the code at https://cs.chromium.org/chromium/src/content/browser/indexed_db/indexed_db_backing_store.cc?l=1593). I don't think either of that changed much in M65 (although the IDB case might happen more often since M64 I think).

And of course it's somewhere on my todo list to get rid of all those cases...
And with the network service, these things may just go away, since we have to completely rewrite everything anyways (And Mojo requests the network service handle out-of-order teardown itself), so, despite what I wrote above, it may not be worth looking into, unless I discover blob URLs are the cause of the recent regression.

Comment 13 by mek@chromium.org, Jan 8 2018

(mostly as note-to-self, blob://see-user-data/ URLs are created by BlobProtocolHandler::CreateBlobRequest, and seem to be caused by CacheStorage, ServiceWorker and FileSystem API related usage. But yeah, all of this is getting rewritten as part of content modularisation/servicication anyway).
Owner: mek@chromium.org
Status: Assigned (was: Untriaged)
assigning to mek@ to get off the triage plate since; please reassign/update status as appropriate
Project Member

Comment 15 by sheriffbot@chromium.org, Jan 18 2018

Labels: FoundIn-M-65
Users experienced this crash on the following builds:

Win Dev 65.0.3322.3 -  0.20 CPM, 17 reports, 17 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 16 by sheriffbot@chromium.org, Jan 23 2018

Labels: FoundIn-M-64
Users experienced this crash on the following builds:

Win Dev 65.0.3322.3 -  0.11 CPM, 41 reports, 37 clients (signature net::URLRequestContext::AssertNoURLRequests)
Ios Beta 64.0.3282.97 -  23.24 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 17 by sheriffbot@chromium.org, Feb 2 2018

Labels: FoundIn-M-66
Users experienced this crash on the following builds:

Win Canary 66.0.3335.0 -  0.25 CPM, 4 reports, 4 clients (signature net::URLRequestContext::AssertNoURLRequests)
Ios Beta 64.0.3282.97 -  4.43 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 18 by sheriffbot@chromium.org, Feb 24 2018

Labels: ReleaseBlock-Dev
This crash has high impact on Chrome's stability.
Signature: net::URLRequestContext::AssertNoURLRequests.
Channel: canary. Platform: win.
Labeling issue 728854 with ReleaseBlock-Dev.


If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Comment 19 by emaxx@chromium.org, Feb 26 2018

Cc: mmenke@chromium.org
mmenke@: Should we file another bug for the recent spike of this signature on Canary 66.0.3352.0 (Win and Mac)? I'm suspecting the previous investigation that pointed towards blobs: urls may have nothing to do with this new spike.
Project Member

Comment 20 by sheriffbot@chromium.org, Feb 26 2018

This issue is marked as a release blocker with no milestone associated. Please add an appropriate milestone.

All release blocking issues should have milestones associated to it, so that the issue can tracked and the fixes can be pushed promptly.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
That would be issue 800391, I assume.
Actually, it's not - looks like another new spike.  :(  I'll file a new bug.
Project Member

Comment 23 by sheriffbot@chromium.org, Mar 2 2018

This issue is marked as a release blocker with no milestone associated. Please add an appropriate milestone.

All release blocking issues should have milestones associated to it, so that the issue can tracked and the fixes can be pushed promptly.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 24 by mek@chromium.org, Mar 12 2018

Labels: -ReleaseBlock-Dev
Unmarking as releaseblock-dev, since if I understand things correctly the related spikes are actually for different (i.e. non blob:) urls that are already tracked in different bugs.
Project Member

Comment 25 by sheriffbot@chromium.org, Mar 15 2018

Labels: FoundIn-66
Users experienced this crash on the following builds:

Ios Beta 66.0.3359.12 -  8.50 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
Project Member

Comment 26 by sheriffbot@chromium.org, Mar 18 2018

Labels: FoundIn-67
Users experienced this crash on the following builds:

Mac Canary 67.0.3373.0 -  0.41 CPM, 1 reports, 1 clients (signature net::URLRequestContext::AssertNoURLRequests)
Ios Beta 66.0.3359.12 -  12.80 CPM, 2 reports, 2 clients (signature net::URLRequestContext::AssertNoURLRequests)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas
could AssertNoURLRequests() be changed not to crash and just log at least with the release build?

in my case, the AssertNoURLRequests() crash happens during the app shutdown procedure according to the crash backtrace (btw, we have no on-demand repro steps).

CHECK() ==> immediate crash
DCHECK() ==> debug check? and no crash with the release build?

153     // We're leaking URLRequests :( Dump the URL of the first one and record how
154     // many we leaked so we have an idea of how bad it is.
...
162     CHECK(false) << "Leaked " << num_requests << " URLRequest(s). First URL: "
163                  << request->url().spec().c_str() << ".";
No, it can't - when you destroy the URLRequest that outlived the URLRequestContext, it will try to call into the destroyed URLRequestContext, which will crash.  AssertNoURLRequests prevents the UAF crash with a more predictable and understandable crash stack.

Sign in to add a comment