New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 130772 link

Starred by 9 users

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Aug 2015
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 123830



Sign in to add a comment

crashing while fetching pac script (system url request context) during shutdown

Project Member Reported by rtenneti@chromium.org, Jun 1 2012

Issue description

Product: Chrome
Stack Signature: base::debug::BreakDebugger()-18B549E
New Signature Label: base::debug::BreakDebugger()
New Signature Hash: 709aa75d_83ac9070_8d0c0e87_4744031f_4f3a1fe3

Report link: http://go/crash/reportdetail?reportid=6f0fc2047867de29

Meta information:
Product Name: Chrome
Product Version: 21.0.1155.2
Report ID: 6f0fc2047867de29
Report Time: 2012/05/31 00:45:37, Thu
Uptime: 18 sec
Cumulative Uptime: 0 sec
OS Name: Windows NT
OS Version: 5.1.2600 Service Pack 3
CPU Architecture: x86
CPU Info: GenuineIntel family 6 model 23 stepping 10

The following are couple of URLs that were pointed out by 	net::URLRequestContext::AssertNoURLRequests.

"http://wpad/wpad.dat"
"http://colegiobobs.com/felicidade/secret.pac"
 
This seems to have started in 1144: http://crash.corp.google.com/search?query=product%3A%22Chrome%22+version%3A%2221.0.1144.0%22+AssertNoURLRequests, but not in http://crash.corp.google.com/search?query=product%3A%22Chrome%22+version%3A%2221.0.1143.0%22+AssertNoURLRequests (these crashes are in ProfileIOData shutdown, so it's a different URLRequestContext, not the PAC context).
Owner: mad@chromium.org
Status: Assigned
If we leak Profiles, then the PAC fetcher context may have ongoing URLRequests on behalf of those Profiles. We need to not leak Profiles.
Cc: lafo...@chromium.org dharani@chromium.org
In Beta (M20), it is one of top 50 crashes.
	net::URLRequestContext::AssertNoURLRequests

In Dev (M21), it is among the top 10 crashes
	net::URLRequestContext::AssertNoURLRequests
Labels: Mstone-21 Stability-Crash ReleaseBlock-Beta
mad - should the change mentioned in comment #2 be reverted?

Comment 6 by mad@chromium.org, Jun 21 2012

This is not a profile leak it's a render process host leak... if the
profile goes away before the render process host, we can get crashes... we
need to fix the render process host leak...

BYE
MAD...
sent from Mobile Answering Device!
Le 6 juin 2012 18:01, <willchan@chromium.org> a 

Comment 7 by mad@chromium.org, Jun 21 2012

As an experiment, I will change the destruction delay to a DCHECK, so that we can catch to render process leak / late delays, and then see if the number of crashes we get from render process hosts holding on to a dead profile or not quite as bad as this one...
Labels: -ReleaseBlock-Beta ReleaseBlock-Stable
Yes, the root cause is the RenderProcessHost leak. We need to fix that. But r138038 causes a RenderProcessHost leak to *also* cause a profile leak. IIUC, r138038 should get reverted, since we're just shuffling around where the crash happens, not actually fixing a crash.

Comment 10 by mad@chromium.org, Jun 21 2012

There are still cases where the fix actually resolve crashes where the render process host didn't leak, it just got released later than the Profile because all this is independently asynchronous... But I guess these should be fixed in some other ways too... :-(
Getting released later than the Profile is a leak in this case, because RenderProcessHost is a UI thread object. The Profile gets deleted after the MessageLoop on the UI thread has stopped pumping events. That means that by the time the ProfileManager starts deleting Profiles, if the RenderProcessHost isn't gone, then it will be leaked, since the MessageLoop will not run any more tasks, so it won't just get deleted later, but it will be completely leaked.

Comment 12 by mad@chromium.org, Jun 21 2012

I don't think this is the case for incognito profile... Which is why this initially only took care of incognito profiles... Which eventually caused problems when incognito profile outlived parent profile...
Yes, sorry for being imprecise. My statement is only true for normal profiles. But as you point out, due to the dependency between incognito profiles and normal profiles, we need to either make profiles live longer (and everything that depends on those profiles live longer), or we need to make sure that everything that uses profiles gets destroyed promptly.

There are a few points to keep in mind here. Even if we decide to let Profiles live longer, it's not acceptable to ever leak them, since then we aren't persisting lots of their data on shutdown, which is clearly a bug. So, we we want to keep Profiles alive longer, and not leak them, then we need to keep pumping the UI MessageLoop. Of course, during shutdown, we've never done this, so re-pumping the UI MessageLoop would probably lead to a lot of bugs. Also, allowing things to keep objects alive longer is an inversion of ownership. Profiles own all these profile-related objects like RenderProcessHost, so those objects should not keep Profiles alive.

As I think we've agreed on, the real solution is to track down this RenderProcessHost delayed shutdown issue and figure out how to make it get destroyed promptly.

Comment 14 by kareng@google.com, Jun 22 2012

Labels: -Pri-2 Pri-1
Project Member

Comment 15 by bugdroid1@chromium.org, Jun 26 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=144251

------------------------------------------------------------------------
r144251 | mad@chromium.org | Tue Jun 26 13:09:16 PDT 2012

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/profiles/profile_impl.cc?r1=144251&r2=144250&pathrev=144251
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/profiles/profile_destroyer.cc?r1=144251&r2=144250&pathrev=144251
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/profiles/profile_destroyer.h?r1=144251&r2=144250&pathrev=144251

Now doesn't wait to destroy non-incognito profiles, and allow immediate destruction of incognito profiles from their parent profile. Also, doesn't wait infinitely for render process hosts to go away, uses a timer to complete destruction. And finally, adds DCHECKs to identify when render process hosts are dead soon enough.

BUG= 130772 
TEST=Make sure Chrome exits without a crash and that Profile info properly got saved. Also make sure info doesn't leak from one incognito session to another.

Review URL: https://chromiumcodereview.appspot.com/10645005
------------------------------------------------------------------------

Comment 16 by mad@chromium.org, Jun 26 2012

Labels: Merge-Requested

Comment 17 by kareng@google.com, Jun 26 2012

let me know once this has made to canary and has been verified. pls. :)

Comment 18 by mad@chromium.org, Jun 26 2012

OK, will do... :-)

Comment 19 by mad@chromium.org, Jun 29 2012

Seems like the crash is still happening with the fix... :-(
http://crash.corp.google.com/search?query=product%3A%22Chrome%22+version%3A%2222.0.1188.0%22+crashed_thread_function_name%3A%22net%3A%3AURLRequestContext%3A%3AAssertNoURLRequests%28%29%22

There are a few occurrences in 1188 which was cut after the fix got it (but none yet in 1189, not sure why though, too soon?)...

So either the fix isn't correct, or the problem wasn't caused by the profile destroyer... :-/

We need to go deeper...

Comment 20 by kareng@google.com, Jun 29 2012

Labels: -Merge-Requested
ok so removing merge request for now :)

Comment 21 by mad@chromium.org, Jun 29 2012

Labels: Merge-Requested
Actually, I take that back, the issues we are seeing after the fix I made were not related, they were in the destructor of the ProfileIOData, so not related as mentioned by willchan in comment 1 above...

There were no occurrence of the non-ProfileIOData destructor crash after this fix, so I guess we are good to go with the merge...

Unless I'm missing something again...

Comment 22 by kareng@google.com, Jun 29 2012

hmm. we have a bit more time for beta2, would u like to stew this a bit? u can merge all the way to july 10 and still be in beta2 at this moment.

Comment 23 by mad@chromium.org, Jun 29 2012

I personally don't mind as lo g as the other stake holders are OK with it...

BYE
MAD...
sent from Mobile Answering Device!
Le 29 juin 2012 13:27, <kareng@google.com> a 

Comment 24 by kareng@google.com, Jun 29 2012

there are no builds between now and then anyway so it's not possible to get it out sooner, might as well wait and make sure there are no side effects.
Blocking: chromium:123830

Comment 26 by kareng@google.com, Jul 9 2012

Labels: -Merge-Requested Merge-Approved

Comment 27 by mad@chromium.org, Jul 9 2012

Status: Fixed
Project Member

Comment 28 by bugdroid1@chromium.org, Jul 9 2012

Labels: -Merge-Approved merge-merged-1180
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=145740

------------------------------------------------------------------------
r145740 | mad@chromium.org | Mon Jul 09 14:14:22 PDT 2012

Changed paths:
 M http://src.chromium.org/viewvc/chrome/branches/1180/src/chrome/browser/profiles/profile_destroyer.h?r1=145740&r2=145739&pathrev=145740
 M http://src.chromium.org/viewvc/chrome/branches/1180/src/chrome/browser/profiles/profile_destroyer.cc?r1=145740&r2=145739&pathrev=145740
 M http://src.chromium.org/viewvc/chrome/branches/1180/src/chrome/browser/profiles/profile_impl.cc?r1=145740&r2=145739&pathrev=145740

Merge 144251 - Now doesn't wait to destroy non-incognito profiles, and allow immediate destruction of incognito profiles from their parent profile. Also, doesn't wait infinitely for render process hosts to go away, uses a timer to complete destruction. And finally, adds DCHECKs to identify when render process hosts are dead soon enough.

BUG= 130772 
TEST=Make sure Chrome exits without a crash and that Profile info properly got saved. Also make sure info doesn't leak from one incognito session to another.

Review URL: https://chromiumcodereview.appspot.com/10645005

TBR=mad@chromium.org
Review URL: https://chromiumcodereview.appspot.com/10763009
------------------------------------------------------------------------

Comment 29 Deleted

Comment 30 Deleted

We had 4 crashes in 22.0.1200.0	in net::URLRequestContext::AssertNoURLRequests.

The following are couple of URLs.
http://wpad/wpad.dat
http://lh6.ggpht.com/FZnwriel5m_myAAt4UGSzR-THOQlqyrHfPyO5FrGV7BaH0gdtW0wN5TsGVS_0290y9u_vWUbv_M

crash id: 6c802a8f5d028365 crashing while deleting SystemURLRequestContext (or non-ProfileIOData).

The following is one call stack:

Thread 13 *CRASHED* ( EXCEPTION_BREAKPOINT @ 0x62d68a75 )

0x62d68a75	 [chrome.dll]	 - debugger_win.cc:107	
base::debug::BreakDebugger()
0x62d31f82	 [chrome.dll]	 - url_request_context.cc:92	
net::URLRequestContext::AssertNoURLRequests()
0x62d31fce	 [chrome.dll]	 - url_request_context.cc:39	
net::URLRequestContext::~URLRequestContext()
0x62d3850c	 [chrome.dll]	 + 0x0050850c]	
`anonymous namespace'::SystemURLRequestContext::`scalar deleting destructor'(unsigned int)
0x62d38459	 [chrome.dll]	 - io_thread.cc:309	
IOThread::Globals::~Globals()
0x62d382ef	 [chrome.dll]	 - io_thread.cc:505	
IOThread::CleanUp()
0x629ab5a6	 [chrome.dll]	 - browser_process_sub_thread.cc:54	
content::BrowserProcessSubThread::CleanUp()

The following is another call stack (crash during ProfileIOData).
Thread 12 *CRASHED* ( EXCEPTION_BREAKPOINT @ 0x6bf17e81 )

0x6bf17e81	 [chrome.dll]	 - debugger_win.cc:107	
base::debug::BreakDebugger()
0x6bee3a0d	 [chrome.dll]	 - url_request_context.cc:92	
net::URLRequestContext::AssertNoURLRequests()
0x6bee815c	 [chrome.dll]	 - profile_io_data.cc:283	
ProfileIOData::~ProfileIOData()
0x6bee5928	 [chrome.dll]	 + 0x00505928]	
ProfileImplIOData::`vector deleting destructor'(unsigned int)

Crash ids in 22.0.1200.0 are:
  0c56dc8ee88df0b8
  6c802a8f5d028365
  1bccfcaf931b802d
  e75b79e2c3487273

Comment 32 by mad@chromium.org, Jul 9 2012

Labels: -ReleaseBlock-Stable
Owner: ----
Status: Available
Oh! Yeah, right, this bug is the more generic one... I was taking this one as just failing because the ProfileDestroyer was holding on Profile objects for too long...

But now that we have fixed the most frequent cause of this symptom, maybe we can remove the ReleaseBlock-Stable label?

And I also remove myself as a owner and make the bug available since I don't know this code that much, I was just owner to fix the ProfileDestroyer issue (which I also don't own, but I had volunteered to introduce to fix another problem many months ago).

OK?

Comment 33 by kareng@google.com, Aug 14 2012

Owner: willchan@chromium.org
Status: Assigned
http://codereview.chromium.org/10006009 looks like u added the CHECK will and i still see this crash on 23.0.1235.0

Comment 34 by kareng@google.com, Aug 14 2012

http://codereview.chromium.org/10006009 looks like u added the CHECK will and i still see this crash on 23.0.1235.0
Cc: rtenneti@chromium.org
Karen, how many crashes are we seeing? Raman, can update the bug with the latest URLs we see? I want to know if we're seeing new, different kinds of URLs due to regressions, or if it's still the same slow trickle due to profile destruction being all messed up.
There were only 3 crashes in 23.0.1235.0 until 8/14 5:04pm PST.
Looks like the profile is getting leaked. This should be a different bug,
it's not the PAC script fetcher. But yeah, until we fix the profile leakage
(which has some other implications), we will continue to have issues.

Comment 39 by kareng@google.com, Sep 2 2012

i see 13 in today's canary. is this not fixable at this point?
Cc: tturchetto@chromium.org krisr@chromium.org creis@chromium.org saintlou@chromium.org
Issue 144260 has been merged into this issue.

Comment 41 by dharani@google.com, Sep 23 2012

recently this is showing up on the top. Is there a recent regression causing the spike? https://crash.corp.google.com/reportdetail?reportid=52ba13429b54f655#crashing_thread
Cc: -saintlou@chromium.org
I'm seeing something that looks similar to this in R25 on Chrome OS.  Is this the same bug?

0x73f44109	 [chrome]	 - base/debug/debugger_posix.cc:245]	base::debug::BreakDebugger()
0x73fb3ad9	 [chrome]	 - net/url_request/url_request_context.cc:122]	net::URLRequestContext::AssertNoURLRequests() const
0x73cad9f1	 [chrome]	 - chrome/browser/profiles/profile_io_data.cc:311]	ProfileIOData::~ProfileIOData()
0x73e2dd19	 [chrome]	 - chrome/browser/profiles/off_the_record_profile_io_data.cc:154]	OffTheRecordProfileIOData::~OffTheRecordProfileIOData()
0x73e2dd2b	 [chrome]	 - chrome/browser/profiles/off_the_record_profile_io_data.cc:154]	OffTheRecordProfileIOData::~OffTheRecordProfileIOData()
0x73cacd69	 [chrome]	 - ./base/sequenced_task_runner_helpers.h:39]	base::DeleteHelper<ProfileIOData>::DoDelete(void const*)
0x75a36f87	 [chrome]	 - ./base/bind_internal.h:171]	base::internal::Invoker<1, base::internal::BindState<base::internal::RunnableAdapter<void (*)(void const*)>, void (void const*), void (void const*)>, void (void const*)>::Run(base::internal::BindStateBase*)
0x7585c4ff	 [chrome]	 - ./base/callback.h:391]	MessageLoop::RunTask(base::PendingTask const&)
0x7585c465	 [chrome]	 - base/message_loop.cc:485]	MessageLoop::DeferOrRunPendingTask(base::PendingTask const&)
0x75846fdb	 [chrome]	 - base/message_loop.cc:668]	MessageLoop::DoWork()
0x75856da1	 [chrome]	 - base/message_pump_libevent.cc:239]	base::MessagePumpLibevent::Run(base::MessagePump::Delegate*)
0x75846dbb	 [chrome]	 - base/message_loop.cc:430]	MessageLoop::RunInternal()
0x75846d4d	 [chrome]	 - base/run_loop.cc:45]	base::RunLoop::Run()
0x75846cab	 [chrome]	 - base/message_loop.cc:310]	MessageLoop::Run()
0x758a7033	 [chrome]	 - content/browser/browser_thread_impl.cc:149]	content::BrowserThreadImpl::IOThreadRun(MessageLoop*)
0x7589dac5	 [chrome]	 - content/browser/browser_thread_impl.cc:177]	content::BrowserThreadImpl::Run(MessageLoop*)
0x75846901	 [chrome]	 - base/threading/thread.cc:195]	base::Thread::ThreadMain()
0x758468a3	 [chrome]	 - base/threading/platform_thread_posix.cc:65]	base::::ThreadFunc
0x73601089	 [libpthread-2.15.so]	 - pthread_create.c:307]	start_thread

This is an incognito request leak. Can you repro in a debugger? Or somehow get the stack data from a crash dump? In URLRequestContext::AssertNoURLRequests(), we'll record the URL requested and the allocation stack trace, so we can debug the leak. More likely than not, it's a regression. All types of regressions will fall into this same stack trace, and we have to analyze the stack to determine the root cause. So I recommend you file a separate bug and cc me on it.
Project Member

Comment 45 by bugdroid1@chromium.org, Mar 10 2013

Labels: -Area-Internals -Internals-Network -Mstone-21 M-21 Cr-Internals Cr-Internals-Network

Comment 46 by laforge@google.com, May 22 2013

Labels: -M-21 M-28
Howdy folks, we are still seeing this crash in Chrome 28 (28.0.1500.20) and it represents about 1% of the browser crashes.

Would greatly appreciate some help putting this monkey to bed, since it looks like it's been poking us since the early 20s.

https://chromecrash.corp.google.com/samples?q=product.name%3D'Chrome'%20AND%20product.version%3D'28.0.1500.20'%20AND%20custom_data.ChromeCrashProto.ptype%3D'browser'%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D'net%3A%3AURLRequestContext%3A%3AAssertNoURLRequests'
Seeing something similar on shutdown in CrOS R29: https://crash.corp.google.com/reportdetail?reportid=a5bce59df8a087dc

Cc: jamescook@chromium.org abod...@chromium.org
 Issue 252538  has been merged into this issue.
Cc: -jamescook@chromium.org

Comment 50 by laforge@google.com, Aug 28 2015

Status: Archived

Sign in to add a comment