New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 154483 link

Starred by 2 users

Issue metadata

Status: Fixed
Closed: Nov 2012
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug

Sign in to add a comment

Mac zombie crash handling key equivalent under [BrowserCrApplication sendEvent:]

Reported by, Oct 7 2012

Issue description

Product: Chrome_Mac
Stack Signature: objc_msgSend-CE387D
New Signature Label: objc_msgSend
New Signature Hash: 87ef465a_e68487b1_587d4abe_0b19625a_05b67034

Report link: http://go/crash/reportdetail?reportid=cb36de3f3ebabfd3

Meta information:
Product Name: Chrome_Mac
Product Version: 24.0.1288.1
Report ID: cb36de3f3ebabfd3
Report Time: 2012/10/07 01:07:41, Sun
Uptime: 514 sec
Cumulative Uptime: 0 sec
OS Name: Mac OS X
OS Version: 10.8.2 12C54
CPU Architecture: x86
CPU Info: GenuineIntel family 6 model 58 stepping 9
ptype: browser


0x96b40a84	 [libobjc.A.dylib]	 + 0x00005a84]	objc_msgSend
0x9859e0fd	 [AppKit]	 + 0x0038a0fd]	-[NSView _performKeyEquivalent:conditionally:]
0x9859e046	 [AppKit]	 + 0x0038a046]	-[NSWindow performKeyEquivalent:]
0x9859ddd6	 [AppKit]	 + 0x00389dd6]	-[NSApplication _handleKeyEquivalent:]
0x98453ee0	 [AppKit]	 + 0x0023fee0]	-[NSApplication sendEvent:]
0x002bb9ad	 [Google Chrome Framework]	 -]	-[BrowserCrApplication sendEvent:]
0x9836d72b	 [AppKit]	 + 0x0015972b]	-[NSApplication run]
0x00b2d5e0	 [Google Chrome Framework]	 -]	base::MessagePumpNSApplication::DoRun
0x00b2d11b	 [Google Chrome Framework]	 -]	base::MessagePumpCFRunLoopBase::Run
0x00b5935f	 [Google Chrome Framework]	 -]	MessageLoop::RunHandler
0x00b6b400	 [Google Chrome Framework]	 -]	base::RunLoop::Run
0x002c2592	 [Google Chrome Framework]	 -]	ChromeBrowserMainParts::MainMessageLoopRun
0x0291355f	 [Google Chrome Framework]	 -]	content::BrowserMainLoop::RunMainMessageLoopParts
0x02913c82	 [Google Chrome Framework]	 -]	(anonymous namespace)::BrowserMainRunnerImpl::Run
0x029125e0	 [Google Chrome Framework]	 -]	BrowserMain
0x00ac02ea	 [Google Chrome Framework]	 -]	content::ContentMainRunnerImpl::Run
0x00abf5ff	 [Google Chrome Framework]	 -]	content::ContentMain
0x00020048	 [Google Chrome Framework]	 -]	ChromeMain
0x0001af77	 [Google Chrome Canary]	 -]	main
0x0001af54	 [Google Chrome Canary]	 + 0x00000f54]	start

Comment 1 by, Oct 8 2012

Status: Assigned
Scott, could you take a look at this?  Robert thinks it looks like a partially constructed zombie.  Doesn't look new.

Comment 2 by, Oct 9 2012

The zombie code fills objects with '!', which is 0x21.  This kind of backtrace implies that either the message was send between the memcpy() and replacing isa (which would be a race condition, as it means some other thread is doing the dealloc), or the more likely case, someone is doing a direct de-reference from within an object which was deallocated.  The most likely case for _that_ is that the view on the second stack frame was deallocated while in the process of accomplishing something (which is why it can see the memory in the first place).

I have a prototype somewhere which fills the instance area with valid isa references to a class which can search out the right ref and dump the intended message and stuff.  I think it will be of mixed utility, as it would make it harder to see non-msgSend cases, but, well, so be it.

Comment 3 by, Oct 15 2012

I was looking at my prototype.  Due to space issues, it is limited to only dumping the message sent, not including the class info.  Unless the message is really unique, it's not going to yield much.

The memset() was mostly to make sure that stale refs didn't continue processing valid data.  It could be removed, and if the sub-objects are being deleted, the backtrace will move to that zombie, plus the dealloc backtrace.  Of course, it's possible that the referenced object isn't really dead, in which case the crash will stop happening but the code will still be incorrect.

It's possible to do things like decode the types of things in the object, but that's not really feasible to be running in a production environment.  Another option would be an additional treadmill to track the previous values of things, which the zombie code could indirect through.  That treadmill would be cycled really fast, so it's not really clear how much utility it would have, either.

Comment 4 by, Oct 15 2012

Sorry - no way to associate through the additional treadmill, so non-starter.

Could also decode the calling |self| off the stack.  Which would be aWeSoMe.

Comment 5 by, Oct 15 2012

Hmm.  Tracing a run of Chrome, looks to me like the flow of control for controls is something like:

-[NSView _performKeyEquivalent:conditionally:] calls -isEnabled which is forwarded to the cell.  Then it forwards to the cell.  AFAICT, it's doing [[self cell] _perform...], so the framework shouldn't be having the problem.

I also find myself looking at the event-hook junk just before the call to forward -sendEvent: to the parent.  AFAICT the only remaining hook is in, and it makes a call to -closeFolderAndStopTrackingMenus, which seems fun.  -isEventAnExitEvent: makes me cry.  I think it would be reasonable to add a couple crash keys in here to provide some hints.
Project Member

Comment 6 by, Oct 16 2012

The following revision refers to this bug:

r162054 | | 2012-10-16T02:50:08.508073Z

Changed paths:

[Mac] Additional crash logging for -sendEvent: crash.

-[NSEvent description] is what gdb's po command would print against
the event.  It is basically an ASCII serialization of the event.

BUG= 154483 

Review URL:
Project Member

Comment 7 by, Oct 17 2012

The following revision refers to this bug:

r162397 | | 2012-10-17T14:17:10.868708Z

Changed paths:

[Mac] Tweak crash logging for -sendEvent: crash. adds the event's
-description as a breakpad key.  Unfortunately, event type 28, which
is undocumented, is sent by Mountain Lion when you do a control-scroll
zoom.  Work around it manually.

BUG= 154483 , 156103 

Review URL:

Comment 8 by, Oct 17 2012


sendevent	 NSEvent: type=KeyDown loc=(0,900) time=2306.6 flags=0x9c0129 win=0x0 winNum=65 ctxt=0xe603 chars=" " unmodchars=" " repeat=0 keyCode=53

I don't even?  flags is Control, Alternate, Command, Function?  Maybe?

Comment 9 by, Oct 18 2012

Sorry, keyCode = 53 is ESC?  When I log events, Control-Alternate-Command-ESC seems to give similar output.

I see only one such crash thus far, so I'm inclined to leave it in to see if it's a trend or a singleton.

Comment 10 by, Oct 18 2012

Three more crashes.  All show:
  sendevent	 NSEvent: type=KeyDown loc=(0,800) time=32685.0 flags=0x100108 win=0x0 winNum=2938 ctxt=0x1483f chars="q" unmodchars="q" repeat=0 keyCode=12
well, times and window info differs, of course.  So that's Command-q.  I'm inclined to imagine the C-A-C-ESC case might be someone remapping the Quit accelerator.

Comment 11 by, Oct 30 2012

Lots of occurrences, all those cases.  I'm going to back out the diagnostic code, it's not adding anything.
Project Member

Comment 12 by, Oct 31 2012

The following revision refers to this bug:

r165077 | | 2012-10-31T00:46:21.873378Z

Changed paths:

Revert [Mac] Crash logging for -sendEvent: crash.

Enough data has been sent up to see that the crash is all Command-Q
(and a variant).

BUG= 154483 , 158171 

Review URL:
Project Member

Comment 13 by, Nov 10 2012

The following revision refers to this bug:

r167021 | | 2012-11-10T00:45:06.756078Z

Changed paths:

[Mac] Temporarily stop clearing zombie objects.

Zombie objects are cleared to '!' to force faster failure.  The bug
involves a containing object being zombied, and then trying to message
something it owns.  This change will provide info on the object being
messaged, and perhaps info on who deallocated it.

BUG= 154483 

Review URL:
Project Member

Comment 14 by, Nov 10 2012

Labels: merge-merged-1312
The following revision refers to this bug:

r167092 | | 2012-11-10T16:43:01.165704Z

Changed paths:

Merge 165077 - Revert [Mac] Crash logging for -sendEvent: crash.

Enough data has been sent up to see that the crash is all Command-Q
(and a variant).

BUG= 154483 , 158171 

Review URL:
Review URL:

Comment 15 by, Nov 10 2012

Over on  issue 158171 , Dharani noted that that crash was like 9% of browser crashes, which seems really high for what should essentially be an impossible crash.  I suspect we have a quit-time bug again, looking for evidence I see  issue 127335 , or  issue 107881 .   Issue 115929  had a couple bits of more-recent activity.

Comment 16 by, Nov 12 2012

25.0.1321.0 had three cases of this crash.  25.0.1322.0 was the first version with the zombie-related change above, and it has not happened since then.  Which is unrealistic.  I think this implies that the messaged object has not been deallocated, though the forwarding object is.  Perhaps it's in the autorelease pool, or is multiply-owned or something.

*befuddled*.  I have code to fill zombies with an object reference which would at least allow seeing what message is being sent.  But that can probably be determined using other means, as there can't be too many possibilities in the parent call's implementation.

We do some interesting interception around command-key events in renderer host views.  My understanding is that we pass it down to the renderer to let it handle things, and it if doesn't, it's passed back up and potentially re-injected.  I wonder if there are any races, like if the renderer sends things back up for re-injection immediately before closing, and the window went away before the re-injection was handled, or perhaps as a nested loop under the re-injection.  If users are attempting to Quit, that might explain why there might be things closing.

Comment 17 by, Nov 16 2012

Status: Started
Previously I was searching using -[NSView _performKeyEquivalent:conditionally:], but today I noticed this crash, which seems likely (it would explain why this crash started happening when it did, and it involves a quirky window-swizzling system to explain how the zombie came to be).

Thread 0 *CRASHED* ( EXC_BAD_INSTRUCTION / EXC_I386_INVOP @ 0x000cd872 )

0x000cd872	 [Google Chrome Framework]	 -]	(anonymous namespace)::ZombieObjectCrash
0x000cd6c7	 [Google Chrome Framework]	 -]	-[CrZombie forwardingTargetForSelector:]
0x92703237	 [CoreFoundation]	 + 0x00088237]	__NSGetForwardingTarget
0x927031bf	 [CoreFoundation]	 + 0x000881bf]	__forwarding_prep_0___
0x98a3b08f	 [AppKit]	 + 0x0038a08f]	-[NSWindow performKeyEquivalent:]
0x98a3add6	 [AppKit]	 + 0x00389dd6]	-[NSApplication _handleKeyEquivalent:]
0x988f0ee0	 [AppKit]	 + 0x0023fee0]	-[NSApplication sendEvent:]
0x0029acfd	 [Google Chrome Framework]	 -]	-[BrowserCrApplication sendEvent:]
0x9880a72b	 [AppKit]	 + 0x0015972b]	-[NSApplication run]

Comment 18 by, Nov 16 2012

   Zombie <PepperFlashFullscreenWindow: 0x7c5b1f70> received -_unlockViewHierarchyForDrawing
not all that helpful, but probably indicates that we've torn down an window or object while executing code for it.  In a debug build, I just repro'ed by going to fullscreen on a youtube video and hitting Command-Option-Control-Function-ESC (see an early comment).  The backtrace matched, the dealloc backtrace made me think of zombie_dealloc_bt, which yields:

0xc4d11 (anonymous namespace)::ZombieDealloc (
0x158bb1 -[NSResponder dealloc] (AppKit)
0x2641a1 -[NSWindow dealloc] (AppKit)
0x129643 -[NSWindow release] (AppKit)
0x4fcf25 content::RenderWidgetHostViewMac::Destroy (scoped_nsobject.h:70)
0x4ea560 content::RenderWidgetHostImpl::OnMessageReceived (
0x4c1563 content::RenderProcessHostImpl::ProcessDied (
0x4c138c content::RenderProcessHostImpl::FastShutdownIfPossible (
0x24372a browser_shutdown::OnShutdownStarting (
0x2bbb860 Browser::OnWindowClosing (browser.h:244)
0x2bf7a9e -[BrowserWindowController windowShouldClose:] (
0x2bf4d8e BrowserWindowCocoa::Close (
0x4ca050 browser::CloseAllBrowsers (stl_iterator.h:675)
0xe0f93 -[AppController tryToTerminateApplication:] (
0x2917f0 -[BrowserCrApplication terminate:] (
0x1e5d3 -[NSObject performSelector:withObject:] (libobjc.A.dylib)
0x24fbd2 -[NSApplication sendAction:to:from:] (AppKit)
0x291ae5 -[BrowserCrApplication sendAction:to:from:] (
0x38c3dc -[NSMenuItem _corePerformAction] (AppKit)
0x38c06b -[NSCarbonMenuImpl performActionWithHighlightingForItemAtIndex:] (AppKit)

that doesn't provide enough detail, to my mind.  Fortunately, my debug build does show where the stacks join, so I'm sure it's the window processing the event being deallocated while processing:

Backtrace from -dealloc:
	0   Chromium Framework                  0x04064088 (anonymous namespace)::ZombieDealloc(objc_object*, objc_selector*) + 680
	1   AppKit                              0x90719bb1 -[NSResponder dealloc] + 156
	2   AppKit                              0x908251a1 -[NSWindow dealloc] + 2564
	3   AppKit                              0x906ea643 -[NSWindow release] + 170
	4   Chromium Framework                  0x01b322df scoped_nsprotocol<NSWindow*>::reset(NSWindow*, base::scoped_policy::OwnershipPolicy) + 127
	5   Chromium Framework                  0x04b4a8e8 content::RenderWidgetHostViewMac::Destroy() + 984
	6   Chromium Framework                  0x04b1b4ca content::RenderWidgetHostImpl::Destroy() + 202
	7   Chromium Framework                  0x04b1b1c2 content::RenderWidgetHostImpl::Shutdown() + 402
	8   Chromium Framework                  0x04b51f29 -[RenderWidgetHostViewCocoa keyEvent:wasKeyEquivalent:] + 1129
	9   Chromium Framework                  0x04b51932 -[RenderWidgetHostViewCocoa performKeyEquivalent:] + 482
	10  AppKit                              0x9094b0fe -[NSView _performKeyEquivalent:conditionally:] + 49
	11  AppKit                              0x9094b267 -[NSView performKeyEquivalent:] + 212
	12  AppKit                              0x9094b0fe -[NSView _performKeyEquivalent:conditionally:] + 49
	13  AppKit                              0x9094b267 -[NSView performKeyEquivalent:] + 212
	14  AppKit                              0x9094b0fe -[NSView _performKeyEquivalent:conditionally:] + 49
	15  AppKit                              0x9094b047 -[NSWindow performKeyEquivalent:] + 125
	16  AppKit                              0x9094add7 -[NSApplication _handleKeyEquivalent:] + 526
	17  AppKit                              0x90800ee1 -[NSApplication sendEvent:] + 5512
	18  Chromium Framework                  0x04ff1f1b -[BrowserCrApplication sendEvent:] + 587
	19  AppKit                              0x9071a72c -[NSApplication run] + 951
[76001:-1406645720:1116/] Zombie <PepperFlashFullscreenWindow: 0x7e9105c0> received -_unlockViewHierarchyForDrawing
	0   Chromium Framework                  0x033483cf base::debug::StackTrace::StackTrace() + 63
	1   Chromium Framework                  0x0334836b base::debug::StackTrace::StackTrace() + 43
	2   Chromium Framework                  0x033971bc logging::LogMessage::~LogMessage() + 76
	3   Chromium Framework                  0x03395c9b logging::LogMessage::~LogMessage() + 43
	4   Chromium Framework                  0x0406320d (anonymous namespace)::ZombieObjectCrash(objc_object*, objc_selector*, objc_selector) + 1197
	5   Chromium Framework                  0x04062d30 -[CrZombie forwardingTargetForSelector:] + 80
	6   CoreFoundation                      0x93616238 __NSGetForwardingTarget + 72
	7   CoreFoundation                      0x936161c0 _CF_forwarding_prep_0 + 16
	8   AppKit                              0x9094b090 -[NSWindow performKeyEquivalent:] + 198
	9   AppKit                              0x9094add7 -[NSApplication _handleKeyEquivalent:] + 526
	10  AppKit                              0x90800ee1 -[NSApplication sendEvent:] + 5512
	11  Chromium Framework                  0x04ff1f1b -[BrowserCrApplication sendEvent:] + 587
	12  AppKit                              0x9071a72c -[NSApplication run] + 951
	13  Chromium Framework                  0x0331d5fe base::MessagePumpNSApplication::DoRun(base::MessagePump::Delegate*) + 350
	14  Chromium Framework                  0x0331c438 base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*) + 104
	15  Chromium Framework                  0x033b1572 MessageLoop::RunInternal() + 290
	16  Chromium Framework                  0x033b142b MessageLoop::RunHandler() + 43
	17  Chromium Framework                  0x0340ac08 base::RunLoop::Run() + 72
	18  Chromium Framework                  0x04ffdaa9 ChromeBrowserMainParts::MainMessageLoopRun(int*) + 505
	19  Chromium Framework                  0x046f5a97 content::BrowserMainLoop::RunMainMessageLoopParts() + 167
	20  Chromium Framework                  0x046f9b1f content::BrowserMainRunnerImpl::Run() + 479
	21  Chromium Framework                  0x046f37b6 content::BrowserMain(content::MainFunctionParams const&) + 198
	22  Chromium Framework                  0x06f0cbb4 content::RunNamedProcessTypeMain(std::string const&, content::MainFunctionParams const&, content::ContentMainDelegate*) + 212
	23  Chromium Framework                  0x06f0e038 content::ContentMainRunnerImpl::Run() + 680
	24  Chromium Framework                  0x06f0c027 content::ContentMain(int, char const**, content::ContentMainDelegate*) + 167
	25  Chromium Framework                  0x000e3c0b ChromeMain + 75
	26  Chromium                            0x000dcf7b main + 43
	27  Chromium                            0x000dcf45 start + 53

Project Member

Comment 19 by, Nov 17 2012

The following revision refers to this bug:

r168463 | | 2012-11-17T22:13:36.075893Z

Changed paths:

[Mac] Prevent ppapi fullscreen window from dealloc while on stack.

If the rwhvm is being destroyed in response to a command accelerator,
the window being deallocated may still be executing code higher on the

BUG= 154483 

Review URL:

Comment 20 by, Nov 17 2012

Status: Fixed
Dharani: Yes, but that other crash is squarely in Apple code and I doubt there's much to about it. For magic signatures with -[BrowserCrApplication sendEvent:], the basic/original stack signature is a good indication. Magic signature does not work well with third party/system code.

Comment 23 by, Nov 28 2012

Labels: -merge-merged-1312 Merge-Requested
Summary: Mac zombie crash handling key equivalent under [BrowserCrApplication sendEvent:]
Yeah, that's different.  -sendEvent: is a core framework method, so lots of crashes on the main thread will be under it.  I'll update the subject a little to be more specific.

Also, nominating r168463 for merge to M-24.  The dev channel hasn't rolled out since it was checked in, but at this point there's been enough time on canary that I'm confident this fixed it.  Might even want to push this all the way to M-23, as it accounts for a fair number of browser crashes per day on stable.

Comment 24 by, Nov 28 2012

Labels: -Merge-Requested Merge-Approved
Project Member

Comment 25 by, Nov 28 2012

Labels: -Merge-Approved merge-merged-1312
The following revision refers to this bug:

r170034 | | 2012-11-28T20:04:39.778591Z

Changed paths:

Merge 168463 - [Mac] Prevent ppapi fullscreen window from dealloc while on stack.

If the rwhvm is being destroyed in response to a command accelerator,
the window being deallocated may still be executing code higher on the

BUG= 154483 

Review URL:
Review URL:

Comment 26 by, Nov 28 2012

Labels: Merge-Requested Mstone-23

Comment 27 by, Nov 28 2012

Dharani, do you know when beta will next spin?  Or dev?  I just spoke with Karen about getting this to M-23, and I'm pretty sure this fixed the crash, but there's always the chance that changing the point of release causes some other problem.

Comment 28 by, Nov 28 2012

Dev is happening today and we have one more beta tomorrow.

Comment 29 by, Nov 28 2012

Labels: -Mstone-24
QA team requires exact repro steps to verify this fix.

Comment #18 had a repro case.  I'm not sure whether this really needs QA verification, even if QA signs off, it's still only fixed if the crash server reports stop coming.
I'm reverting r167021 (comment 13) because it's no longer necessary.
Project Member

Comment 33 by, Dec 3 2012

The following revision refers to this bug:

r170825 | | 2012-12-03T22:15:53.293170Z

Changed paths:

Revert 167021, info lead to bug-fix.
> [Mac] Temporarily stop clearing zombie objects.
> Zombie objects are cleared to '!' to force faster failure.  The bug
> involves a containing object being zombied, and then trying to message
> something it owns.  This change will provide info on the object being
> messaged, and perhaps info on who deallocated it.
> BUG= 154483 
> Review URL:
Review URL:

Comment 34 by, Dec 5 2012

Labels: -Merge-Requested Merge-Rejected
not taking for m23.
Project Member

Comment 35 by, Mar 10 2013

Labels: -Mstone-23 M-23

Sign in to add a comment