>This has regressed from the last canary and has shown 11 reports from 9 clients till now on the latest canary:53.0.2754.0.

>All the crash instances are seen on 10.9 (Mavericks) & 10.10 (Yosemite) but none on 10.11 yet.

Considering below as the change log:

https://chromium.googlesource.com/chromium/src/+log/53.0.2753.0..53.0.2754.0?pretty=fuller&n=10000

Suspected change: https://codereview.chromium.org/2025923002

nektar@: Could you please take a look and confirm if this is the related change.

Thank you!

Users experienced this crash on the following builds:

Mac Canary 53.0.2754.0 -  3.82 CPM, 11 reports, 9 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Canary 53.0.2754.0 -  3.30 CPM, 21 reports, 17 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Beta 52.0.2743.24 -  4.15 CPM, 7 reports, 6 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Beta 52.0.2743.24 -  5.25 CPM, 11 reports, 10 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Beta 52.0.2743.24 -  5.25 CPM, 11 reports, 10 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Don't think this is a recent regression.This crash started in M51- 51.0.2690.0 canary and is spiking in M52 and M53. Below is the channel wise impact.

Regression Range
================
https://chromium.googlesource.com/chromium/src/+log/51.0.2685.0..51.0.2690.0?pretty=fuller&n=10000

Possible suspect
================
https://chromium.googlesource.com/chromium/src/+/d4372178c9652339528e915d83139d50cdd1de24

May be this crash is the after effect of the above CL, but not sure though.
The timeframe in which the crash appeared also matches with the CL.
I am looping to David for clarification.

Canary
======

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BCocoa%20Zombie%5D%20-%5BNativeWidgetMacNSWindow%20_bindingAdaptor%5D%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

Dev
===
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BCocoa%20Zombie%5D%20-%5BNativeWidgetMacNSWindow%20_bindingAdaptor%5D%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27dev%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

Beta
====

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BCocoa%20Zombie%5D%20-%5BNativeWidgetMacNSWindow%20_bindingAdaptor%5D%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27beta%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

None of the code in the "Possible suspect" patch is even compiled on Mac so we can easily eliminate that as a suspect.

Great, thanks for the confirmation.

This is the #1 browser crash for mac in current Beta - 52.0.2743.24. I need to get this fixed by EOD tomorrow 06/06. Requesting Stability Sheriff for a followup.

This looks like a new views crash. Unfortunately, the zombie_dealloc_bt isn't enlightening.

WARNING: crash2: received RPC error DATA_NOT_FOUND, details: Couldn't find symbol data for module Foundation|E0B0FAF65CA83EEB8BF2104C0AEEF9250

0x00719c32 [Google Chrome Framework -	 algorithm:708] (anonymous namespace)::ZombieDealloc(objc_object*, objc_selector*)
0x00073240 [AppKit +	 0x73240] -[NSResponder dealloc]
0x002922b0 [AppKit +	 0x2922b0] -[NSWindow dealloc]
0x000435ea [AppKit +	 0x435ea] -[NSWindow release]
0x00003137 [Foundation +	 0x3137] 
0x0028f1ed [AppKit +	 0x28f1ed] __18-[NSWindow _close]_block_invoke
0x0028f100 [AppKit +	 0x28f100] -[NSWindow _close]
0x0011ce0c [CoreFoundation +	 0x11ce0c] __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__
0x0001082d [CoreFoundation +	 0x1082d] _CFXNotificationPost
0x00002dda [Foundation +	 0x2dda] 
0x0028f1ed [AppKit +	 0x28f1ed] __18-[NSWindow _close]_block_invoke
0x0028f100 [AppKit +	 0x28f100] -[NSWindow _close]
0x037cdb6b [Google Chrome Framework -	 bind_internal.h:186] base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (Browser::*)()>, void (Browser*), base::WeakPtr<Browser> >, base::internal::InvokeHelper<true, void, base::internal::RunnableAdapter<void (Browser::*)()> >, void ()>::Run(base::internal::BindStateBase*)
0x00584f6b [Google Chrome Framework -	 callback.h:397] base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&)
0x005a76ac [Google Chrome Framework -	 vector:640] base::MessageLoop::RunTask(base::PendingTask const&)
0x005a79bc [Google Chrome Framework -	 message_loop.cc:484] base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&)
0x005a7d1b [Google Chrome Framework -	 message_loop.cc:601] base::MessageLoop::DoWork()
0x0057a2ad [Google Chrome Framework -	 message_pump_mac.mm:330] base::MessagePumpCFRunLoopBase::RunWork()

Users experienced this crash on the following builds:

Mac Beta 52.0.2743.24 -  7.68 CPM, 129 reports, 113 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Latest Beta - 52.0.2743.24 reports 131	instances so far.

Please have a fix ASAP before the next Beta cut at 4.00 pm PST Tuesday 06/07.

I think the `NativeWidgetMacNSWindow _bindingAdaptor` bit is a red herring - I don't see any other hints that MacViews is involved. No switches. And this actually looks like it's coming from the browser window.

The common pattern in the stack seems to be an `[NSWindow _close]` call triggered from base::Bind on a Browser*

base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (Browser::*)()>, void (Browser*), base::WeakPtr<Browser> >, base::internal::InvokeHelper<true, void, base::internal::RunnableAdapter<void (Browser::*)()> >, void ()>::Run(base::internal::BindStateBase*)

I'm pretty sure these are being triggered from this code:

void Browser::TabStripEmpty() {
  // Close the frame after we return to the message loop (not immediately,
  // otherwise it will destroy this object before the stack has a chance to
  // cleanly unwind.)
  // Note: This will be called several times if TabStripEmpty is called several
  //       times. This is because it does not close the window if tabs are
  //       still present.
  base::ThreadTaskRunnerHandle::Get()->PostTask(
      FROM_HERE, base::Bind(&Browser::CloseFrame, weak_factory_.GetWeakPtr()));

  // Instant may have visible WebContents that need to be detached before the
  // window system closes.
  instant_controller_.reset();
}

(and TabStripEmpty is only called from browser/ui/fast_unload_controller.cc)

There's not a lot of activity around there - maybe r394883
author	changwan <changwan@chromium.org>	Thu May 19 22:03:54 2016
Stop using nested message loop for alert() and other JS dialogs

That landed in 52.0.2742.0, so it could be responsible for the big spike in crashes we see after the trickle leading up to 52.0.2739.0


BrowserWindowCocoa only has a weak pointer to the BrowserWindowController*.. -[BrowserWindowController dealloc] should by nerfing the WeakPtr, but maybe the actual _window_ is dealloced before BrowserWindowController completes its dealloc. But then why would [bwc window] return non-nil :/.

NativeWidgetMac actually has a much clearer lifetime - when AppKit calls onWindowWillClose everything is destroyed. But in Cocoa things hang around.

So.. I only have a speculative fix - which is to set a flag in BrowserWindowCocoa when the controller receives - (void)windowWillClose:(NSNotification*)notification, and never return non-nil from BrowserWindowCocoa::window() in such cases.

So re: #c1 I can find this crash on all OSes, including 10.11 by tweaking the search params. Perhaps <= 10.10 just has a bogus symbol that's pointing the magic signature at  _bindingAdapator instead of BrowserWindowCocoa::Close

I think this is the same crash as, e.g.,

https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27BrowserWindowCocoa%3A%3AClose%27%20OMIT%20RECORD%20IF%20SUM(CrashedStackTrace.StackFrame.FunctionName%3D%27base%3A%3Ainternal%3A%3AInvoker%3Cbase%3A%3AIndexSequence%3C0ul%3E%2C%20base%3A%3Ainternal%3A%3ABindState%3Cbase%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%2C%20void%20(Browser*)%2C%20base%3A%3AWeakPtr%3CBrowser%3E%20%3E%2C%20base%3A%3Ainternal%3A%3AInvokeHelper%3Ctrue%2C%20void%2C%20base%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%20%3E%2C%20void%20()%3E%3A%3ARun(base%3A%3Ainternal%3A%3ABindStateBase*)%27)%20%3D%200&ignore_case=false&enable_rewrite=true&omit_field_name=CrashedStackTrace.StackFrame.FunctionName&omit_field_value=base%3A%3Ainternal%3A%3AInvoker%3Cbase%3A%3AIndexSequence%3C0ul%3E%2C%20base%3A%3Ainternal%3A%3ABindState%3Cbase%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%2C%20void%20(Browser*)%2C%20base%3A%3AWeakPtr%3CBrowser%3E%20%3E%2C%20base%3A%3Ainternal%3A%3AInvokeHelper%3Ctrue%2C%20void%2C%20base%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%20%3E%2C%20void%20()%3E%3A%3ARun(base%3A%3Ainternal%3A%3ABindStateBase*)&omit_field_opt=%3D

This has 558 reports versus the 346 being attributed to_bindingAdaptor

Oops - left off a filter. Try

https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27BrowserWindowCocoa%3A%3AClose%27%20OMIT%20RECORD%20IF%20SUM(CrashedStackTrace.StackFrame.FunctionName%3D%27base%3A%3Ainternal%3A%3AInvoker%3Cbase%3A%3AIndexSequence%3C0ul%3E%2C%20base%3A%3Ainternal%3A%3ABindState%3Cbase%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%2C%20void%20(Browser*)%2C%20base%3A%3AWeakPtr%3CBrowser%3E%20%3E%2C%20base%3A%3Ainternal%3A%3AInvokeHelper%3Ctrue%2C%20void%2C%20base%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%20%3E%2C%20void%20()%3E%3A%3ARun(base%3A%3Ainternal%3A%3ABindStateBase*)%27)%20%3D%200&ignore_case=false&enable_rewrite=true&omit_field_name=CrashedStackTrace.StackFrame.FunctionName&omit_field_value=base%3A%3Ainternal%3A%3AInvoker%3Cbase%3A%3AIndexSequence%3C0ul%3E%2C%20base%3A%3Ainternal%3A%3ABindState%3Cbase%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%2C%20void%20(Browser*)%2C%20base%3A%3AWeakPtr%3CBrowser%3E%20%3E%2C%20base%3A%3Ainternal%3A%3AInvokeHelper%3Ctrue%2C%20void%2C%20base%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%20%3E%2C%20void%20()%3E%3A%3ARun(base%3A%3Ainternal%3A%3ABindStateBase*)&omit_field_opt=%3D

That has the 37 crashes on OSX 10.11 crashes, even though there's no OS filter. It just tickles a codepath inside AppKit, so gets a different magic signature.

Suggested fix - https://codereview.chromium.org/2041213002/

Re #15, as found in comment #17, this was happening on 50.0.2661.102, so I guess my CL only changed the stack traces.

Robert, could you please review the fix in comment #18 first thing in the morning. Once it lands I need to kick a canary to verify the patch & merge to M52 branch by 4.00 pm PST tomorrow ( 06/07 - Tuesday)

Re: #15, you're right that this is coming off TabStripEmpty. I dumped the stack from https://crash.corp.google.com/browse?q=reportid=%2724fbc65a00000000%27#0 and did a manual FROM_HERE recovery:

80c09658ff7f0000
bc298b0701000000  ; base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&)
60884043dd7f0000
e0c09658ff7f0000
10c19658ff7f0000
1b2d8b0701000000  ; base::MessageLoop::DoWork()
90cb4502
00610000
01480401ff7f0000
0000000000000000
b096650100610000
008bad0a01000000  ; base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (Browser::*)()>, void (B
3089840c01000000  000000010c848930 --> 0x553d930 *** FROM_HERE ***
3e89840c01000000
8d040000          0x048d = 1165 (line_number)
80610000
f650ad0a01000000  ; Browser::TabStripEmpty()
afc51d0001000000
1037650080610000
206f0c0000600000
0000000000000000
90c29658ff7f0000
1037650080610000
88c0150000600000
40c19658ff7f0000
ad52880701000000  ; base::MessagePumpCFRunLoopBase::RunWork()
7052880701000000  ; ___ZN4base24MessagePumpCFRunLoopBase13RunWorkSourceEPv_block_invoke
c0dd170000600000
d8dd170000600000
88c0150000600000
50c19658ff7f0000
ea878a0701000000  ; base::mac::CallWithEHFrame(void () block_pointer)
90c19658ff7f0000

rsesek@hotwire:/Users/rsesek % hexdump -C -n 50 -s 0x553d930 ./Downloads/chrome-mac/Google\ Chrome.app/Contents/Versions/53.0.2754.0/Google\ Chrome\ Framework.framework/Google\ Chrome\ Framework
0553d930  54 61 62 53 74 72 69 70  45 6d 70 74 79 00 2e 2e  |TabStripEmpty...|
0553d940  2f 2e 2e 2f 63 68 72 6f  6d 65 2f 62 72 6f 77 73  |/../chrome/brows|
0553d950  65 72 2f 75 69 2f 62 72  6f 77 73 65 72 2e 63 63  |er/ui/browser.cc|
0553d960  00 34                                             |.4|
0553d962


> BrowserWindowCocoa only has a weak pointer to the BrowserWindowController*.. -[BrowserWindowController dealloc] should by nerfing the WeakPtr, but maybe the actual _window_ is dealloced before BrowserWindowController completes its dealloc. But then why would [bwc window] return non-nil :/.

The BrowserWindowController should own the underlying window, so I don't think that can happen. When we come through BrowserWindowCocoa::Close(), we use the re-implemented version of AppKit`-[NSWindow performClose:] that doesn't spin a nested run loop. It's maybe possible that closing the window through the GUI is spinning that loop, pumping close tasks, and releasing the window with code expecting to call back into the window lower on the stack. Unfortunately for use, the zombie_dealloc_bt isn't long enough to show that.

What if we autoreleased the window and set it to nil in -[BrowserWindowController windowWillClose:]. I think that's the point of no return for us, so that should be safe.

so I don't lose it, crash url to monitor - https://crash.corp.google.com/browse?q=OMIT%20RECORD%20IF%20SUM(CrashedStackTrace.StackFrame.FunctionName%3D%27base%3A%3Ainternal%3A%3AInvoker%3Cbase%3A%3AIndexSequence%3C0ul%3E%2C%20base%3A%3Ainternal%3A%3ABindState%3Cbase%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%2C%20void%20(Browser*)%2C%20base%3A%3AWeakPtr%3CBrowser%3E%20%3E%2C%20base%3A%3Ainternal%3A%3AInvokeHelper%3Ctrue%2C%20void%2C%20base%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%20%3E%2C%20void%20()%3E%3A%3ARun(base%3A%3Ainternal%3A%3ABindStateBase*)%27)%20%3D%200&ignore_case=false&enable_rewrite=true&omit_field_name=CrashedStackTrace.StackFrame.FunctionName&omit_field_value=base%3A%3Ainternal%3A%3AInvoker%3Cbase%3A%3AIndexSequence%3C0ul%3E%2C%20base%3A%3Ainternal%3A%3ABindState%3Cbase%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%2C%20void%20(Browser*)%2C%20base%3A%3AWeakPtr%3CBrowser%3E%20%3E%2C%20base%3A%3Ainternal%3A%3AInvokeHelper%3Ctrue%2C%20void%2C%20base%3A%3Ainternal%3A%3ARunnableAdapter%3Cvoid%20(Browser%3A%3A*)()%3E%20%3E%2C%20void%20()%3E%3A%3ARun(base%3A%3Ainternal%3A%3ABindStateBase*)&omit_field_opt=%3D#samplereports:5,productversion:1000,magicsignature

(but it could be a subset of those - the magic signature is wonky and changes depending on OS version, so it's searching on a stack frame)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7e91932ca0df94f1d3ebe9303e6ff31637751c17

commit 7e91932ca0df94f1d3ebe9303e6ff31637751c17
Author: tapted <tapted@chromium.org>
Date: Wed Jun 08 03:54:34 2016

Address a crash under -[NSWindow close] via a WeakPtr PostTask from Browser::TabStripEmpty()

This is the #1 browser crash for Mac in current Beta - 52.0.2743.24. The
stacks all have in common a Posted Task that's triggering -[NSWindow
close] via a base::WeakPtr<Browser>. This only happens via
Browser::TabStripEmpty().

The WeakPtr is only nerfed when BrowserWindowController's dealloc fully
completes. It seems plausible that this can be after the NSWindow's
dealloc fully completes, leading to an invalid access. One would hope
that [NSWindowController window] returns nil once the controlled window
is destroyed, but this seems to not be guaranteed.

To (speculatively) fix, set a flag when the controlled window invokes
-[NSWindowController windowWillClose]. The window shouldn't be accessed
after this. Ensure the C++ BrowserWindowCocoa shim returns nil in this
case for the window, even if -[NSWindowController window] doesn't.

BUG= 616701 

Review-Url: https://codereview.chromium.org/2041213002
Cr-Commit-Position: refs/heads/master@{#398474}

[modify] https://crrev.com/7e91932ca0df94f1d3ebe9303e6ff31637751c17/chrome/browser/ui/cocoa/browser_window_cocoa.h
[modify] https://crrev.com/7e91932ca0df94f1d3ebe9303e6ff31637751c17/chrome/browser/ui/cocoa/browser_window_cocoa.mm
[modify] https://crrev.com/7e91932ca0df94f1d3ebe9303e6ff31637751c17/chrome/browser/ui/cocoa/browser_window_controller.mm

So, there actually hasn't been a crash on canary since 53.0.2754.0 which is a few days old. But it's likely we are still crashing in beta channel. Repeating a comment at https://codereview.chromium.org/2041213002#msg21 :

<quote>
On 2016/06/08 02:47:27, tapted wrote:
> On 2016/06/08 02:00:32, Robert Sesek wrote:
> > I'm not totally convinced this will fix the issue, but let's go with it as a
> > speculative fix. LGTM.
> 
> updated CL description with "speculative" :)
> 
> > Do we have enough samples on canary/dev to verify that this works before
> needing
> > to merge it to beta?
> 
> I've added a go/crash search link to the bug that I'll monitor. There seems to
> be a few crashes in each Canary going back pretty far. But there's a big
uptick
> (60!) in the most recent Canary -- 53.0.2754.0. Spread across a whole bunch of
> Client IDs too. I'll see if I can extract any new insights with this nugget of
> information.

So 53.0.2754.0 is not actually the most recent Canary, but it is the last one
that we have crashes for :(.
And that's actually a few days old. So there's the possibility that this no
longer crashes in Canary, but I'm stumped why that would be.

Could this fix beta.... maybe? I'm kinda out of ideas at this point.

I've scanned both

https://chromium.googlesource.com/chromium/src/+log/53.0.2754.0..53.0.2756.0?...
and
https://chromium.googlesource.com/chromium/src/+log/53.0.2753.0..53.0.2754.0?...

but couldn't find anything. Note there doesn't seem to be a 53.0.2755.? tag.

Let's get this in so it can bake and leave our options open for the beta. But if
there's no blip in the crash reporter on m53, we should probably revert it in
Canary to get more data.

</quote>

This crash has high impact on Chrome's stability.
Channel: beta. Platform: mac.
Labeling  issue 616701  with Pri-0.
Labeling  issue 616701  with ReleaseBlock-Stable.


If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

New hypothesis! I think this was caused by https://chromium.googlesource.com/chromium/src/+/d8fdd17d87b906d6a0e21bd6366dface55132440, which is in the range of 53.0.2753.0 - 53.0.2754.0 when this spiked.

Looking at this crash: https://crash.corp.google.com/browse?q=reportid=%275610313a00000000%27

Thread 0 CRASHED [EXC_BAD_INSTRUCTION / 0x00000001 @ 0x0000000108f52b9c ] MAGIC SIGNATURE THREAD
0x0000000108f52b9c	(Google Chrome Framework -objc_zombie.mm:235 )	(anonymous namespace)::ZombieObjectCrash(objc_object*, objc_selector*, objc_selector*)
0x0000000108f529c0	(Google Chrome Framework -objc_zombie.mm:270 )	-[CrZombie forwardingTargetForSelector:]
0x00007fff93a9a72b	(CoreFoundation + 0x000ac72b )	___forwarding___
0x00007fff93a9a617	(CoreFoundation + 0x000ac617 )	__forwarding_prep_0___
0x00007fff8d2dca7d	(AppKit + 0x0023ba7d )	__18-[NSWindow _close]_block_invoke
0x00007fff8d2dc984	(AppKit + 0x0023b984 )	-[NSWindow _close]
0x00007fff93b0ccdb	(CoreFoundation + 0x0011ecdb )	__CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__
0x00007fff939fe243	(CoreFoundation + 0x00010243 )	_CFXNotificationPost
0x00007fff8aeecc30	(Foundation + 0x00002c30 )	-[NSNotificationCenter postNotificationName:object:userInfo:]
0x00007fff8d2dca70	(AppKit + 0x0023ba70 )	__18-[NSWindow _close]_block_invoke
0x00007fff8d2dc984	(AppKit + 0x0023b984 )	-[NSWindow _close]
0x000000010c034670	(Google Chrome Framework -bind_internal.h:187 )	base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (Browser::*)()>, void (Browser*), base::WeakPtr<Browser> >, true, void ()>::Run(base::internal::BindStateBase*)
0x0000000108db99fa	(Google Chrome Framework -callback.h:397 )	base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&)
0x0000000108ddc13b	(Google Chrome Framework -message_loop.cc:475 )	base::MessageLoop::RunTask(base::PendingTask const&)
0x0000000108ddc44b	(Google Chrome Framework -message_loop.cc:484 )	base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&)
0x0000000108ddc7aa	(Google Chrome Framework -message_loop.cc:601 )	base::MessageLoop::DoWork()
0x0000000108daecec	(Google Chrome Framework -message_pump_mac.mm:330 )	base::MessagePumpCFRunLoopBase::RunWork()
0x0000000108dd2259	(Google Chrome Framework + 0x0059e259 )	base::mac::CallWithEHFrame(void () block_pointer)
0x0000000108dae6f3	(Google Chrome Framework -message_pump_mac.mm:306 )	base::MessagePumpCFRunLoopBase::RunWorkSource(void*)
0x00007fff93a6e680	(CoreFoundation + 0x00080680 )	__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
0x00007fff93a6080c	(CoreFoundation + 0x0007280c )	__CFRunLoopDoSources0
0x00007fff93a5fe3e	(CoreFoundation + 0x00071e3e )	__CFRunLoopRun
0x00007fff93a5f857	(CoreFoundation + 0x00071857 )	CFRunLoopRunSpecific
0x00007fff91557aee	(HIToolbox + 0x0002eaee )	RunCurrentEventLoopInMode
0x00007fff91557869	(HIToolbox + 0x0002e869 )	ReceiveNextEventCommon
0x00007fff915576aa	(HIToolbox + 0x0002e6aa )	_BlockUntilNextEventMatchingListInModeWithFilter
0x00007fff8d0c4f80	(AppKit + 0x00023f80 )	_DPSNextEvent
0x00007fff8d0c472f	(AppKit + 0x0002372f )	-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]
0x00007fff8d0b8592	(AppKit + 0x00017592 )	-[NSApplication run]
0x0000000108daf515	(Google Chrome Framework -message_pump_mac.mm:665 )	base::MessagePumpNSApplication::DoRun(base::MessagePump::Delegate*)
0x0000000108daeb43	(Google Chrome Framework -message_pump_mac.mm:238 )	base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*)
0x0000000108df4030	(Google Chrome Framework -run_loop.cc:35 )	base::RunLoop::Run()
0x00000001088d52b4	(Google Chrome Framework -chrome_browser_main.cc:1895 )	ChromeBrowserMainParts::MainMessageLoopRun(int*)
0x000000010c2d2c93	(Google Chrome Framework -browser_main_loop.cc:958 )	content::BrowserMainLoop::RunMainMessageLoopParts()
0x000000010c2d5121	(Google Chrome Framework -browser_main_runner.cc:154 )	content::BrowserMainRunnerImpl::Run()
0x000000010c2cf24c	(Google Chrome Framework -browser_main.cc:46 )	content::BrowserMain(content::MainFunctionParams const&)
0x0000000108d6a5ff	(Google Chrome Framework -content_main_runner.cc:787 )	content::ContentMainRunnerImpl::Run()
0x0000000108d69845	(Google Chrome Framework -content_main.cc:20 )	content::ContentMain(content::ContentMainParams const&)
0x0000000108836b69	(Google Chrome Framework -chrome_main.cc:84 )	ChromeMain
0x00000001087bbd41	(Google Chrome Canary -chrome_exe_main_mac.c:87 )	main
0x00000001087bbb23	(Google Chrome Canary + 0x00000b23 )	start

And its zombie_dealloc_bt:
0x0071f132 [Google Chrome Framework -	 algorithm:708] (anonymous namespace)::ZombieDealloc(objc_object*, objc_selector*)
0x0006ab8c [AppKit +	 0x6ab8c] -[NSResponder dealloc]
0x0023ea43 [AppKit +	 0x23ea43] -[NSWindow dealloc]
0x00042857 [AppKit +	 0x42857] -[NSWindow release]
0x00003173 [Foundation +	 0x3173] -[NSConcreteNotification dealloc]
0x0002189c [libobjc.A.dylib +	 0x2189c] objc_object::sidetable_release(bool)
0x0023ba71 [AppKit +	 0x23ba71] __18-[NSWindow _close]_block_invoke
0x0023b985 [AppKit +	 0x23b985] -[NSWindow _close]
0x0011ecdc [CoreFoundation +	 0x11ecdc] __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__
0x00010244 [CoreFoundation +	 0x10244] _CFXNotificationPost
0x00002c31 [Foundation +	 0x2c31] -[NSNotificationCenter postNotificationName:object:userInfo:]
0x0023ba71 [AppKit +	 0x23ba71] __18-[NSWindow _close]_block_invoke
0x0023b985 [AppKit +	 0x23b985] -[NSWindow _close]
0x03800671 [Google Chrome Framework -	 bind_internal.h:187] base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (Browser::*)()>, void (Browser*), base::WeakPtr<Browser> >, true, void ()>::Run(base::internal::BindStateBase*)
0x005859fb [Google Chrome Framework -	 callback.h:397] base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&)
0x005a813c [Google Chrome Framework -	 vector:640] base::MessageLoop::RunTask(base::PendingTask const&)
0x005a844c [Google Chrome Framework -	 message_loop.cc:484] base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&)
0x005a87ab [Google Chrome Framework -	 message_loop.cc:601] base::MessageLoop::DoWork()

In AppKit -[NSWindow close] just thunks to -[NSWindow _close], which basically just invokes the __18__NSWindow__close__block_invoke. Relevant disassembly of that function:

__text:00000000002B1B63 loc_2B1B63:                             ; CODE XREF: ___18__NSWindow__close__block_invoke+395j
__text:00000000002B1B63                 mov     rdi, cs:classRef_NSNotificationCenter
__text:00000000002B1B6A                 mov     rsi, cs:selRef_defaultCenter
__text:00000000002B1B71                 mov     rbx, cs:_objc_msgSend_ptr
__text:00000000002B1B78                 call    rbx ; _objc_msgSend
__text:00000000002B1B7A                 lea     rcx, _NSWindowWillCloseNotification
__text:00000000002B1B81                 mov     rdx, [rcx]
__text:00000000002B1B84                 mov     rcx, [r13+20h]
__text:00000000002B1B88                 mov     rsi, cs:selRef_postNotificationName_object_
__text:00000000002B1B8F                 mov     rdi, rax
__text:00000000002B1B92                 call    rbx ; _objc_msgSend
__text:00000000002B1B94                 mov     rdi, [r13+20h]
__text:00000000002B1B98                 mov     rsi, cs:selRef__bindingAdaptor
__text:00000000002B1B9F                 call    rbx ; _objc_msgSend
__text:00000000002B1BA1                 mov     rdx, [r13+20h]
__text:00000000002B1BA5                 mov     rsi, cs:selRef_window_didChangeToVisibleState_
__text:00000000002B1BAC                 xor     ecx, ecx
__text:00000000002B1BAE                 mov     rdi, rax
__text:00000000002B1BB1                 call    rbx ; _objc_msgSend
__text:00000000002B1BB3                 mov     rax, [r13+20h]
__text:00000000002B1BB7                 mov     r12, cs:_OBJC_IVAR_$_NSWindow__wFlags ; struct __wFlags _wFlags;
__text:00000000002B1BBE                 mov     rcx, [rax+r12]
__text:00000000002B1BC2                 test    cl, 40h
__text:00000000002B1BC5                 jz      short loc_2B1BE2
__text:00000000002B1BC7                 and     rcx, 0FFFFFFFFFFEFFFFFh
__text:00000000002B1BCE                 mov     [rax+r12], rcx
__text:00000000002B1BD2                 mov     rcx, cs:_OBJC_IVAR_$_NSWindow__auxiliaryStorage ; NSWindowAuxiliary *_auxiliaryStorage;
__text:00000000002B1BD9                 mov     rax, cs:_OBJC_IVAR_$_NSWindowAuxiliary__auxWFlags ; struct __auxWFlags _auxWFlags;
__text:00000000002B1BE0                 jmp     short loc_2B1C21

Cocoa first posts NSWindowWillCloseNotification and then calls _bindingAdaptor. The second stack shows the following: the last reference to the window being released is actually owned by the NSNotification (for NSWindowWillCloseNotification notification), which means that whatever was owning the window originally has already gone away. My hypothesis is that the owning object deletes itself in response to -windowWillClose:, and when the NSNotification is deallocated (per zombie_dealloc_bt) the window turns to zombie. But __18__NSWindow__close__block_invoke is not done with the window reference, and when it sends a _bindingAdapter message to it at __text:00000000002B1B9F we get the zombie crash. If that's the case, then I think the window closing at 0x00007fff8d2dc984 is distinct from the window closing at 0x00007fff8d2dc984 in stack 1.

It looks like NativeWidgetMac::OnWindowWillClose() does something remarkably similar to what I described above...

Would like to take a fix for M52, Beta- so tagging accordingly.

Thanks to Robert's awesome detective work, I think we've homed in on the root cause. The real fix is in https://codereview.chromium.org/2056593002/ . It includes a unit test that I'm fairly certain reproduces the situation that we're seeing here. The test fails with a crash on 10.9 without the fix.

I have a repro case too. It only happens on OSX 10.9 (10.10: maybe, 10.11: no).

Steps:

1. On OSX 10.9, have one browser tab
2. Navigate to a page (e.g. chrome://version)
3. Press Backspace. The "Press Cmd+Left to go back" bubble should appear.
4. Wait for it to fade out completely.
5. Then close the tab via the tabstrip. Chrome shouldn't crash.

I've verified the crash is gone with the fix in https://codereview.chromium.org/2056593002/


Of course step 4. means that the "Press Cmd+Left to go back" bubble isn't getting destroyed at the end of the animation, which isn't nice and should be fixed separately since it wastes resources. It's not leaked exactly -- the bubble gets reused, and is still owned by the window, but there's no point keeping it around.

> I've verified the crash is gone with the fix in https://codereview.chromium.org/2056593002/

\o/

> bubble isn't getting destroyed at the end of the animation, which isn't nice

As far as I can tell, this is exactly the same as for fullscreen. (Albeit, when you *exit* fullscreen mode, it will delete the bubble as the type == EXCLUSIVE_ACCESS_BUBBLE_TYPE_NONE; but after the animation ends nothing happens other than it being hidden.) I don't think this is a big problem.

-RV-EI since this can be public.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6777bcc56aff3a1dc27fb0adced9a462b22e4e33

commit 6777bcc56aff3a1dc27fb0adced9a462b22e4e33
Author: tapted <tapted@chromium.org>
Date: Thu Jun 09 15:25:38 2016

Mac: Retain the child NSWindow in WidgetOwnerNSWindowAdapter before invoking close

AppKit has gotten better at handling the last reference to an NSWindow
going away inside -[NSWindow close]. However, on 10.9, bad things can
happen if we don't retain the NSWindow being closed until after the
-[NSWindow close] call returns.

WidgetOwnerNSWindowAdapter::OnWindowWillClose() wasn't doing this. The
result: in some codepaths, a zombie access under the close call.

To fix, retain the window being closed.

BUG= 616701 
TEST=On OSX 10.9, have one browser tab, navigate to a page (e.g.
chrome://version), then press Backspace. The "Press Cmd+Left to go back"
bubble should appear. Wait for it to fade out completely. Then close the
tab via the tabstrip. Chrome shouldn't crash.

Review-Url: https://codereview.chromium.org/2056593002
Cr-Commit-Position: refs/heads/master@{#398890}

[modify] https://crrev.com/6777bcc56aff3a1dc27fb0adced9a462b22e4e33/ui/views/cocoa/native_widget_mac_nswindow.mm
[modify] https://crrev.com/6777bcc56aff3a1dc27fb0adced9a462b22e4e33/ui/views/cocoa/widget_owner_nswindow_adapter.mm
[modify] https://crrev.com/6777bcc56aff3a1dc27fb0adced9a462b22e4e33/ui/views/widget/native_widget_mac_unittest.mm

Users experienced this crash on the following builds:

Mac Dev 53.0.2756.0 -  4.46 CPM, 57 reports, 49 clients (signature [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

tapted@, can we have the CL shown in #32 merged in to M52 branch ?

No, it should bake on canary first.

This crash has high impact on Chrome's stability.
Signature: [Cocoa Zombie] -[NativeWidgetMacNSWindow _bindingAdaptor].
Channel: dev. Platform: mac.
Labeling  issue 616701  with ReleaseBlock-Dev.


If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Verified on the 52.0.2764.0 Canary with the repro steps in #c29 on OSX 10.9. No longer crashes (does crash on the previous canary - 2763).

Requesting merge of the patch in #c32 (r398890) to m52 beta.

Your change meets the bar and is auto-approved for M52 (branch: 2743)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f74417df0eb4a034004fa587c8835cf3ee69edac

commit f74417df0eb4a034004fa587c8835cf3ee69edac
Author: Trent Apted <tapted@chromium.org>
Date: Fri Jun 10 11:55:31 2016

[merge-m52] Mac: Retain the child NSWindow in WidgetOwnerNSWindowAdapter before invoking close

AppKit has gotten better at handling the last reference to an NSWindow
going away inside -[NSWindow close]. However, on 10.9, bad things can
happen if we don't retain the NSWindow being closed until after the
-[NSWindow close] call returns.

WidgetOwnerNSWindowAdapter::OnWindowWillClose() wasn't doing this. The
result: in some codepaths, a zombie access under the close call.

To fix, retain the window being closed.

BUG= 616701 
TEST=On OSX 10.9, have one browser tab, navigate to a page (e.g.
chrome://version), then press Backspace. The "Press Cmd+Left to go back"
bubble should appear. Wait for it to fade out completely. Then close the
tab via the tabstrip. Chrome shouldn't crash.

Review-Url: https://codereview.chromium.org/2056593002
Cr-Commit-Position: refs/heads/master@{#398890}
(cherry picked from commit 6777bcc56aff3a1dc27fb0adced9a462b22e4e33)

Review URL: https://codereview.chromium.org/2056273002 .

Cr-Commit-Position: refs/branch-heads/2743@{#313}
Cr-Branched-From: 2b3ae3b8090361f8af5a611712fc1a5ab2de53cb-refs/heads/master@{#394939}

[modify] https://crrev.com/f74417df0eb4a034004fa587c8835cf3ee69edac/ui/views/cocoa/native_widget_mac_nswindow.mm
[modify] https://crrev.com/f74417df0eb4a034004fa587c8835cf3ee69edac/ui/views/cocoa/widget_owner_nswindow_adapter.mm
[modify] https://crrev.com/f74417df0eb4a034004fa587c8835cf3ee69edac/ui/views/widget/native_widget_mac_unittest.mm

Removing from the stability sheriff queue.

Should be fixed. The earlier attempt in #c23 didn't do anything - reverted that in r399164.

No new crashes since 53.0.2763.0 in Canary. But we need a new beta spin before they dry up there.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6777bcc56aff3a1dc27fb0adced9a462b22e4e33

commit 6777bcc56aff3a1dc27fb0adced9a462b22e4e33
Author: tapted <tapted@chromium.org>
Date: Thu Jun 09 15:25:38 2016

Mac: Retain the child NSWindow in WidgetOwnerNSWindowAdapter before invoking close

AppKit has gotten better at handling the last reference to an NSWindow
going away inside -[NSWindow close]. However, on 10.9, bad things can
happen if we don't retain the NSWindow being closed until after the
-[NSWindow close] call returns.

WidgetOwnerNSWindowAdapter::OnWindowWillClose() wasn't doing this. The
result: in some codepaths, a zombie access under the close call.

To fix, retain the window being closed.

BUG= 616701 
TEST=On OSX 10.9, have one browser tab, navigate to a page (e.g.
chrome://version), then press Backspace. The "Press Cmd+Left to go back"
bubble should appear. Wait for it to fade out completely. Then close the
tab via the tabstrip. Chrome shouldn't crash.

Review-Url: https://codereview.chromium.org/2056593002
Cr-Commit-Position: refs/heads/master@{#398890}

[modify] https://crrev.com/6777bcc56aff3a1dc27fb0adced9a462b22e4e33/ui/views/cocoa/native_widget_mac_nswindow.mm
[modify] https://crrev.com/6777bcc56aff3a1dc27fb0adced9a462b22e4e33/ui/views/cocoa/widget_owner_nswindow_adapter.mm
[modify] https://crrev.com/6777bcc56aff3a1dc27fb0adced9a462b22e4e33/ui/views/widget/native_widget_mac_unittest.mm

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f74417df0eb4a034004fa587c8835cf3ee69edac

commit f74417df0eb4a034004fa587c8835cf3ee69edac
Author: Trent Apted <tapted@chromium.org>
Date: Fri Jun 10 11:55:31 2016

[merge-m52] Mac: Retain the child NSWindow in WidgetOwnerNSWindowAdapter before invoking close

AppKit has gotten better at handling the last reference to an NSWindow
going away inside -[NSWindow close]. However, on 10.9, bad things can
happen if we don't retain the NSWindow being closed until after the
-[NSWindow close] call returns.

WidgetOwnerNSWindowAdapter::OnWindowWillClose() wasn't doing this. The
result: in some codepaths, a zombie access under the close call.

To fix, retain the window being closed.

BUG= 616701 
TEST=On OSX 10.9, have one browser tab, navigate to a page (e.g.
chrome://version), then press Backspace. The "Press Cmd+Left to go back"
bubble should appear. Wait for it to fade out completely. Then close the
tab via the tabstrip. Chrome shouldn't crash.

Review-Url: https://codereview.chromium.org/2056593002
Cr-Commit-Position: refs/heads/master@{#398890}
(cherry picked from commit 6777bcc56aff3a1dc27fb0adced9a462b22e4e33)

Review URL: https://codereview.chromium.org/2056273002 .

Cr-Commit-Position: refs/branch-heads/2743@{#313}
Cr-Branched-From: 2b3ae3b8090361f8af5a611712fc1a5ab2de53cb-refs/heads/master@{#394939}

[modify] https://crrev.com/f74417df0eb4a034004fa587c8835cf3ee69edac/ui/views/cocoa/native_widget_mac_nswindow.mm
[modify] https://crrev.com/f74417df0eb4a034004fa587c8835cf3ee69edac/ui/views/cocoa/widget_owner_nswindow_adapter.mm
[modify] https://crrev.com/f74417df0eb4a034004fa587c8835cf3ee69edac/ui/views/widget/native_widget_mac_unittest.mm

Verified that the fix merged in to M52 branch [2743] is working fine using the steps provided in # 29.

OSX --> 10.9.5
Chrome --> 52.0.2743.41