I see this in my system log too. mark@, can you take a peek? :)

* thread #29: tid = 0x1cda47, 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error, name = 'Service Discovery Thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error
    frame #1: 0x00007fffb7ae78de libobjc.A.dylib`weak_clear_no_lock + 140
    frame #2: 0x00007fffb7ae13c4 libobjc.A.dylib`objc_object::sidetable_clearDeallocating() + 168
    frame #3: 0x00007fffb7ae10b7 libobjc.A.dylib`objc_destructInstance + 143
    frame #4: 0x0000000103988109 Google Chrome Framework`(anonymous namespace)::ZombieDealloc(self=0x0000000115c4c820, _cmd=<unavailable>) + 89 at objc_zombie.mm:115 [opt]
    frame #5: 0x00007fffb7ae6c49 libobjc.A.dylib`objc_object::sidetable_release(bool) + 285
    frame #6: 0x0000000102f79327 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() [inlined] base::internal::ScopedNSProtocolTraits<objc_object*>::Release(nst=<unavailable>) + 55 at scoped_nsobject.h:75 [opt]
    frame #7: 0x0000000102f79322 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() [inlined] base::ScopedTypeRef<objc_object*, base::internal::ScopedNSProtocolTraits<objc_object*> >::~ScopedTypeRef() + 9 at scoped_typeref.h:84 [opt]
    frame #8: 0x0000000102f79319 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() [inlined] base::scoped_nsprotocol<objc_object*>::~scoped_nsprotocol() at scoped_nsobject.h:82 [opt]
    frame #9: 0x0000000102f79319 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() [inlined] base::scoped_nsobject<objc_object*>::~scoped_nsobject() at scoped_nsobject.h:196 [opt]
    frame #10: 0x0000000102f79319 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() [inlined] base::scoped_nsobject<objc_object*>::~scoped_nsobject() at scoped_nsobject.h:196 [opt]
    frame #11: 0x0000000102f79319 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer(this=0x0000000100216180) + 41 at service_discovery_client_mac.mm:189 [opt]
    frame #12: 0x0000000102f7bf66 Google Chrome Framework`base::DeleteHelper<local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer>::DoDelete(void const*) [inlined] local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer(this=0x0000000100216180) + 22 at service_discovery_client_mac.mm:187 [opt]
    frame #13: 0x0000000102f7bf5e Google Chrome Framework`base::DeleteHelper<local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer>::DoDelete(object=0x0000000100216180) + 14 at sequenced_task_runner_helpers.h:40 [opt]
    frame #14: 0x0000000102fe5f69 Google Chrome Framework`base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) [inlined] base::internal::RunMixin<base::Callback<void (), (base::internal::CopyMode)0, (base::internal::RepeatMode)0> >::Run() + 22 at callback.h:47 [opt]
    frame #15: 0x0000000102fe5f53 Google Chrome Framework`base::debug::TaskAnnotator::RunTask(this=<unavailable>, queue_function="MessageLoop::PostTask", pending_task=0x0000700019ddae08) + 195 at task_annotator.cc:52 [opt]
    frame #16: 0x0000000103009536 Google Chrome Framework`base::MessageLoop::RunTask(this=0x0000000100215e10, pending_task=0x0000700019ddae08) + 390 at message_loop.cc:413 [opt]
    frame #17: 0x00000001030097fc Google Chrome Framework`base::MessageLoop::DeferOrRunPendingTask(this=0x0000000100215e10, pending_task=PendingTask @ 0x0000700019ddae08) + 44 at message_loop.cc:422 [opt]
    frame #18: 0x0000000103009b43 Google Chrome Framework`base::MessageLoop::DoWork(this=0x0000000100215e10) + 371 at message_loop.cc:515 [opt]
    frame #19: 0x000000010300c1cd Google Chrome Framework`base::MessagePumpCFRunLoopBase::RunWork(this=0x0000000115c4d240) + 45 at message_pump_mac.mm:330 [opt]
    frame #20: 0x0000000102ffe8da Google Chrome Framework`base::mac::CallWithEHFrame(void () block_pointer) + 10
    frame #21: 0x000000010300bbe4 Google Chrome Framework`base::MessagePumpCFRunLoopBase::RunWorkSource(info=<unavailable>) + 68 at message_pump_mac.mm:306 [opt]
    frame #22: 0x00007fffa3316581 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    frame #23: 0x00007fffa32f798c CoreFoundation`__CFRunLoopDoSources0 + 556
    frame #24: 0x00007fffa32f6e76 CoreFoundation`__CFRunLoopRun + 934
    frame #25: 0x00007fffa32f6874 CoreFoundation`CFRunLoopRunSpecific + 420
    frame #26: 0x00007fffa4d12cb2 Foundation`-[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 277
    frame #27: 0x000000010300c84e Google Chrome Framework`base::MessagePumpNSRunLoop::DoRun(this=0x0000000115c4d240, delegate=<unavailable>) + 126 at message_pump_mac.mm:608 [opt]
    frame #28: 0x000000010300c02c Google Chrome Framework`base::MessagePumpCFRunLoopBase::Run(this=0x0000000115c4d240, delegate=0x0000000100215e10) + 92 at message_pump_mac.mm:238 [opt]
    frame #29: 0x0000000103026971 Google Chrome Framework`base::RunLoop::Run(this=0x0000700019ddbe60) + 113 at run_loop.cc:35 [opt]
    frame #30: 0x0000000103050d42 Google Chrome Framework`base::Thread::ThreadMain(this=0x0000000100215c00) + 386 at thread.cc:333 [opt]
    frame #31: 0x000000010304c8d7 Google Chrome Framework`base::(anonymous namespace)::ThreadFunc(params=<unavailable>) + 87 at platform_thread_posix.cc:71 [opt]
    frame #32: 0x00007fffb85deabb libsystem_pthread.dylib`_pthread_body + 180
    frame #33: 0x00007fffb85dea07 libsystem_pthread.dylib`_pthread_start + 286
    frame #34: 0x00007fffb85de231 libsystem_pthread.dylib`thread_start + 13

And 0x115c4c820, the thing we’re calling objc_destructInstance() on, is a NetServiceBrowserDelegate.

Better stack from a debug build (gotta be using Google Chrome branding, though):

(lldb) bt
* thread #27: tid = 0x1fc40f, 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error, name = 'Service Discovery Thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error
    frame #1: 0x00007fffb7ae78de libobjc.A.dylib`weak_clear_no_lock + 140
    frame #2: 0x00007fffb7ae13c4 libobjc.A.dylib`objc_object::sidetable_clearDeallocating() + 168
    frame #3: 0x00007fffb7ae10b7 libobjc.A.dylib`objc_destructInstance + 143
    frame #4: 0x000000010a16455b Google Chrome Framework`(anonymous namespace)::ZombieDealloc(self=0x000000013bc9e200, _cmd="dealloc") + 395 at objc_zombie.mm:115
    frame #5: 0x00007fffb7ae6c49 libobjc.A.dylib`objc_object::sidetable_release(bool) + 285
    frame #6: 0x0000000107d0e11c Google Chrome Framework`base::internal::ScopedNSProtocolTraitsRelease(obj=0x000000013bc9e200) + 28 at scoped_nsobject.mm:19
    frame #7: 0x00000001067e1085 Google Chrome Framework`base::internal::ScopedNSProtocolTraits<objc_object*>::Release(nst=0x000000013bc9e200) + 21 at scoped_nsobject.h:75
    frame #8: 0x00000001067e106a Google Chrome Framework`base::ScopedTypeRef<objc_object*, base::internal::ScopedNSProtocolTraits<objc_object*> >::~ScopedTypeRef(this=0x0000000100438b70) + 42 at scoped_typeref.h:84
    frame #9: 0x00000001067e1035 Google Chrome Framework`base::scoped_nsprotocol<objc_object*>::~scoped_nsprotocol(this=0x0000000100438b70) + 21 at scoped_nsobject.h:82
    frame #10: 0x00000001067e1015 Google Chrome Framework`base::scoped_nsobject<objc_object*>::~scoped_nsobject(this=0x0000000100438b70) + 21 at scoped_nsobject.h:196
    frame #11: 0x00000001067e0ee5 Google Chrome Framework`base::scoped_nsobject<objc_object*>::~scoped_nsobject(this=0x0000000100438b70) + 21 at scoped_nsobject.h:196
    frame #12: 0x0000000107a15aba Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer(this=0x0000000100438b40) + 362 at service_discovery_client_mac.mm:189
    frame #13: 0x0000000107a15ce5 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer(this=0x0000000100438b40) + 21 at service_discovery_client_mac.mm:187
    frame #14: 0x0000000107a1eca7 Google Chrome Framework`base::DeleteHelper<local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer>::DoDelete(object=0x0000000100438b40) + 39 at sequenced_task_runner_helpers.h:40
    frame #15: 0x0000000107df7015 Google Chrome Framework`void base::internal::FunctorTraits<void (*)(void const*), void>::Invoke<void const* const&>(function=(Google Chrome Framework`base::DeleteHelper<local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer>::DoDelete(void const*) at sequenced_task_runner_helpers.h:39), args=0x00000001402fb038)(void const*), void const* const&&&) + 37 at bind_internal.h:164
    frame #16: 0x0000000107df6fc0 Google Chrome Framework`void base::internal::InvokeHelper<false, void>::MakeItSo<void (functor=0x00000001402fb030, args=0x00000001402fb038)(void const*), void const* const&>(void (* const&&&)(void const*), void const* const&&&) + 48 at bind_internal.h:285
    frame #17: 0x0000000107df6f88 Google Chrome Framework`void base::internal::Invoker<base::internal::BindState<void (*)(void const*), void const*>, void ()>::RunImpl<void (functor=0x00000001402fb030, bound=0x00000001402fb038, (null)=IndexSequence<0> @ 0x00007000135c6258)(void const*), std::__1::tuple<void const*> const&, 0ul>(void (* const&&&)(void const*), std::__1::tuple<void const*> const&&&, base::IndexSequence<0ul>) + 72 at bind_internal.h:361
    frame #18: 0x0000000107df6edc Google Chrome Framework`base::internal::Invoker<base::internal::BindState<void (*)(void const*), void const*>, void ()>::Run(base=0x00000001402fb010) + 44 at bind_internal.h:339
    frame #19: 0x0000000107eda04a Google Chrome Framework`base::internal::RunMixin<base::Callback<void (), (base::internal::CopyMode)0, (base::internal::RepeatMode)0> >::Run(this=0x00007000135c6888) + 90 at callback.h:47
    frame #20: 0x0000000107c6abbf Google Chrome Framework`base::debug::TaskAnnotator::RunTask(this=0x0000000100447e30, queue_function="MessageLoop::PostTask", pending_task=0x00007000135c6870) + 687 at task_annotator.cc:52
    frame #21: 0x0000000107d364bb Google Chrome Framework`base::MessageLoop::RunTask(this=0x0000000100447d00, pending_task=0x00007000135c6870) + 923 at message_loop.cc:413
    frame #22: 0x0000000107d369e4 Google Chrome Framework`base::MessageLoop::DeferOrRunPendingTask(this=0x0000000100447d00, pending_task=PendingTask @ 0x00007000135c6870) + 68 at message_loop.cc:422
    frame #23: 0x0000000107d3741d Google Chrome Framework`base::MessageLoop::DoWork(this=0x0000000100447d00) + 669 at message_loop.cc:515
    frame #24: 0x0000000107d483e8 Google Chrome Framework`base::MessagePumpCFRunLoopBase::RunWork(this=0x000000013bc9fd00) + 104 at message_pump_mac.mm:330
    frame #25: 0x0000000107d4836c Google Chrome Framework`::___ZN4base24MessagePumpCFRunLoopBase13RunWorkSourceEPv_block_invoke(.block_descriptor=<unavailable>) + 28 at message_pump_mac.mm:307
    frame #26: 0x0000000107cf2aea Google Chrome Framework`base::mac::CallWithEHFrame(void () block_pointer) + 10 at call_with_eh_frame_asm.S:36
    frame #27: 0x0000000107d478e5 Google Chrome Framework`base::MessagePumpCFRunLoopBase::RunWorkSource(info=0x000000013bc9fd00) + 101 at message_pump_mac.mm:306
    frame #28: 0x00007fffa3316581 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    frame #29: 0x00007fffa32f798c CoreFoundation`__CFRunLoopDoSources0 + 556
    frame #30: 0x00007fffa32f6e76 CoreFoundation`__CFRunLoopRun + 934
    frame #31: 0x00007fffa32f6874 CoreFoundation`CFRunLoopRunSpecific + 420
    frame #32: 0x00007fffa4d12cb2 Foundation`-[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 277
    frame #33: 0x0000000107d48ec9 Google Chrome Framework`base::MessagePumpNSRunLoop::DoRun(this=0x000000013bc9fd00, delegate=0x0000000100447d00) + 137 at message_pump_mac.mm:608
    frame #34: 0x0000000107d480ca Google Chrome Framework`base::MessagePumpCFRunLoopBase::Run(this=0x000000013bc9fd00, delegate=0x0000000100447d00) + 122 at message_pump_mac.mm:238
    frame #35: 0x0000000107d35f45 Google Chrome Framework`base::MessageLoop::RunHandler(this=0x0000000100447d00) + 645 at message_loop.cc:378
    frame #36: 0x0000000107df3dc5 Google Chrome Framework`base::RunLoop::Run(this=0x00007000135c85a8) + 85 at run_loop.cc:35
    frame #37: 0x0000000107ec5cac Google Chrome Framework`base::Thread::Run(this=0x0000000100448800, run_loop=0x00007000135c85a8) + 428 at thread.cc:245
    frame #38: 0x0000000107ec6db8 Google Chrome Framework`base::Thread::ThreadMain(this=0x0000000100448800) + 3512 at thread.cc:333
    frame #39: 0x0000000107ea69a1 Google Chrome Framework`base::(anonymous namespace)::ThreadFunc(params=0x0000000100498ed0) + 705 at platform_thread_posix.cc:71
    frame #40: 0x00007fffb85deabb libsystem_pthread.dylib`_pthread_body + 180
    frame #41: 0x00007fffb85dea07 libsystem_pthread.dylib`_pthread_start + 286
    frame #42: 0x00007fffb85de231 libsystem_pthread.dylib`thread_start + 13
(lldb) po 0x000000013bc9e200
<NetServiceBrowserDelegate: 0x13bc9e200>

> Google Chrome[29616]: objc[29616]: __weak variable at 0x600002654890 holds 0x2121212121212121 instead of 0x60000263f8e0. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug.

The 0x2121212121212121 comes from ZombieDealloc() (components/crash/core/common/objc_zombie.mm). I changed the '!' to '@', and the message changed to implicate 0x4040404040404040. I tried 0xff and 0x01 and the message changed accordingly in those cases too. I changed it to 0x00 and the messages stopped.

I wonder if this is still true:

  // NOTE(shess): |object_dispose()| will call this again when the
  // zombie falls off the treadmill!  But by then |isa| will be a
  // class without C++ destructors or associative references, so it
  // won't hurt anything.

Of course the 0x00 makes the messages stop, but that’s probably just masking the problem. Here’s where the messages come from. The 0x2121212121212121 is what’s showing up as *referrer.

objc4-680/runtime/objc-weak.mm weak_clear_no_lock()
485              else if (*referrer) {
486                  _objc_inform("__weak variable at %p holds %p instead of %p. "
487                               "This is probably incorrect use of "
488                               "objc_storeWeak() and objc_loadWeak(). "
489                               "Break on objc_weak_error to debug.\n", 
490                               referrer, (void*)*referrer, (void*)referent);
491                  objc_weak_error();
492              }

They're always changing things, but doesn't this imply that the runtime got all the way through destruction once?  So why is it still referencing anything inside the object?  The comment is because the original |isa| is gone, so it's unclear how it can be making dynamic decisions.  Though I suppose sidetable_clearDeallocating() implies that it's squirreling stuff away on the side, so maybe multiple freed objects are sharing context somehow (could 0x(21)* sometimes be valid memory someone's assuming contains structure?).

There's another table in the zombie code storing backtrace info from where the object was originally destructed.  |wasa| comes after |isa| and is the original |isa|.

My current theory is that nothing is destroying the weak associations or invalidating the old object’s side table entry. I don’t see any calls to objc_destroyWeak() or equivalent in objc4, so if a side table entry is present for a former object at the same address, it’ll come back to haunt a later object.

This isn’t really about our zombies, anyway. Disabling our zombies entirely still produces

objc[35442]: __weak variable at 0x10058f2c0 holds 0x100580003 instead of 0x10058f100. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug.
[…]
(lldb) bt
* thread #28: tid = 0x286cf6, 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error, name = 'Service Discovery Thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error
    frame #1: 0x00007fffb7ae78de libobjc.A.dylib`weak_clear_no_lock + 140
    frame #2: 0x00007fffb7ae13c4 libobjc.A.dylib`objc_object::sidetable_clearDeallocating() + 168
    frame #3: 0x00007fffb7ae10b7 libobjc.A.dylib`objc_destructInstance + 143
    frame #4: 0x00007fffb7ae1017 libobjc.A.dylib`object_dispose + 22
    frame #5: 0x00007fffb7ae6c49 libobjc.A.dylib`objc_object::sidetable_release(bool) + 285
    frame #6: 0x0000000107d0dd1c Google Chrome Framework`base::internal::ScopedNSProtocolTraitsRelease(obj=0x000000010058f100) + 28 at scoped_nsobject.mm:19
    frame #7: 0x00000001067e0c85 Google Chrome Framework`base::internal::ScopedNSProtocolTraits<objc_object*>::Release(nst=0x000000010058f100) + 21 at scoped_nsobject.h:75
    frame #8: 0x00000001067e0c6a Google Chrome Framework`base::ScopedTypeRef<objc_object*, base::internal::ScopedNSProtocolTraits<objc_object*> >::~ScopedTypeRef(this=0x000000013bba2e20) + 42 at scoped_typeref.h:84
    frame #9: 0x00000001067e0c35 Google Chrome Framework`base::scoped_nsprotocol<objc_object*>::~scoped_nsprotocol(this=0x000000013bba2e20) + 21 at scoped_nsobject.h:82
    frame #10: 0x00000001067e0c15 Google Chrome Framework`base::scoped_nsobject<objc_object*>::~scoped_nsobject(this=0x000000013bba2e20) + 21 at scoped_nsobject.h:196
    frame #11: 0x00000001067e0ae5 Google Chrome Framework`base::scoped_nsobject<objc_object*>::~scoped_nsobject(this=0x000000013bba2e20) + 21 at scoped_nsobject.h:196
    frame #12: 0x0000000107a156ba Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer(this=0x000000013bba2df0) + 362 at service_discovery_client_mac.mm:189
    frame #13: 0x0000000107a158e5 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer(this=0x000000013bba2df0) + 21 at service_discovery_client_mac.mm:187
    frame #14: 0x0000000107a1e8a7 Google Chrome Framework`base::DeleteHelper<local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer>::DoDelete(object=0x000000013bba2df0) + 39 at sequenced_task_runner_helpers.h:40
(lldb) po 0x000000010058f100
<NetServiceBrowserDelegate: 0x10058f100>

objc[35544]: __weak variable at 0x13453d1a0 holds 0x2121212121212121 instead of 0x10012ccc0. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug.

The “__weak variable at” (here, 0x13453d1a0) is at offset 0x10 in an ex-NSNetServiceBrowser object (would have been 0x13453d190). No surprise, since NetServiceBrowserDelegate is the delegate that we give to NSNetServiceBrowser.

(Calling objc_getClass to verify that 0x11bd51858 is CrFatZombie because real CrFatZombies don’t have a -class or +class, they prefer to crash on any message.)

(lldb) x -c 64 0x13453d190
0x13453d190: 58 18 d5 1b 01 00 00 00 f0 78 8a bd ff 7f 00 00  X.?.....?x.??...
0x13453d1a0: 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21  !!!!!!!!!!!!!!!!
0x13453d1b0: 21 21 21 21 21 21 21 21 00 00 00 00 00 00 00 00  !!!!!!!!........
0x13453d1c0: ea 74 45 13 00 00 00 a0 a2 66 45 13 00 00 00 50  ?tE....??fE....P
(lldb) print (void*)objc_getClass("CrFatZombie")
(void *) $3 = 0x000000011bd51858
(lldb) po 0x7fffbd8a78f0
NSNetServiceBrowser
(lldb) print (void*)malloc(sizeof(ZombieRecord))
(void *) $5 = 0x000000010022d470
(lldb) print GetZombieRecord((id)0x13453d190, (ZombieRecord*)$5)
(BOOL) $6 = YES
(lldb) x --format p --count 23 $26
0x10074ed30: 0x000000013453d190 0x00007fffbd8a78f0 0x000000010a16462d 0x00007fffa273091d
0x10074ed50: 0x00007fffb7ae6c49 0x0000000107d0e11c 0x0000000107a1d2a5 0x0000000107a1d28a
0x10074ed70: 0x0000000107a1d255 0x0000000107a1d235 0x0000000107a158f5 0x0000000107a15aa7
0x10074ed90: 0x0000000107a15ce5 0x0000000107a1eca7 0x0000000107df7015 0x0000000107df6fc0
0x10074edb0: 0x0000000107df6f88 0x0000000107df6edc 0x0000000107eda04a 0x0000000107c6abbf
0x10074edd0: 0x0000000107d364bb 0x0000000107d369e4 0x0000000000000014
(lldb) image lookup --address 0x10a16462d
[…]
      Summary: Google Chrome Framework`(anonymous namespace)::ZombieDealloc(objc_object*, objc_selector*) + 605 at objc_zombie.mm:139

and doing that for each element in the ZombieRecord’s backtrace gives

0x000000010a16462d Google Chrome Framework`(anonymous namespace)::ZombieDealloc(objc_object*, objc_selector*) + 605 at objc_zombie.mm:139
0x00007fffa273091d CFNetwork`-[NSNetServiceBrowser dealloc] + 103
0x00007fffb7ae6c49 libobjc.A.dylib`objc_object::sidetable_release(bool) + 285
0x0000000107d0e11c Google Chrome Framework`base::internal::ScopedNSProtocolTraitsRelease(objc_object*) + 28 at scoped_nsobject.mm:19
0x0000000107a1d2a5 Google Chrome Framework`base::internal::ScopedNSProtocolTraits<NSNetServiceBrowser*>::Release(NSNetServiceBrowser*) + 21 at scoped_nsobject.h:76
0x0000000107a1d28a Google Chrome Framework`base::ScopedTypeRef<NSNetServiceBrowser*, base::internal::ScopedNSProtocolTraits<NSNetServiceBrowser*> >::~ScopedTypeRef() + 42 at scoped_typeref.h:85
0x0000000107a1d255 Google Chrome Framework`base::scoped_nsprotocol<NSNetServiceBrowser*>::~scoped_nsprotocol() + 21 at scoped_nsobject.h:82
0x0000000107a1d235 Google Chrome Framework`base::scoped_nsobject<NSNetServiceBrowser>::~scoped_nsobject() + 21 at scoped_nsobject.h:147
0x0000000107a158f5 Google Chrome Framework`base::scoped_nsobject<NSNetServiceBrowser>::~scoped_nsobject() + 21 at scoped_nsobject.h:147
0x0000000107a15aa7 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() + 343 at service_discovery_client_mac.mm:189
0x0000000107a15ce5 Google Chrome Framework`local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer::~NetServiceBrowserContainer() + 21 at service_discovery_client_mac.mm:189
0x0000000107a1eca7 Google Chrome Framework`base::DeleteHelper<local_discovery::ServiceWatcherImplMac::NetServiceBrowserContainer>::DoDelete(void const*) + 39 at sequenced_task_runner_helpers.h:40
0x0000000107df7015 Google Chrome Framework`void base::internal::FunctorTraits<void (*)(void const*), void>::Invoke<void const* const&>(void (*)(void const*), void const* const&&&) + 37 at bind_internal.h:164
0x0000000107df6fc0 Google Chrome Framework`void base::internal::InvokeHelper<false, void>::MakeItSo<void (* const&)(void const*), void const* const&>(void (* const&&&)(void const*), void const* const&&&) + 48 at bind_internal.h:285
0x0000000107df6f88 Google Chrome Framework`void base::internal::Invoker<base::internal::BindState<void (*)(void const*), void const*>, void ()>::RunImpl<void (* const&)(void const*), std::__1::tuple<void const*> const&, 0ul>(void (* const&&&)(void const*), std::__1::tuple<void const*> const&&&, base::IndexSequence<0ul>) + 72 at bind_internal.h:361
0x0000000107df6edc Google Chrome Framework`base::internal::Invoker<base::internal::BindState<void (*)(void const*), void const*>, void ()>::Run(base::internal::BindStateBase*) + 44 at bind_internal.h:339
0x0000000107eda04a Google Chrome Framework`base::internal::RunMixin<base::Callback<void (), (base::internal::CopyMode)0, (base::internal::RepeatMode)0> >::Run() + 90 at callback.h:47
0x0000000107c6abbf Google Chrome Framework`base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) + 687 at task_annotator.cc:52
0x0000000107d364bb Google Chrome Framework`base::MessageLoop::RunTask(base::PendingTask*) + 923 at message_loop.cc:413
0x0000000107d369e4 Google Chrome Framework`base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) + 68 at message_loop.cc:425

This is the NSNetServiceBrowser owned by ServiceWatcherImplMac:: NetServiceBrowserContainer in browser_. In fact, it’s probably the NSNetServiceBrowser that was just torn down a moment before doing the same to NetServiceBrowserDelegate, since they’re declared right next to each other in the class, the delegate first, and thus last to be released.

> This is the NSNetServiceBrowser owned by
> ServiceWatcherImplMac:: NetServiceBrowserContainer in
> browser_. In fact, it’s probably the NSNetServiceBrowser that
> was just torn down a moment before doing the same to
> NetServiceBrowserDelegate, since they’re declared right next to
> each other in the class, the delegate first, and thus last to
> be released.

|delegate_| is before |browser_| - so shouldn't the delegate be released before the browser?  Wonder what happens if you swap the order (or call -setDelegate: in the destructor).

Reversing the declaration order of browser_ and delegate_ so that browser_ is declared first and thus destroyed last appears to prevent this from happening.

#c12: Destruction starts with the last-declared member and works backwards to the first-declared member. http://en.cppreference.com/w/cpp/language/destructor#Destruction_sequence

Grr, I think I had Australian scrolling enabled.

If cross-refs in destruction matters for this case, perhaps it also matters for |delegate_| and |service_| in NetServiceContainer.

Note that the code also does [netService setDelegate:self] in didFindService:, without explicitly clearing it in didRemoveService.  It's not obvious at all whether that delegate setting is now dangling in some object out there or not.

[Sigh.  And docs say NSNetServiceBrowser is async, and our code runs it on a separate thread, so it's extra-async?]

Yeah. We suck. Threads are free, though. Here, have a few extra.

Proof that the bug has nothing to do with Chrome or its usage of these interfaces:

mark@garbage bash$ clang -g weaksauce.m -o weaksauce -framework Foundation
mark@garbage bash$ lldb weaksauce
(lldb) target create "weaksauce"
Current executable set to 'weaksauce' (x86_64).
(lldb) env MallocScribble=1
(lldb) b objc_weak_error
Breakpoint 1: where = libobjc.A.dylib`objc_weak_error, address = 0x000000000002025a
(lldb) r
Process 36547 launched: '/Users/mark/weaksauce' (x86_64)
weaksauce(36547,0x7fffc10cb3c0) malloc: enabling scribbling to detect mods to free blocks
objc[36547]: __weak variable at 0x1001070d0 holds 0x5555555555555555 instead of 0x100105260. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug.

Process 36547 stopped
* thread #1: tid = 0x2ba94b, 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error
libobjc.A.dylib`objc_weak_error:
->  0x7fffb7af825a <+0>: retq   

libobjc.A.dylib`bad_weak_table:
    0x7fffb7af825b <+0>: pushq  %rbp
    0x7fffb7af825c <+1>: movq   %rsp, %rbp
    0x7fffb7af825f <+4>: movq   %rdi, %rcx
(lldb) bt
* thread #1: tid = 0x2ba94b, 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007fffb7af825a libobjc.A.dylib`objc_weak_error
    frame #1: 0x00007fffb7ae78de libobjc.A.dylib`weak_clear_no_lock + 140
    frame #2: 0x00007fffb7ae7800 libobjc.A.dylib`objc_object::clearDeallocating_slow() + 104
    frame #3: 0x00007fffb7ae10c1 libobjc.A.dylib`objc_destructInstance + 153
    frame #4: 0x00007fffb7ae1017 libobjc.A.dylib`object_dispose + 22
    frame #5: 0x0000000100001aad weaksauce`main(argc=1, argv=0x00007fff5fbffa30) + 285 at weaksauce.m:26
    frame #6: 0x00007fffb83c8255 libdyld.dylib`start + 1
(lldb) frame select 5
frame #5: 0x0000000100001aad weaksauce`main(argc=1, argv=0x00007fff5fbffa30) + 285 at weaksauce.m:26
   23  	    }
   24  	#if !__has_feature(objc_arc)
   25  	    [browser release];
-> 26  	    [delegate release];
   27  	#endif
   28  	#if !__has_feature(objc_arc)
   29  	  }
(lldb) frame variable
(int) argc = 1
(char **) argv = 0x00007fff5fbffa30
(NetServiceBrowserDelegate *) delegate = 0x0000000100105260
(NSNetServiceBrowser *) browser = 0x00000001001070c0

Deleted: weaksauce.m 1.0 KB

weaksauce.m 1.0 KB View Download

The same error when run Chrome in the new profile as:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir="$PWD/Foo"

Then this warning:

> objc[7592]: __weak variable at 0x7f8727017b70 holds 0x2121212121212121 instead of 0x7f87270d57d0. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug.

is repeated on the screen every few minutes, even when the Chrome is in the background, so I'm basically doing nothing apart of using another instance of Chrome (not related to this which gives the error).

In #c15, shess suggested that this may be relevant to NSNetService (not just NSNetServiceBrowser) and its delegate too. This version of the testcase proves that this is the case.

Deleted: weaksauce.m 1.1 KB

weaksauce.m 1.1 KB View Download

Went off to do some reading, and zeroing weak refs sound like a good solution to the problem of delegates (basically the ordering thing here).  So my hypothesis is that the NSNetServiceBrowser is maintaining a weak ref to the delegate, then when the delegate is in dealloc, the system calls in to "Clear all the weak refs to me", and the deallocated NSNetServiceBrowser instance is still on some list.

That may have been obvious to you, I dunno.

Anyhow, that means the immediate issue would be resolved by explicitly clearing the delegate before anyone is deallocated, and deallocating the delegate first would also work (and maybe would appear to have cleared the delegate?).  But it does imply that the zombie code needs to learn a new trick.

Hmm, this might also imply that the problem is that NSNetServiceBrowser is still on the zombie treadmill, so:

@implementation NSNetServiceBrowser (CrZombie)
- (BOOL)shouldBecomeCrZombie {
  return NO;
}
@end

plus changing ZombieEnable() to pass false for the first parameter would also change things.  I'm actually not clear why we have that lever, as all of the current calls pass YES or true when setting it up.

So that means that objc_destructInstance() isn't sufficient to remove all traces of the object from the runtime structures.

The problem with my reasoning in #20 and #21 is that reading the runtime, I can't see why it wouldn't equally apply to any object without zombie being involved.  Which is probably what #17 (it has nothing to do with us) points at, but I find it nervous-making.  I think it means that the machinery could just happen to reach in to someone else's allocation and clear things out, even if the pattern matched merely by coincidence.  Admittedly, that would be pretty unlikely, it's not like you could reallocate the object and end up with the currently-clearing-weak-points object in that slot, since that object was being freed, but is not yet available to be reallocated.

Anyhow, if that's correct reasoning, the issue is not solvable in the limit, but the zombie treadmill would exacerbate it.  For the message to be logged without the treadmill, the object would have to be freed, the space reallocated, and then overwritten with new data that doesn't match.  That certainly would be expected to happen, but in a case like this where the two objects are deallocated one after the other, and were probably allocated on the same thread, it's not likely.  Groups of objects with weak references to each other probably are often deallocated together in bursts, not leaving space for someone else to reallocate their buffers before everything unwinds.

I can see why they wouldn't have a set of forward references to clean this up in deallocation, as it would double the runtime overhead.  Right now all of that overhead is concentrated on the object which chooses to maintain weak references to others, which seems reasonable.

This isn’t really about zombies at all (see #c10), but having our zombies enabled makes the problem easier to spot because we overwrite most of the object (aside from the isa and wasa pointers) with a 0x21 pattern. Without that, the memory would need to be reused in order for it to contain an unexpected value when the delegate is deallocated. That’s certainly possible, but with the delegate being deallocated immediately after the thing that’s delegating to it, the window to see a problem is much smaller.

In other words, I’m able to reproduce the problem without zombies at all. That’s what #c17 and #c19 are.

For this to be consistently reproducible without zombies, use MallocScribble=1.

This isn’t our bug, and our zombie code is sound. There’s a problem with the NSNetServiceBrowser and NSNetService classes. It doesn’t appear to be a problem with the __weak system overall, because I’m unable to reproduce this bad behavior in my own class with a __weak instance variable. Rather, it appears that NSNetServiceBrowser and NSNetService are doing their own thing, perhaps interfacing with the objc_weak portion of the runtime “manually,” and getting it wrong. The “forward” use of objc_weak is probably working for them, in that a deallocated delegate nils out the delegate pointer in the NS* object, but the “backwards” use of it is not working at all, in that a deallocating delegate still tries to clear the pointer to itself in an already-deallocated NS* object.

I’m writing the bug report up for Apple and will stash a link here. In the mean time, if we decide we want to work around it on our end, we can nil out the delegates before releasing NSNetServiceBrowser and NSNetService, or we can reverse the release order to get rid of the delegates before the objects that delegate to them.

I filed radar 28943305. https://openradar.appspot.com/28943305

Deleted: weaksauce.m 1.1 KB

weaksauce.m 1.1 KB View Download

When I write my own class with a __weak instance variable, clang creates a synthetic .cxx_destruct method that calls objc_destroyWeak. That’s what I said was missing in #c9. You can even do this with a simple property (-fobjc-arc is required for “weak”):

@interface Doer : NSObject
@property(nonatomic, weak) Delegate* delegate;
@end
@implementation Doer
@end

synthesizes

- (Delegate*)delegate {
  return [objc_loadWeakRetained(&_delegate) autorelease];
}
- (void)setDelegate:(Delegate*)delegate {
  objc_storeWeak(&_delegate, delegate);
}
- (void).cxx_destruct {
  objc_destroyWeak(&_delegate);
}

NSNetService and NSNetServiceBrowser don’t do this. (If you’re looking for them, they’re in CFNetwork.framework.) Here’s what they’ve got:

- (id)delegate {
  return objc_loadWeak(&_delegate);  // note: not retained and autoreleased
}
- (void)setDelegate:(Delegate*)delegate {
  objc_storeWeak(&_delegate, delegate);
}

There’s no -.cxx_destruct, and -dealloc doesn’t mention _delegate at all. So based on the fact that their property definitions don’t mention “weak” at all (they’re (nullable, assign)) and the fact that they’ve got “_delegate” instance variables that also aren’t called weak, it’s seems like they’ve hand-written the code I showed above. Perhaps this is because they didn’t want to turn ARC on for these implementations. But it’s clear that they’re never calling objc_destroyWeak, which is exactly why our delegate object never finds out that their object has been deallocated.

Ah, that makes sense - an implementation using objc_destroyWeak() would be essentially the same as us calling -setDelegate:nil .

Yes. In fact, objc_destroyWeak(&_delegate) is equivalent to objc_storeWeak(&_delegate, nil) (per https://opensource.apple.com/source/objc4/objc4-680/runtime/NSObject.mm), which is exactly what you get with [o setDelegate:nil].

Follow-up to #c25: the comment “note: not retained and autoreleased” is incorrect. objc_loadWeak(&_delegate) is exactly equivalent to [objc_loadWeakRetained(&_delegate) autorelease]. Both return a retained autoreleased object.

I’m not seeing this bug in 10.11.6 15G31. I am seeing it in 10.12.0 16A323 and 10.12.1 16B2555. This is new to 10.12.

Chrome-side workaround: https://codereview.chromium.org/2445343005

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/97de983007388a7a537ad8c1e5fbbd1bb20f4305

commit 97de983007388a7a537ad8c1e5fbbd1bb20f4305
Author: mark <mark@chromium.org>
Date: Wed Oct 26 17:39:12 2016

Work around a macOS 10.12 bug involving objects with weak delegates

NSNetService and NSNetServiceBrowser each have non-owned delegates. In
10.12, Apple appears to have updated them to maintain their delegate
references as zeroing weak pointers. Unfortunately, they seem to have
done this by hand instead of using the compiler's support for it, and
they left something out. When objects of either of these classes are
deallocated, they fail to unregister their weak interest in their
delegates. When the delegates subsequently are deallocated, they will
attempt to clear the pointers to themselves in objects that no longer
exist. This manifests itself in messages like:

Google Chrome[29616]: objc[29616]: __weak variable at 0x600002654890
holds 0x2121212121212121 instead of 0x60000263f8e0. This is probably
incorrect use of objc_storeWeak() and objc_loadWeak(). Break on
objc_weak_error to debug.

This bug has been filed with Apple as
http://openradar.appspot.com/28943305. As a workaround, set these
objects' delegates to nil when destroying them. This removes their weak
interest in their delegates.

BUG= 657495 

Review-Url: https://codereview.chromium.org/2445343005
Cr-Commit-Position: refs/heads/master@{#427736}

[modify] https://crrev.com/97de983007388a7a537ad8c1e5fbbd1bb20f4305/chrome/browser/local_discovery/service_discovery_client_mac.mm

The ball’s in Apple’s court now, but since we worked around it, let’s call it Fixed instead of waiting for them to fix it, which would be ExternalDependency. Follow-ups on the latter will be at https://openradar.appspot.com/28943305.

I hope I'm posting this comment in the right place. In my `system.log` I still see A LOT of error messages of the following form:

Oct  5 08:09:07 hypercube Google Chrome[281]: objc[281]: __weak variable at 0x608000e398b0 holds 0x2121212121212121 instead of 0x608000e34460. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug

As the fix date is October 2016, I was expecting these errors not to appear. I'm running a macOS 10.12.6 and my Chrome is 61.0.3163.100 (Official Build) (64-bit).

Just tested it again on macOS 10.13 and Chrome 64.0.3251.0, can confirm those errors are still showing up.