This has spiked inconsistently on Mac hence unable to pin point to any regression range.

Note:
=====
1. Latest stable(53.0.2785.116/143) has reported 559/93 crash instances from 306/61 clients.
2. All the crashes are from Mac OS 10.12(Sierra) on the stable builds.
3. Looks like upgrade to Mac OS 10.12(Sierra) is exposing these crashes.

Marking this as Stable blocker for M-55. Looping Mac owners for help in further investigation of this.

 

Users experienced this crash on the following builds:

Mac Canary 55.0.2880.0 -  1.00 CPM, 3 reports, 2 clients (signature -[_NSTableRowHeightStorage numberOfRows])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

98% of the crashes are on 10.12, so it's either a 10.12 bug or a Chrome bug being tickled by 10.12.

NSTableView appears in the omnibox, task manager, and Font panel.

[mac triage] And an AppKit bug in 10.12 sounds likely. Omnibox has an NSTableView, but maybe not in an NSScrollView which seems to be interacting here. I'm not sure how to invoke the font panel (is it owned by the chrome process?). So.. my money is on the task manager.

viewDidChangeBackingProperties is usually invoked when moving a window between monitors (e.g. of different resolutions or colorspaces), which ccameron knows a lot about [dobbing you in - sorry]. avi also played with the Cocoa task manager recently. Maybe moving the task manager between retina/non-retina screens will get a repro that we can send to Apple (I'm currently lacking equipment for that).

But also this probably isn't a regression wrt Chrome, so doesn't need to block stable. E.g. there are reports on m51. Oddly... only 2 reports on m52 (and one of those is 10.9). Maybe not so odd since m52 was out of stable when Sierra went GM. But 85% of reports are m53, which we can't block for.


910 crash reports in https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27-%5B_NSTableRowHeightStorage%20numberOfRows%5D%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=#samplereports:5,productversion:1000

Users experienced this crash on the following builds:

Mac Canary 56.0.2884.0 -  0.38 CPM, 1 reports, 1 clients (signature -[_NSTableRowHeightStorage numberOfRows])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Updating the latest behavior of the crash, After latest stable has pushed to market observing only single crash instance on #54.0.2840.59 as of now. No crashes has been observed on latest dev channel #55.0.2883.9. The last crash is observed on latest canary #56.0.2888.0 with 6 instances.

Below link gives in details of the number of instances in which the crash has occurred for associated builds:

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20%20AND%20custom_data.ChromeCrashProto.ptype%3D%27browser%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27-%5B_NSTableRowHeightStorage%20numberOfRows%5D%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

Users experienced this crash on the following builds:

Mac Beta 54.0.2840.59 -  0.21 CPM, 1 reports, 1 clients (signature -[_NSTableRowHeightStorage numberOfRows])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

ccameron@ Ping! This issue is marked as RB-Stable, could you please take a look in to this issue ?

As per #5, this should not be RBS.

The viewDidChangeBackingProperties message comes when monitor changes. I've added handlers for that to RWHVMac, but that's the extent of my familiarity with the message. This should go in the generic "Sierra bugs that need to be reproduced and investigated" bucket.

Issue 662636 has been merged into this issue.

 Issue 663720  has been merged into this issue.

The reporter in  issue 663720  confirms that this is related to unplugging a monitor.

Just to update:

This is the top#2 browser  crash on mac chrome version 56.0.2914.3 (released 10 hours ago) with 3 instances from 2 different client Ids

56.0.2915.0	0.05%	3	
56.0.2914.3	0.05%	3	from 2 unique client Ids
56.0.2914.0	0.14%	9	
56.0.2913.3	0.19%	12	
56.0.2913.0	0.02%	1	
56.0.2912.0	0.21%	13	
56.0.2911.0	0.08%	5	
56.0.2910.0	0.02%	1	
56.0.2909.0	0.13%	8	
56.0.2908.0	0.14%	9	
56.0.2907.0	0.13%	8	
56.0.2906.0	0.21%	13	
56.0.2905.0	0.02%	1	
56.0.2904.0	0.02%	1	
56.0.2903.0	0.02%	1	
56.0.2900.0	0.02%	1	
56.0.2899.0	0.03%	2	
56.0.2897.0	0.02%	1	
56.0.2896.0	0.11%	7	
56.0.2895.0	0.13%	8	
56.0.2894.0	0.13%	8	
56.0.2891.0	0.32%	20	
56.0.2890.0	0.16%	10	
56.0.2889.0	0.11%	7	
56.0.2888.0	0.16%	10	
56.0.2887.0	0.19%	12	
56.0.2886.0	0.21%	13	
56.0.2884.0	0.05%	3	
55.0.2883.35	0.19%	12	
55.0.2883.28	0.24%	15	
55.0.2883.21	0.22%	14	
55.0.2883.18	0.06%	4	
55.0.2883.11	0.06%	4	
55.0.2883.0	0.06%	4	
55.0.2882.0	0.10%	6	
55.0.2881.0	0.11%	7	
55.0.2880.0	0.08%	5	
55.0.2879.0	0.05%	3	
55.0.2878.0	0.05%	3	
55.0.2876.0	0.02%	1	
55.0.2875.0	0.05%	3	
55.0.2873.4	0.13%	8	
55.0.2873.0	0.02%	1	
55.0.2872.0	0.02%	1	
55.0.2868.3	0.03%	2	
55.0.2867.0	0.03%	2	
55.0.2859.0	0.08%	5	
55.0.2858.0	0.02%	1	
55.0.2855.0	0.02%	1	
55.0.2853.0	0.02%	1	
55.0.2851.0	0.02%	1	
55.0.2844.0	0.02%	1	
54.0.2840.98	0.16%	10	
54.0.2840.87	6.33%	399	
54.0.2840.71	41.24%	2600	

Effecting MacOS as below

1	10.12 (Sierra)	        99.84%	6295	
2	10.11 (El Capitan)	0.11%	7	
3	10.9 (Mavericks)	0.05%	3	

From the backtrace in  Issue 663720 , it looks like a tableview is trying to talk to its datasource after its datasource has been released. There are 8 classes in the codebase that define -numberOfRowsInTableView:. I think in all cases these classes also set instances of themselves to be a tableview's datasource. One of these classes appears to be used in test/demo code. Three others nil their tableview's datasource within their -dealloc method. These remaining four classes don't define a -dealloc method, and therefore set themselves to be a tableview datasource but don't clear the tableview datasource before they are themselves dealloc'ed:

ChooserBubbleUiController
ChooserDialogCocoaController
DevicePermissionsPrompt
OmniboxPopupMatrix

Suspecting ChooserDialogCocoaController, because its cl https://codereview.chromium.org/2005443002/ landed in 53.0.2756.0, and according to the link in c#7 there was a burst of crashes in 53.0.2785.116 and 53.0.2785.143. But in any case, we could go ahead and add the missing -deallocs to these classes (although ideally we reproduce the crasher so that we verify the fix).

sdy@ - do you have a few cycles to try to reproduce this crasher on a 10.12 machine? It seems to be related to detaching a monitor, and instantiating a ChooserDialogCocoaController and closing it before the detach might be the trick.

This is top#5 browser crash on previous beta i.e., 55.0.2883.44, Want to check if we have any update on the fix. Since this is one of the contributor for overall Mac browser crash spikes from M54 to M55.  

Please find the comparison from below url :
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27browser%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&compProp=product.Version&v1=55.0.2883.44&v2=54.0.2840.50#-magicsignature2:30,-stablesignature:30

Users experienced this crash on the following builds:

Mac Canary 57.0.2925.0 -  0.90 CPM, 2 reports, 2 clients (signature -[_NSTableRowHeightStorage numberOfRows])

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Issue 651617 has been merged into this issue.

This is proving to be tricky. I've been alternating between looking at the suspicious classes and trying to reproduce it (haven't been able to make it happen so far).

FWIW, it may not be an issue with the datasource (unless I misunderstood shrike@). If I break at numberOfRowsInTableView:, the call stack looks more like this:

  #0 0x0000000100001185 in -[DataSource numberOfRowsInTableView:]
  #1 0x00007fff9fdd18e9 in -[NSTableView _uncachedNumberOfRows]      <-- extra frame
  #2 0x00007fff9fd305ae in -[_NSTableRowHeightStorage numberOfRows]
  #3 0x00007fff9fd3010e in -[NSTableView _totalHeightOfTableView]

This crash is more like the table view itself is in an inconsistent state, but I'm still working on the details.

"This crash is more like the table view itself is in an inconsistent state"

+nick

It's something about the scrollbars changing when a pointing USB device is
plugged/unplugged

Thanks — I've been investigating that because of your stack trace, but unfortunately I haven't found a state where it causes the crash.

I am able to consistently reproduce this issue on my day to day workflow and have narrowed it down to the following use case:
1. Browse to a site using SSL client side certificate
2. Choose a certificate and complete loading the page
3. Connect external monitor
4. Chrome crash as soon as it appears on the 2nd monitor

This happens across multiple version, but this is the current configuration I'm testing on:
Chrome Version: 55.0.2883.59 (2883.59)
OS Version: Mac OS X 10.12.1 (16B2657)

Is it possibly related to https://bugs.chromium.org/p/chromium/issues/detail?id=313243 and the usage of SystemPrivate SFChooseIdentityPanel ?

omri@: Thanks so much for that insight. The crash is absolutely caused by the client cert dialog. It's not related to the private API use (but we might be able to get rid of it regardless).

I'll post a writeup soon.

This is an AppKit bug introduced in 10.12. It's a set of memory leaks that only lead to a crash when an app is linked against an SDK older than 10.11.

Chrome uses SFChooseIdentityPanel to offer the choice of client certs. That class, and the rest of the SecurityInterface framework, appears to have been updated in 10.12 to use Auto Layout. The changes also introduce a number of leaks — a bunch of views and windows are retained but only some of them are ever released.

The crash happens because the NSTableView inside the panel outlives its data source, the SFChooseIdentityPanel. I was wrong that this isn't a data source-related crash and I'm interested in why that frame isn't in the stack traces. Two events trigger the crash:

- A display change, like adding/removing a screen, or changing resolutions, causes -[NSView viewDidChangeBackingProperties] to be called for every view in every window of NSApp.windows. The sheet window was leaked and never explicitly closed, so it's still in the window list (this issue's stack trace).

- A scroll bar appearance change, which can be caused by adding/removing a mouse or just by changing the "Show scroll bars" setting in System Preferences, causes +[NSScrollerImpPair _updateAllScrollerImpPairsForNewRecommendedScrollerStyle:] to be called (issue 651617's stack trace). Explicitly closing the window doesn't help, because the notification is sent through NSNotificationCenter instead of the view hierarchy.

Safari experiences the leak. You can see it by running it in a debugger:

    $ lldb /Applications/Safari.app/Contents/MacOS/SafariForWebKitDevelopment
    (lldb) run
    # Trigger a client cert dialog, then close all windows and come back.
    ^C
    (lldb) po [NSApp windows]
    <__NSArrayM 0x107c1d9d0>(
        <NSWindow: 0x110614810> # Leaked!
    )

Safari doesn't crash because NSTableView's dataSource property became is a zeroing weak reference in 10.11. A `_CFExecutableLinkedOnOrAfter` check in AppKit keeps the non-zeroing behavior for apps linked against older SDKs, which bites us.

- - -

So far, it seems like the best workaround involves detecting the leak (does the window outlive the SFChooseIdentityPanel?), finding the table view, and nilling out its dataSource. I'll also file a radar. If anyone has other thoughts, speak up!

Filed: rdar://29409207

CL is up: https://codereview.chromium.org/2532203005

I also attached a fun little test that illustrates the leak.

Deleted: SFChooseIdentityPanel-leaks.swift 2.5 KB

SFChooseIdentityPanel-leaks.swift 2.5 KB Download

Also potentially useful for future similar problems, here's a program which prompts for a client cert when you visit https://127.0.0.1:8988/.

Deleted: client_cert_tester.go 1.4 KB

client_cert_tester.go 1.4 KB View Download

The CL has landed as 892a81ed3ba0c0ec6a2a5c79f1b91a3759be8c54, so let's watch tonight's Canary. (I'm not sure why bugdroid1 hasn't dropped in, will check with infra.)

I'm going to be OOO until 2016-12-8, would anyone be willing to shepherd some merges while I'm away?

Perhaps lgrey@?

Issue 670199 has been merged into this issue.

The CL was initially landed in 57.0.2938.0.The crash is not reported after- 57.0.2937.0, currently we have 57.0.2944.0 in production.

Guess its safe to merge.Lopping to the reviewer for further updates.

FYI:

M56 Beta promotion is scheduled on tomorrow, Dec 7 & RC cut today, Wednesday @ 4.00 PM PST.
Please request a merge if you would like it to make for next Release.

Yup, we should take this to M56 -- I can do the merge.

Your change meets the bar and is auto-approved for M56 (branch: 2924)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b368c8b8de696cfdb73772dae31712ba5d8eac75

commit b368c8b8de696cfdb73772dae31712ba5d8eac75
Author: Robert Sesek <rsesek@chromium.org>
Date: Thu Dec 08 18:06:12 2016

Work around an AppKit bug that can crash Chrome after using the client certificate prompt.

BUG= 653093 

Review-Url: https://codereview.chromium.org/2532203005
Cr-Commit-Position: refs/heads/master@{#435335}
(cherry picked from commit 892a81ed3ba0c0ec6a2a5c79f1b91a3759be8c54)

Review URL: https://codereview.chromium.org/2556423003 .

Cr-Commit-Position: refs/branch-heads/2924@{#409}
Cr-Branched-From: 3a87aecc31cd1ffe751dd72c04e5a96a1fc8108a-refs/heads/master@{#433059}

[modify] https://crrev.com/b368c8b8de696cfdb73772dae31712ba5d8eac75/chrome/browser/ui/cocoa/ssl_client_certificate_selector_cocoa.mm

I can confirm that with Chrome Beta v56 I do not experience the crash anymore. Thank you for the quick fix!

Whoo!

sdy - unfortunately this started happening again. I am not sure why I wasn't able to reproduce all day at work, but the moment I connected to my external monitor at home it started crashing again.

Anything I can provide that can assist ?

Stack trace is slightly different than before, see below -

Process:               Google Chrome [2132]
Path:                  /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
Identifier:            com.google.Chrome
Version:               56.0.2924.21 (2924.21)
Code Type:             X86-64 (Native)
Parent Process:        ??? [1]
Responsible:           Google Chrome [2132]
User ID:               501

Date/Time:             2016-12-09 01:05:46.610 -0800
OS Version:            Mac OS X 10.12.1 (16B2657)
Report Version:        12
Anonymous UUID:        03742708-92C6-2192-1A84-9CA4A7F73925

Sleep/Wake UUID:       520B43B0-A1DA-494C-933C-73068242814C

Time Awake Since Boot: 440 seconds

System Integrity Protection: enabled

Crashed Thread:        0  CrBrowserMain  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x00003a7370747480

VM Regions Near 0x3a7370747480:
    MALLOC_LARGE_REUSABLE  0000000138d15000-0000000139125000 [ 4160K] rw-/rwx SM=PRV  
--> 
    MALLOC_NANO            0000600000000000-000060000ac00000 [172.0M] rw-/rwx SM=PRV  

Application Specific Information:
objc_msgSend() selector name: numberOfRowsInTableView:


Thread 0 Crashed:: CrBrowserMain  Dispatch queue: com.apple.main-thread
0   libobjc.A.dylib               	0x00007fff8ed83b5d objc_msgSend + 29
1   com.apple.AppKit              	0x00007fff7813b8e9 -[NSTableView _uncachedNumberOfRows] + 250
2   com.apple.AppKit              	0x00007fff7809a5ae -[_NSTableRowHeightStorage numberOfRows] + 62
3   com.apple.AppKit              	0x00007fff7809a10e -[NSTableView _totalHeightOfTableView] + 224
4   com.apple.AppKit              	0x00007fff78099eaa -[NSTableView _minimumFrameSize] + 77
5   com.apple.AppKit              	0x00007fff77f8a86e -[NSScrollView reflectScrolledClipView:] + 245
6   com.apple.AppKit              	0x00007fff77f895f5 -[NSClipView _selfBoundsChanged] + 821
7   com.apple.AppKit              	0x00007fff7808a29d -[NSClipView setBoundsSize:] + 239
8   com.apple.AppKit              	0x00007fff7808a08d -[NSView setBounds:] + 185
9   com.apple.AppKit              	0x00007fff78089f46 -[NSScrollView viewDidChangeBackingProperties] + 164
10  com.apple.AppKit              	0x00007fff77fb80f3 _NSViewHierarchyDidChangeBackingProperties + 433
11  com.apple.AppKit              	0x00007fff7882d297 -[NSWindow _postWindowDidChangeBackingPropertiesAndDisplayWindowForPreviousBackingScaleFactor:previousColorSpace:] + 226
12  com.apple.AppKit              	0x00007fff78142631 __67-[NSWindow _updateSettingsSendingScreenChangeNotificationIfNeeded:]_block_invoke + 202
13  com.apple.AppKit              	0x00007fff780ce65c NSPerformVisuallyAtomicChange + 147
14  com.apple.AppKit              	0x00007fff7805033f -[NSWindow _updateSettingsSendingScreenChangeNotificationIfNeeded:] + 467
15  com.apple.AppKit              	0x00007fff78345019 -[NSWindow _displayChangedSoAdjustWindows:] + 139
16  com.apple.AppKit              	0x00007fff78120e97 __44-[NSApplication makeWindowsPerform:inOrder:]_block_invoke + 27
17  com.apple.AppKit              	0x00007fff783d3ce5 __52-[NSApplication _findWindowWithOptions:passingTest:]_block_invoke + 26
18  com.apple.AppKit              	0x00007fff783d3ebb -[NSApplication enumerateWindowsWithOptions:usingBlock:] + 340
19  com.apple.AppKit              	0x00007fff783d3c92 -[NSApplication _findWindowWithOptions:passingTest:] + 152
20  com.apple.AppKit              	0x00007fff77f980a9 -[NSApplication makeWindowsPerform:inOrder:] + 257
21  com.apple.AppKit              	0x00007fff78344dad -[NSApplication _reactToScreenInvalidationImmediately:] + 66
22  com.apple.AppKit              	0x00007fff78344d37 __44-[NSApplication _reactToScreenInvalidation:]_block_invoke + 59
23  com.apple.CoreFoundation      	0x00007fff7a42830c __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
24  com.apple.CoreFoundation      	0x00007fff7a409384 __CFRunLoopDoBlocks + 356
25  com.apple.CoreFoundation      	0x00007fff7a408ec6 __CFRunLoopRun + 1878
26  com.apple.CoreFoundation      	0x00007fff7a408514 CFRunLoopRunSpecific + 420
27  com.apple.HIToolbox           	0x00007fff799992ac RunCurrentEventLoopInMode + 240
28  com.apple.HIToolbox           	0x00007fff799990e1 ReceiveNextEventCommon + 432
29  com.apple.HIToolbox           	0x00007fff79998f16 _BlockUntilNextEventMatchingListInModeWithFilter + 71
30  com.apple.AppKit              	0x00007fff77f9b6cd _DPSNextEvent + 1093
31  com.apple.AppKit              	0x00007fff78712830 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 2730
32  com.google.Chrome.framework   	0x00000001109041c0 0x10f4b1000 + 21311936
33  com.google.Chrome.framework   	0x0000000110d2bfda 0x10f4b1000 + 25669594
34  com.google.Chrome.framework   	0x00000001109040f9 0x10f4b1000 + 21311737
35  com.apple.AppKit              	0x00007fff77f9006d -[NSApplication run] + 926
36  com.google.Chrome.framework   	0x0000000110d3a1be 0x10f4b1000 + 25727422
37  com.google.Chrome.framework   	0x0000000110d3985c 0x10f4b1000 + 25725020
38  com.google.Chrome.framework   	0x0000000110d54e53 0x10f4b1000 + 25837139
39  com.google.Chrome.framework   	0x0000000110909855 0x10f4b1000 + 21334101
40  com.google.Chrome.framework   	0x0000000110163dc4 0x10f4b1000 + 13315524
41  com.google.Chrome.framework   	0x00000001101664f2 0x10f4b1000 + 13325554
42  com.google.Chrome.framework   	0x000000011015fefc 0x10f4b1000 + 13299452
43  com.google.Chrome.framework   	0x00000001108c177d 0x10f4b1000 + 21038973
44  com.google.Chrome.framework   	0x00000001108c0a16 0x10f4b1000 + 21035542
45  com.google.Chrome.framework   	0x000000010f4b3cac ChromeMain + 60
46  com.google.Chrome             	0x000000010f437d9a main + 522
47  libdyld.dylib                 	0x00007fff8f670255 start + 1v

It looks like that release was cut before the merge. Infra folks, what's the right way forward?

Yes, this was only merged yesterday so it hasn't made it into a released M56. It will be in the next build.

[Automated comment] Request affecting a post-stable build (M55), manual review required.

This crash no more seen on latest beta version 	56.0.2924.28 released 5 days ago

Link to the builds:
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27-%5B_NSTableRowHeightStorage%20numberOfRows%5D%27#samplereports:5,productversion:1000

Issue 675546 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/145b27b6e14ae6dd557c823e39680d62f16b1cf3

commit 145b27b6e14ae6dd557c823e39680d62f16b1cf3
Author: sdy <sdy@chromium.org>
Date: Thu Jan 12 00:45:55 2017

Add a test for our workaround of an AppKit crash in SFChooseIdentityPanel.

BUG= 679992 , 653093 

Review-Url: https://codereview.chromium.org/2624273003
Cr-Commit-Position: refs/heads/master@{#443090}

[modify] https://crrev.com/145b27b6e14ae6dd557c823e39680d62f16b1cf3/chrome/browser/ui/cocoa/ssl_client_certificate_selector_cocoa_browsertest.mm

 Issue 680933  has been merged into this issue.

We are not planning any further M55 stable releases.

Issue 687163 has been merged into this issue.