New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 105513 link

Starred by 31 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2012
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug

Restricted
  • Only users with EditIssue permission may comment.



Sign in to add a comment

Crash with __THE_SYSTEM_HAS_NO_PORT_SETS_AVAILABLE__ on the stack

Project Member Reported by rohitrao@chromium.org, Nov 28 2011

Issue description

Chrome Version       : 17.0.949.0
OS Version: OS X 10.7.2

Canary has been super-crashy for me today.  Looking at the Apple Crash Reporter dumps, I've had six crashes in the last eight hours.

Breakpad doesn't appear to be firing for these crashes, as I have nothing in chrome://crashes or in the Crash Reports directory.

Five of the six Apple crash reporter logs show a crash in a thread with no Chromium code on the stack, but I'm not sure how trustworthy these reports are:

Thread 31 Crashed:: Dispatch queue: NSPersistentUI I/O
0   com.apple.CoreFoundation      	0x97df61e7 __THE_SYSTEM_HAS_NO_PORTS_AVAILABLE__ + 7
1   com.apple.CoreFoundation      	0x97d265bb __CFRunLoopCreate + 219
2   com.apple.CoreFoundation      	0x97d2637f _CFRunLoopGet0 + 591
3   com.apple.CoreFoundation      	0x97d217e2 CFRunLoopGetCurrent + 66
4   com.apple.CoreServicesInternal	0x97ed52b7 _FSURLEndResourcePropertyCacheAccess + 61
5   com.apple.CoreFoundation      	0x97d41579 __CFURLEndResourcePropertyCacheAccess + 73
6   com.apple.CoreFoundation      	0x97d3f5a5 -[NSURL getResourceValue:forKey:error:] + 85
7   com.apple.Foundation          	0x9153faf7 -[NSURL(NSURLPathUtilities) URLByAppendingPathComponent:] + 266
8   com.apple.AppKit              	0x93a35ee5 -[NSPersistentUIManager persistentStateFileURL] + 62
9   com.apple.AppKit              	0x93a35cf4 -[NSPersistentUIManager openPersistentStateFileForReading:] + 86
10  com.apple.AppKit              	0x942921bb -[NSPersistentUIManager writeRecords:withWindowInfos:flushingStaleData:] + 57
11  com.apple.AppKit              	0x93a35c31 __-[NSPersistentUIManager flushAllChangesOptionallyWaitingUntilDone:updatingSnapshots:]_block_invoke_4 + 309
12  libdispatch.dylib             	0x9138fe11 _dispatch_call_block_and_release + 15
13  libdispatch.dylib             	0x91391797 _dispatch_queue_drain + 224
14  libdispatch.dylib             	0x9139163c _dispatch_queue_invoke + 47
15  libdispatch.dylib             	0x91390e44 _dispatch_worker_thread2 + 187
16  libsystem_c.dylib             	0x90c44b24 _pthread_wqthread + 346
17  libsystem_c.dylib             	0x90c466fe start_wqthread + 30



The sixth stack is:

Thread 45 Crashed:: Dispatch queue: com.apple.root.default-priority
0   com.apple.CoreFoundation      	0x97df61b7 __THE_SYSTEM_HAS_NO_PORT_SETS_AVAILABLE__ + 7
1   com.apple.CoreFoundation      	0x97d20b6c __CFRunLoopFindMode + 444
2   com.apple.CoreFoundation      	0x97d21919 CFRunLoopAddSource + 297
3   com.apple.Foundation          	0x9154af38 -[NSMachPort scheduleInRunLoop:forMode:] + 158
4   com.apple.Foundation          	0x9154ad04 -[NSRunLoop(NSRunLoop) _addPort:forMode:] + 452
5   com.apple.Foundation          	0x9154ab38 -[NSRunLoop(NSRunLoop) addPort:forMode:] + 272
6   com.apple.Foundation          	0x9154aa21 -[NSPort addConnection:toRunLoop:forMode:] + 89
7   com.apple.Foundation          	0x9154a9ab -[NSMachPort addConnection:toRunLoop:forMode:] + 75
8   com.apple.Foundation          	0x9155ed08 -[NSConnection sendInvocation:internal:] + 2329
9   com.apple.Foundation          	0x9155e1cc -[NSDistantObject forwardInvocation:] + 270
10  com.apple.CoreFoundation      	0x97d7d20e ___forwarding___ + 894
11  com.apple.CoreFoundation      	0x97d7ce22 _CF_forwarding_prep_0 + 50
12  com.apple.AppKit              	0x93faaf3b -[NSSpellChecker _checkSpellingAndGrammarInString:range:enclosingRange:offset:types:options:orthography:inSpellDocumentWithTag:mutableResults:wordCount:] + 2498
13  com.apple.AppKit              	0x93ba6caf NSSpellCheckerCheckString + 9822
14  com.apple.AppKit              	0x93ba45f5 -[NSTextCheckingOperation main] + 156
15  com.apple.Foundation          	0x9158949f -[__NSOperationInternal start] + 1177
16  com.apple.Foundation          	0x91588fff -[NSOperation start] + 67
17  com.apple.Foundation          	0x9159d152 ____NSOQSchedule_block_invoke_2 + 135
18  libdispatch.dylib             	0x9138fe11 _dispatch_call_block_and_release + 15
19  libdispatch.dylib             	0x91390e70 _dispatch_worker_thread2 + 231
20  libsystem_c.dylib             	0x90c44b24 _pthread_wqthread + 346
21  libsystem_c.dylib             	0x90c466fe start_wqthread + 30

 
Uploading full crash dumps.

I also see the following lines in Console.app, generally correlated with my crashes:

11/27/11 9:01:44.661 PM [0x0-0x138b38a].com.google.GoogleTalkPluginD: DVFreeThread - CFMachPortCreateWithPort hack = 0xe72b80, fPowerNotifyPort= 0xe72500
11/27/11 9:01:44.661 PM [0x0-0x138b38a].com.google.GoogleTalkPluginD: DVFreeThread - CFMachPortCreateWithPort hack = 0x2560f50, fPowerNotifyPort= 0x25759b0
11/27/11 9:01:44.661 PM [0x0-0x138b38a].com.google.GoogleTalkPluginD: DVFreeThread - CFMachPortCreateWithPort hack = 0xe7bb40, fPowerNotifyPort= 0xe7af30
11/27/11 9:01:44.661 PM [0x0-0x138b38a].com.google.GoogleTalkPluginD: DVFreeThread - CFMachPortCreateWithPort hack = 0x256a120, fPowerNotifyPort= 0x2564660
11/27/11 9:01:44.661 PM [0x0-0x138b38a].com.google.GoogleTalkPluginD: DVFreeThread - CFMachPortCreateWithPort hack = 0x255f6d0, fPowerNotifyPort= 0x255aa70
11/27/11 9:01:44.979 PM [0x0-0x137b37a].com.google.Chrome.canary: DVFreeThread - CFMachPortCreateWithPort hack = 0x7966dce0, fPowerNotifyPort= 0x7968fd20
11/27/11 9:01:44.979 PM [0x0-0x137b37a].com.google.Chrome.canary: DVFreeThread - CFMachPortCreateWithPort hack = 0x796ae290, fPowerNotifyPort= 0x796acfb0
11/27/11 9:01:45.191 PM com.apple.launchd.peruser.13139: ([0x0-0x137b37a].com.google.Chrome.canary[48843]) Job appears to have crashed: Trace/BPT trap: 5
11/27/11 9:01:45.000 PM root: Application Quit: bundle_id: com.google.Chrome.canary version: 949.0 path: /Applications/Google Chrome Canary.app
11/27/11 9:01:45.342 PM com.googlecode.pymacadmin.crankd: INFO: Application Quit: bundle_id: com.google.Chrome.canary version: 949.0 path: /Applications/Google Chrome Canary.app
11/27/11 9:01:47.076 PM ReportCrash: Saved crash report for Google Chrome Canary[48843] version 17.0.949.0 (949.0) to /Users/rohitrao/Library/Logs/DiagnosticReports/Google Chrome Canary_2011-11-27-210147_rohitrao-macbookpro.crash

Google Chrome Canary_2011-11-27-210147_rohitrao-macbookpro.crash
62.9 KB View Download
Google Chrome Canary_2011-11-27-184947_rohitrao-macbookpro.crash
73.1 KB View Download
Robert thinks something on my laptop is hosed, based on "__THE_SYSTEM_HAS_NO_PORT_SETS_AVAILABLE__."  Gonna try restarting to see if that helps.

Comment 3 by k...@google.com, Nov 28 2011

Status: Invalid
Reopen if you think it's real.
Status: Untriaged
Reopening because at least three other people are seeing similar crashes.  All of us are running Lion, and there may be a correlation with laptops and with the Google image.

Restarting fixes the problem for a few days, but eventually I started seeing the same __THE_SYSTEM_HAS_NO_PORT_SETS_AVAILABLE__ crash again.
Summary: Crash with __THE_SYSTEM_HAS_NO_PORT_SETS_AVAILABLE__ on the stack
I've also seen reports of this on both Chrome Canary (m17 branch) and Chrome Stable (m15 branch).
Cc: thakis@chromium.org mark@chromium.org sh...@chromium.org rsesek@chromium.org
Labels: -Pri-2 Pri-1
Owner: rohitrao@chromium.org
Status: Assigned
I spent some time looking at this today and have created a StartupItem that runs a DTrace script, which will print the execname and pid of each process that make some kind of mach_port* call. It will also aggregate all function calls into various mach_port management routines. When DTrace exits (on shutdown), it will print the aggregation results for these. Combining these two data sets may hopefully reveal any source of port leakage.

Other things to note: the __THE_SYSTEM_HAS_NO_PORT{S,_SETS}__AVAILABLE_ calls pass the kern_result_t from the Mach call to that function. If caught in the debugger, inspecting that value may be useful. The code that calls into those is in 10.7-opensource/CF/CFRunLoop.c. The corresponding code in Mach is in xnu/osfmk/ipc/mach_port.c and xnu/osfmk/ipc/ipc_right.c.

To install my startup item:

$ unzip MachPortCounter.zip
$ sudo chown -R root:wheel MachPortCounter/
$ sudo mv MachPortCounter /Library/StartupItems/

And then reboot. It will start streaming a log file to /mach_port_counter.log, and on next reboot, it should have aggregations.

When you get in this state, try to attach in gdb and see if you can't figure out the kern_result_t. Then reboot and copy the log file somewhere else and send it off to me. I'll write a script to slice the data into something maybe useful.
MachPortCounter.zip
3.4 KB Download

Comment 8 by jyass...@gmail.com, Dec 16 2011

I finally got the NO_PORTS crash again. Here's the crash report, and /mach_port_counter.log.
Google Chrome_2011-12-16-002620_jyasskin-macbookpro.crash
74.3 KB View Download
mach_port_counter.log
522 KB View Download

Comment 9 by rsesek@chromium.org, Dec 21 2011

I took a look at the log and for some reason DTrace didn't print the aggregate information into the log. It generated the list of all the pids, but not the actual allocation and deallcation counts. Bummer.
Cc: mmenke@chromium.org dominich@chromium.org
 Issue 108914  has been merged into this issue.
Cc: -dominich@chromium.org -mmenke@chromium.org

Comment 12 by j...@amusive.com, Jan 3 2012

I've been experiencing this crash a lot.  I don't know much (anything) about mach ports, but it looks like chrome is seriously leaking them, leading up to this if your browser session is long lasting.

When I first started chrome, according to activity monitor it had ~500 ports reserved.  This morning -- with only a small number of open pages (3) across 2 browser windows, it has 1,628 reserved.  I noticed this number will significantly rise upon loading a seemingly complex page (I noticed it especially on Google+ or Facebook) and typically steady-state at +30 from the original count, just prior to loading that page.

No idea if any of that helps or matters, but if you'd like me to run a trace or anything I'd be happy to help; I seem to crash once or twice a day because of this.
fyi - I've been seeing this quite a lot, starting 12-31 with version 17.0.963.12

I can pretty much get it to crash by restoring my tabs.  If there's anything I can do to pinpoint the error I'm happy to help...

Comment 14 Deleted

Comment 15 by Deleted ...@, Jan 4 2012

Here's a bunch of crash reports

(oddly enough I'm getting crashes in the file upload dialog now, every time...)

Crashes.zip
5.3 KB Download
plinder: Can you install the MachPortCounter in comment 7? The only modification on the instructions I'd have is:

Before shutting down when you're experiencing the crash, send SIGINT to the dtrace process running on the system. That'd look like:

$ ps ax | grep dtrace
11715 s001  R+     0:00.00 dtrace -s /Library/StartupItems/MachPortCounter/mach_ports.d
$ kill -SIGINT 11715

I think when the system shuts down, dtrace doesn't have adequate time to flush the aggregations log. Interrupting it (Ctrl+C) before halting seems to be better.

It does sound like this is something Chrome (or a framework which it uses) is leaking. So this may not be too helpful in the end, but it'd be good to verify that.

If you're sure that this started 17.0.963.12, then maybe you can help us bisect. Do you know which version you were running previously?
FYI: That ZIP file contain aliases to files, not the actual crash reports themselves. But at this point, crash reports aren't too helpful. Where it's crashing doesn't provide any diagnostic information as to what's leaking ports. If you have a consistent reproduction case, then maybe we can have someone live debug on your machine.
@rdsmith @jim @plindner Are you guys running Snow Leopard or Lion?

Comment 19 by Deleted ...@, Jan 4 2012

Running Lion..  Now running the MachPortCounter... But now seeing crashes like this when I try to upload a file to this very form:

Thread 0 Crashed:: CrBrowserMain  Dispatch queue: com.apple.main-thread
0   libobjc.A.dylib               	0x91e27e49 _objc_trap + 3
1   libobjc.A.dylib               	0x91e27ef0 _objc_fatal + 99
2   libobjc.A.dylib               	0x91e1eeab _collecting_in_critical + 290
3   libobjc.A.dylib               	0x91e1e429 _cache_collect + 43
4   libobjc.A.dylib               	0x91e1d087 _cache_fill + 582
5   libobjc.A.dylib               	0x91e1ce39 log_and_fill_cache + 128
6   libobjc.A.dylib               	0x91e1c70a lookUpMethod + 344
7   libobjc.A.dylib               	0x91e274bb _class_lookupMethodAndLoadCache + 40
8   libobjc.A.dylib               	0x91e1bd83 objc_msgSend + 83
9   com.apple.AppKit              	0x9142882a -[NSView(NSInternal) _updateTrackingAreas] + 685
10  com.apple.AppKit              	0x91a736c3 __-[NSView(NSInternal) _updateTrackingAreas]_block_invoke_1 + 33
11  com.apple.CoreFoundation      	0x99e0a470 __NSArrayEnumerate + 704
12  com.apple.CoreFoundation      	0x99e09fab -[NSArray enumerateObjectsWithOptions:usingBlock:] + 219
13  com.apple.CoreFoundation      	0x99e09ec2 -[NSArray enumerateObjectsUsingBlock:] + 178
14  com.apple.AppKit              	0x91428ad3 -[NSView(NSInternal) _updateTrackingAreas] + 1366
15  com.apple.AppKit              	0x91a736c3 __-[NSView(NSInternal) _updateTrackingAreas]_block_invoke_1 + 33
16  com.apple.CoreFoundation      	0x99e0a470 __NSArrayEnumerate + 704
17  com.apple.CoreFoundation      	0x99e09fab -[NSArray enumerateObjectsWithOptions:usingBlock:] + 219
18  com.apple.CoreFoundation      	0x99e09ec2 -[NSArray enumerateObjectsUsingBlock:] + 178
19  com.apple.AppKit              	0x91428ad3 -[NSView(NSInternal) _updateTrackingAreas] + 1366
20  com.apple.AppKit              	0x91a736c3 __-[NSView(NSInternal) _updateTrackingAreas]_block_invoke_1 + 33
21  com.apple.CoreFoundation      	0x99e0a470 __NSArrayEnumerate + 704
22  com.apple.CoreFoundation      	0x99e09fab -[NSArray enumerateObjectsWithOptions:usingBlock:] + 219
23  com.apple.CoreFoundation      	0x99e09ec2 -[NSArray enumerateObjectsUsingBlock:] + 178
24  com.apple.AppKit              	0x91428ad3 -[NSView(NSInternal) _updateTrackingAreas] + 1366
25  com.apple.AppKit              	0x914283b8 __-[NSWindow _postInvalidCursorRects]_block_invoke_1 + 1135
26  com.apple.CoreFoundation      	0x99e0323d _runLoopObserverWithBlockContext + 29
27  com.apple.CoreFoundation      	0x99dcf7be __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 30
28  com.apple.CoreFoundation      	0x99dcf6fd __CFRunLoopDoObservers + 413
29  com.apple.CoreFoundation      	0x99da2094 __CFRunLoopRun + 1044
30  com.apple.CoreFoundation      	0x99da18ec CFRunLoopRunSpecific + 332
31  com.apple.CoreFoundation      	0x99da1798 CFRunLoopRunInMode + 120
32  com.apple.HIToolbox           	0x944d3a7f RunCurrentEventLoopInMode + 318
33  com.apple.HIToolbox           	0x944dad9b ReceiveNextEventCommon + 381
34  com.apple.HIToolbox           	0x944dac0a BlockUntilNextEventMatchingListInMode + 88
35  com.apple.AppKit              	0x9135d040 _DPSNextEvent + 678
36  com.apple.AppKit              	0x9135c8ab -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 113
37  com.apple.AppKit              	0x91358c22 -[NSApplication run] + 911
38  com.google.Chrome.framework   	0x008a4a4d ChromeMain + 8011469
39  com.google.Chrome.framework   	0x008a454c ChromeMain + 8010188
40  com.google.Chrome.framework   	0x008ce09d ChromeMain + 8181021
41  com.google.Chrome.framework   	0x0026b819 ChromeMain + 1485977
42  com.google.Chrome.framework   	0x01df3de2 ChromeMain + 30356066
43  com.google.Chrome.framework   	0x01df352f ChromeMain + 30353839
44  com.google.Chrome.framework   	0x008885fd ChromeMain + 7895677
45  com.google.Chrome.framework   	0x00100ba9 ChromeMain + 41
46  com.google.Chrome             	0x000f7f58 main + 24
47  com.google.Chrome             	0x000f7f16 0xf7000 + 3862


Thread 43 Crashed:
0   com.apple.CoreFoundation      	0x99da1b9c __CFRunLoopFindMode + 492
1   com.apple.CoreFoundation      	0x99da76be __CFRunLoopCreate + 478
2   com.apple.CoreFoundation      	0x99da737f _CFRunLoopGet0 + 591
3   com.apple.CoreFoundation      	0x99da27e2 CFRunLoopGetCurrent + 66
4   com.apple.Foundation          	0x90064aac __NSThread__main__ + 429
5   libsystem_c.dylib             	0x96a32ed9 _pthread_start + 335
6   libsystem_c.dylib             	0x96a366de thread_start + 34



Thread 36 Crashed:: Dispatch queue: NSPersistentUI I/O
0   com.apple.CoreFoundation      	0x99e771e7 __THE_SYSTEM_HAS_NO_PORTS_AVAILABLE__ + 7
1   com.apple.CoreFoundation      	0x99da75bb __CFRunLoopCreate + 219
2   com.apple.CoreFoundation      	0x99da737f _CFRunLoopGet0 + 591
3   com.apple.CoreFoundation      	0x99da27e2 CFRunLoopGetCurrent + 66
4   com.apple.CoreServicesInternal	0x99d562b7 _FSURLEndResourcePropertyCacheAccess + 61
5   com.apple.CoreFoundation      	0x99dc2579 __CFURLEndResourcePropertyCacheAccess + 73
6   com.apple.CoreFoundation      	0x99dc05a5 -[NSURL getResourceValue:forKey:error:] + 85
7   com.apple.Foundation          	0x90008af7 -[NSURL(NSURLPathUtilities) URLByAppendingPathComponent:] + 266
8   com.apple.AppKit              	0x913d8ee5 -[NSPersistentUIManager persistentStateFileURL] + 62
9   com.apple.AppKit              	0x913d8cf4 -[NSPersistentUIManager openPersistentStateFileForReading:] + 86
10  com.apple.AppKit              	0x91c351bb -[NSPersistentUIManager writeRecords:withWindowInfos:flushingStaleData:] + 57
11  com.apple.AppKit              	0x913d8c31 __-[NSPersistentUIManager flushAllChangesOptionallyWaitingUntilDone:updatingSnapshots:]_block_invoke_4 + 309
12  libdispatch.dylib             	0x929a5e11 _dispatch_call_block_and_release + 15
13  libdispatch.dylib             	0x929a7797 _dispatch_queue_drain + 224
14  libdispatch.dylib             	0x929a763c _dispatch_queue_invoke + 47
15  libdispatch.dylib             	0x929a6e44 _dispatch_worker_thread2 + 187
16  libsystem_c.dylib             	0x96a34b24 _pthread_wqthread + 346
17  libsystem_c.dylib             	0x96a366fe start_wqthread + 30


Comment 20 by Deleted ...@, Jan 4 2012

Okay, hit the port allocation errors a few times.  I didn't see anything spectacular in the mach_port_counter.log  (note that I did not reboot..)

Looking at the Activity Monitor I see Chrome with 424 ports, Shockwave Flash with 330 ports.

All of my most recent crashes look like the following stack trace:


Crashed Thread:  28  AudioThread


Thread 28 Crashed:: AudioThread
0   com.apple.CoreFoundation      	0x99e771e7 __THE_SYSTEM_HAS_NO_PORTS_AVAILABLE__ + 7
1   com.apple.CoreFoundation      	0x99da75bb __CFRunLoopCreate + 219
2   com.apple.CoreFoundation      	0x99da737f _CFRunLoopGet0 + 591
3   com.apple.CoreFoundation      	0x99da27e2 CFRunLoopGetCurrent + 66
4   com.apple.audio.CoreAudio     	0x9a1dda4f HALPropertyListener::Call(unsigned long, unsigned long, AudioObjectPropertyAddress const*) + 467
5   com.apple.audio.CoreAudio     	0x9a1dafd8 HALObject::PropertiesChanged(unsigned long, AudioObjectPropertyAddress const*) + 1258
6   com.apple.audio.CoreAudio     	0x9a19df9f HALSystem::CheckOutInstance() + 553
7   com.apple.audio.CoreAudio     	0x9a19dbe9 AudioObjectGetPropertyData + 201
8   com.google.Chrome.framework   	0x01065537 ChromeMain + 16685495
9   com.google.Chrome.framework   	0x010615a0 ChromeMain + 16669216
10  com.google.Chrome.framework   	0x010614a8 ChromeMain + 16668968
11  com.google.Chrome.framework   	0x01062173 ChromeMain + 16672243
12  com.google.Chrome.framework   	0x0105fbb3 ChromeMain + 16662579
13  com.google.Chrome.framework   	0x01061229 ChromeMain + 16668329
14  com.google.Chrome.framework   	0x00849586 ChromeMain + 8182278
15  com.google.Chrome.framework   	0x008496fc ChromeMain + 8182652
16  com.google.Chrome.framework   	0x008499cd ChromeMain + 8183373
17  com.google.Chrome.framework   	0x0084c31a ChromeMain + 8193946
18  com.google.Chrome.framework   	0x0084909d ChromeMain + 8181021
19  com.google.Chrome.framework   	0x00868a51 ChromeMain + 8310481
20  com.google.Chrome.framework   	0x00868ad7 ChromeMain + 8310615
21  com.google.Chrome.framework   	0x00867d4a ChromeMain + 8307146
22  libsystem_c.dylib             	0x96a32ed9 _pthread_start + 335
23  libsystem_c.dylib             	0x96a366de thread_start + 34

mach_port_counter.log
361 KB View Download
Robert can confirm, but I think the port counter script is only useful if you install it as a startup item (so it runs automatically on startup) and then reboot.  Now that your system is already in a bad state, the script probably won't print out anything useful.
I wonder if comment #19 could be a Lion-specific version of  issue 94658 .

Can we measure our own port/portset usage?  I could work up some code to histogram things on a timer or something, so that we can see if we're seeing spiky behavior.

Comment 23 by j...@amusive.com, Jan 4 2012

@rohitrao this is on Lion.
@rohitrao: Lion for me as well.

Issue 107820 has been merged into this issue.
If you're seeing this, when you get into this crashy state, please open /Applications/Utilities/Activity Monitor.app. Make sure you have the process list open (Window --> Activity Monitor). Then, enable View --> Columns --> # Ports. Sort the activity monitor by the number of ports descending and please screen shot the window and attach it to this bug. Thanks!

Comment 27 by j...@amusive.com, Jan 6 2012

rsesek: here you go, I feel like my chrome is about to crash and it's at ~3,000 ports.  I saw it yesterday at ~4,500 though, just shy of when it did crash.
Screen Shot 2012-01-06 at 2.42.42 PM.png
96.2 KB View Download
jim: When this happens, is there system-wide instability? If you quit Chrome, do things go back to normal? If you quit Chrome, does Chrome work again for some period of time, or is a full system restart necessary?
Also, how many tabs do you have open and for how long has Chrome been running?

Comment 30 by j...@amusive.com, Jan 6 2012

rsesek: no system-wide instability that I've noticed.  Maybe a slight sluggishness, but nothing extremely noticeable outside of Chrome.

Chrome itself, though, seems to slow down.  It seems to have a 'feel' to when it's about to crash (and if I look at Activity Monitor, I notice high port usage).

Just quitting & restarting chrome makes things work fine again, for a while. Generally upon starting up, chrome will be at ~500 ports.  It certainly has the feel of an over-time leak.

Comment 31 by j...@amusive.com, Jan 6 2012

Tabwise, right now I have two windows open logged in to two different accounts (work and personal).  Personal has 2 tabs, work has 10.  Though I've had it crash when I've had very few tabs open (~2 each window).
When you say that things work fine again, "for a while," is a "while" measured in minutes, hours, or days?

Comment 33 by j...@amusive.com, Jan 6 2012

I think chrome generally gets crashy after ~10-15 hours.  Sometimes it happens quicker, but generally I get several hours of bliss followed by a few hours of waiting for chrome to crash.

I generally have at least one crash a day, unless I restart chrome at some point.
Cc: -rsesek@chromium.org rohitrao@chromium.org
Owner: rsesek@chromium.org
What Rohit and I have both been able to observe is this:

Upon opening Chromium, there's about 250 ports. Opening a new tab will increase the port count by about five ports. Navigating to a complex page like Google+ brings the count up by at least 50. The troubling part is that closing these tabs doesn't ever bring the count back down to even close to the startup count.

That raises two questions:

1. What is "Ports" in Activity Monitor measuring, how is it measuring, and is it accurate?
2. Assuming (1) is an accurate count of Mach ports, why are we leaking so many on Lion?

I was running with --no-sandbox to make sure it wasn't a bug around something like  issue 87290 , and Rohit was running Chrome normally.

The next steps I'm going to be taking are to verify independently that we are in fact leaking Mach ports like Activity Monitor indicates. And then find a way to instrument this leakage in Chrome more granularly than the MachPortCounter startup item does (which measures port usage at the process-level).
I do believe we're leaking Mach ports. Activity Monitor relies on /usr/libexec/activitymonitord, a privileged helper that can talk to the kernel about all tasks on the system. It queries information about the system using libtop, and the information is updated via libtop_pinfo_update_mach_ports() in see http://www.opensource.apple.com/source/top/top-73/libtop.c. It counts Mach ports using the mach_port_names() call, which is the recommended way for counting ports and types.

To verify, I added a recurring delayed task right after the MessageLoop starts and it counts Mach ports using the same system call. Obviously, this number and the one from Activity Monitor reconciled. The next step is figuring out who and what is leaking.
Some initial analysis on our port usage: This is a list of all mach_port* calls made within Chromim.

  mach_port_get_attributes                                          1
  mach_port_names                                                   4
  mach_port_get_set_status                                          5
  mach_ports_register                                               6
  mach_port_destroy                                                 9
  mach_port_set_context                                            14
  mach_port_mod_refs                                               20
  mach_port_get_refs                                               24
  mach_port_set_attributes                                         31
  mach_port_insert_right                                           45
  mach_port_move_member                                            51
  mach_port_request_notification                                   51
  mach_port_allocate                                               92
  mach_port_type                                                  196
  mach_port_deallocate                                           2351
  mach_port_extract_member                                       3835
  mach_port_insert_member                                        3932

It looks like we have a lot more deallocations than explicit allocations: this means that some other task is sending us ports and we're somehow leaking them. That's not all that surprising. I looked at most of the calls to mach_port_deallocate(), and most come through _collecting_in_critical() in libobjc.A.dylb.

_collecting_in_critical() (objc4-493.9/runtime/objc-cache.m) gets all the threads for the current process' Mach task (which come in the form of ports), iterates over each and checks if the PC is in specific cache read functions. (Don't ask me why they took this approach.) They do this to avoid doing cache GC while reading. They then deallocate the list of thread ports, which accounts for the large number of calls, especially in thread-happy Chromium.

The number of explicit port allocations doesn't match the number that our task has associated with it, so there's more work to be done in finding out who's sending us these ports.
Two questions, while these numbers are still fresh in your mind:

1) How long a time period are these stats over? a minute? an hour?  I'm trying to put ~11000 total mach_port calls into context.  Were the stats collected all the way from first launch, or did you start collecting at some point after the process was launched?

2) What did Activity Monitor list as the number of open ports at the time these stats were collected?
1) These were collected over the period of a few minutes and navigating to cnn.com and plus.google.com and then closing each individually after a complete load.

2) The number of ports peaked at 354 with both pages open and went down to 292 after closing both.
FYI for those of you who are running the script from comment #7, we have all the data we need for now.  You can go ahead and uninstall the script by:
1) Deleting /Library/StartupItems/MachPortCounter
2) Deleting /mach_port_counter.log
3) Restarting

Per comment 35, we believe that Chrome is leaking mach ports, and we're now trying to hunt down which ports and why.
Histogramming our port usage by type reveals some interesting data:

There's almost 2x send ports than receive ports (Mach ports are unidirectional). We also have a steady but small increase in the number of dead ports that goes up by 1 or 2 every time a tab is opened and closed. I found some small port leakage because of the semantics of mach_thread_self() and fixed those: http://codereview.chromium.org/9169016/. But, in my observations, this didn't significantly reduce the number of ports.
Project Member

Comment 41 by bugdroid1@chromium.org, Jan 11 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=117127

------------------------------------------------------------------------
r117127 | rsesek@chromium.org | Tue Jan 10 16:15:53 PST 2012

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/base/threading/platform_thread_posix.cc?r1=117127&r2=117126&pathrev=117127
 M http://src.chromium.org/viewvc/chrome/trunk/src/base/logging.cc?r1=117127&r2=117126&pathrev=117127
 M http://src.chromium.org/viewvc/chrome/trunk/src/base/threading/platform_thread_unittest.cc?r1=117127&r2=117126&pathrev=117127

mach_port_deallocate() the result of mach_thread_self(), which obtains a port send right.

BUG= 105513 
TEST=none

Review URL: http://codereview.chromium.org/9169016
------------------------------------------------------------------------
As another data point, I see chrome crash quite often on OS X Lion, although I don't notice any sluggishness on the part of the browser before the crash.  It simply beachballs for about 3-4 seconds and then the browser window instantly disappears right out from under whatever action I was doing (typing in a text field, attaching a document, switching tab focus, etc...).  It never happens spontaneously, and always seems to follow a (seemingly) random user interaction with the browser.

Comment 43 by k...@google.com, Jan 12 2012

Labels: ReleaseBlock-Stable Mstone-17
Labels: Merge-Requested
After an exhaustive investigation and some helpful testing from Rohit, I'm going to call this tentatively fixed via the change at r117127 (comment 41). That fix will roll out to dev channel next week. If you're seeing this, please verify if you can.

Requesting merge because that fix is appropriate for all channels and will fix real Mach port leaks, hopefully including the one that causes this crash to occur.
Cc: meh...@chromium.org wtc@chromium.org rsesek@chromium.org
Issue 109514 has been merged into this issue.

Comment 46 by k...@google.com, Jan 17 2012

Labels: -Merge-Requested Merge-Approved
Today, please.
Project Member

Comment 47 by bugdroid1@chromium.org, Jan 17 2012

Labels: -merge-approved merge-merged-963
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=117935

------------------------------------------------------------------------
r117935 | rsesek@chromium.org | Tue Jan 17 11:34:58 PST 2012

Changed paths:
 M http://src.chromium.org/viewvc/chrome/branches/963/src/base/threading/platform_thread_posix.cc?r1=117935&r2=117934&pathrev=117935
 M http://src.chromium.org/viewvc/chrome/branches/963/src/base/logging.cc?r1=117935&r2=117934&pathrev=117935
 M http://src.chromium.org/viewvc/chrome/branches/963/src/base/threading/platform_thread_unittest.cc?r1=117935&r2=117934&pathrev=117935

Merge 117127 - mach_port_deallocate() the result of mach_thread_self(), which obtains a port send right.

BUG= 105513 
TEST=none

Review URL: http://codereview.chromium.org/9169016

TBR=rsesek@chromium.org
Review URL: https://chromiumcodereview.appspot.com/9178025
------------------------------------------------------------------------

Comment 48 by sh...@chromium.org, Jan 17 2012

Randomness: In the last stability triage, someone commented that they were seeing GDI leaks in Skia.  I'm not entirely certain what that means in real terms, but it made my brain wonder about the potential for port leaks WRT passing memory-mapped image buffers between processes.  [I'm really arm-waving here, though.  Just figured I'd mention it.]
mmap goes through the BSD part of the kernel, not the Mach part, so I think we're okay there. I also audited all instances of our use of mach* calls and only found the above issues.

Comment 50 by uke@google.com, Jan 18 2012

Are there any workaround for this issue at the moment?  Are other browsers affected?  

Comment 51 Deleted

No. But restarting Chrome regularly will prevent it from happening. The fix should be on dev and beta channels soon.

Comment 53 by dskl...@gmail.com, Jan 18 2012

Will it not be merged into the stable channel?
Stable channel doesn't appear to be affected.

Comment 55 by k...@google.com, Jan 18 2012

Status: Fixed
In that case -> Fixed to verify.

Comment 56 by dskl...@gmail.com, Jan 18 2012

I'm on stable (and lion) and my chrome is crashing all the time. I (and techstop) was hoping this was going to fix it for me.  
Please open a new bug and attach crash reports. We've received zero logs of this happening on 16.
 Issue 110609  has been merged into this issue.

Comment 60 by j...@amusive.com, Jan 19 2012

I haven't been crashed by this since the latest update.

Also just occasionally looking at port usage on Activity Monitor, I've never seen it change much off the start-up value of ~500.

Or, in other and more simpler words, thanks!
 Issue 110397  has been merged into this issue.

Comment 62 by ekay@google.com, Jan 21 2012

I was hoping that this (being in 17.0.963.38 unless I'm misunderstanding your build process) would fix my issues using Chrome with multiple profiles in Lion, but now instead of crashing outright, Chrome just becomes completely unresponsive after a bit of use. Can't force quit, can't put the computer to sleep, and actually end up needing to hard reboot.

I'm inexperienced in debugging behavior like this on Macs, but will be happy to provide any additional information that might help.
Project Member

Comment 63 by bugdroid1@chromium.org, Jan 24 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=118780

------------------------------------------------------------------------
r118780 | rsesek@chromium.org | Mon Jan 23 18:04:37 PST 2012

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/base/threading/platform_thread.h?r1=118780&r2=118779&pathrev=118780
 M http://src.chromium.org/viewvc/chrome/trunk/src/base/threading/platform_thread_posix.cc?r1=118780&r2=118779&pathrev=118780
 M http://src.chromium.org/viewvc/chrome/trunk/src/base/threading/worker_pool_posix_unittest.cc?r1=118780&r2=118779&pathrev=118780

[Mac] In PlatformThread::CurrentId(), use pthread_self() instead of mach_thread_self().

BUG= 105513 
TEST=none


Review URL: http://codereview.chromium.org/9281011
------------------------------------------------------------------------
Issue 110570 has been merged into this issue.
Project Member

Comment 67 by bugdroid1@chromium.org, Mar 8 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=125712

------------------------------------------------------------------------
r125712 | rsesek@chromium.org | Thu Mar 08 15:09:27 PST 2012

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/tools/perf/generate_perf.py?r1=125712&r2=125711&pathrev=125712

Add mach_ports to the list of perf tests.

BUG= 105513 
Review URL: https://chromiumcodereview.appspot.com/9651007
------------------------------------------------------------------------
Project Member

Comment 68 by bugdroid1@chromium.org, Oct 13 2012

Labels: Restrict-AddIssueComment-Commit
This issue has been closed for some time. No one will pay attention to new comments.
If you are seeing this bug or have new data, please click New Issue to start a new bug.
Project Member

Comment 69 by bugdroid1@chromium.org, Nov 14 2012

The following revision refers to this bug:
    http://goto.ext.google.com/viewvc/chrome-internal?view=rev&revision=21554

------------------------------------------------------------------------
r21554 | rsesek@google.com | 2012-01-26T20:58:03.313554Z

------------------------------------------------------------------------
Project Member

Comment 70 by bugdroid1@chromium.org, Nov 14 2012

The following revision refers to this bug:
    http://goto.ext.google.com/viewvc/chrome-internal?view=rev&revision=21581

------------------------------------------------------------------------
r21581 | rsesek@google.com | 2012-01-27T15:52:17.758854Z

------------------------------------------------------------------------
Project Member

Comment 71 by bugdroid1@chromium.org, Mar 10 2013

Labels: -Area-UI -Mstone-17 M-17 Cr-UI
Project Member

Comment 72 by bugdroid1@chromium.org, Mar 13 2013

Labels: -Restrict-AddIssueComment-Commit Restrict-AddIssueComment-EditIssue

Sign in to add a comment