I'm happy to email our net trace - but it's 19 MB zipped.

Thanks for filing the issue!

The issue seems to be out of scope for triaging from our end as this speaks about SSL connections with client certificate and requests, hence adding label "TE-NeedsTriageHelp" and requesting some one from Internals>Network>Certificate team to have a look into it and help in triaging it further.

Adding a pcap which captures this issue.

Deleted: chrome_os_x_10.13.4_hang_loading_news.google.com.pcapng 29.1 KB

chrome_os_x_10.13.4_hang_loading_news.google.com.pcapng 29.1 KB Download

Net export attached

Deleted: chrome-net-export-log.json 799 KB

chrome-net-export-log.json 799 KB View Download

I'm not seeing any client certificate requests in that log. I'm guessing that happened before you started capturing? I do, however, see a bunch of stalled CERT_VERIFIER_JOB events. Likely something about the client certificate path is hanging the certificate verifier.

We don't put certificate verification and the client certificate signing on the same threads, but there is a bit of locking, both in our code and Apple's. On our side, we externally lock a number of accesses to Security.framework, to work around historical threading bugs in Apple's code.
https://cs.chromium.org/search/?q=GetMacSecurityServicesLock&type=cs

On Apple's side, I've seen Keychain lock up before.

That said, whether the lock is on our end or Apple's, the root cause is that the client certificate path is hanging, which is probably an Apple bug. (Unless we have some truly amazing macOS-version-specific bug where we take that lock without releasing it. That seems unlikely.)

Some things that might help narrow it down, if you can get them. (I understand it's intermittent.)

1. You mentioned that rebooting or putting the machine to sleep solves it. Does that mean that restarting Chrome is not sufficient? (If so, that definitely sounds like the global Keychain service locked up.)

2. When this problem occurs, does Safari also hang up, or is it just Chrome? (More evidence for whether it's global or process-local.)

3. When you say "Visit a site that requires a client certificate stored in a local keychain on MacOS 10.13.4 / That request will hang", how far does that request get before it hangs? (I.e. does the hang happen when we look for matching client certificates or when we go and sign with the key?)

4. If you could attach a net-internals that includes the client certificate request, that might be useful.

5. https://www.chromium.org/for-testers/bug-reporting-guidelines/hanging-tabs contains instructions to forcibly crash a hanging process in a way that should generate a crash report. That will give us stack traces of what various threads are up to, which will help figure out exactly what operation(s) are stuck. When it says to open Chrome's task manager to find the process ID, use the Browser process. Or, if that's not responding, the macOS task manager will work fine too.

Thanks!

Thanks for the response! 

1. Restarting Chrome will allow new SSL connections successfully, as will sleeping. The client certificate requests will not succeed until a full restart of the machine.

2. Safari seems completely unaffected.

3. It seems to eventually get reset - we'll try and get some deeper traces with Burp or Charles on this.

4 and 5. Attached. I apologize if these don't have relevant traffic, I'm playing a bit of telephone with users that are affected.

Deleted: chrome-net-export-log-jp.json 296 KB

chrome-net-export-log-jp.json 296 KB View Download

Deleted: cda335cd-01b4-4306-b219-217b6db64675.dmp 278 KB

cda335cd-01b4-4306-b219-217b6db64675.dmp 278 KB Download

Deleted: c6210892-c61c-4cd7-be19-428e9d6052fd.dmp 291 KB

c6210892-c61c-4cd7-be19-428e9d6052fd.dmp 291 KB Download

Deleted: chrome-net-export-log-jb.json 2.7 MB

chrome-net-export-log-jb.json 2.7 MB View Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Once it does go into this hang it hangs all other SSL connections not just client certificate connections.

With some further research we have determined internally that the resource contention is resolved (when the bug occurs) by removing the Yubikey U2F token from the machine. For any hung connections removing the Yubikey instantly causes those connections to succeed, likewise the client certificate negotiation immediately completes once the Yubikey is removed. 

Which Yubikey device are you using? And do you have any Yubikey middleware involved (namely, CCID support)?

Several - Yubikey 4 nano, Yubikey 4c, Yubikey 4c nano. CCID is disabled (We only enable U2F) 

I'm going to tag this as WebAuthN related, for visibility. It's unclear if Security.framework is getting jammed up (more aptly, securityd), if Chrome is doing something on the IO thread related to WebAuthN, or something else.

If you can reproduce it reliably, can you include an Activity Monitor sample (click the Gear -> Sample Process) for Chrome? And see if securityd / securityd_service / trustd is doing anything exciting?

WebAuthN is behind a flag in 65; do you have it enabled?

We do not have WebAuthN enabled on machines that are exhibiting this issue.

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Resetting NF for the Activity Monitor traces

Do you have a crash ID for those crashes you attached in comment #8? Alternatively, +eroman, do you know how one would correlate/upload those to our crash reporting stack?They're macOS, so WinDbg will presumably not like them.

Unfortunately I don't believe we have the crash IDs anymore. 

Attached are two activity monitor samples. One is from the client cert negotiation and the other is from a new SSL connection after the client cert notification.

Deleted: Sample-of-Google-Chrome-mid-hanging.txt 173 KB

Sample-of-Google-Chrome-mid-hanging.txt 173 KB View Download

Deleted: Sample-of-Google-Chrome-hanging-load-new-page.txt 212 KB

Sample-of-Google-Chrome-hanging-load-new-page.txt 212 KB View Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Thanks.

From the sample of hanging load-new-page, we see the TaskSchedulerForegroundBlockingWorker threads in the CertVerifyProc blocking on the GetMacSecurityServicesLock(). The Platform Key thread is in GetClientCertsOnBackgroundThread, which holds the lock.

That thread is calling into Keychain via SecItemCopyMatching, which is dispatching into SecTokenCreate as part of SecItemResultCopyPrepared, which is waiting for TKTokenCreateWithConnection to synchronously respond.

This supports the theory that there's a CCID interaction going on (or other form of token), as macOS would not otherwise be calling into TokenKit directly. This also explains why removing the Yubikey resolves this - this would cause TKTokenCreateWithConnection to fail, thus releasing the lock, thus allowing certificate verification to resume.

We have the lock in place because on various versions of macOS, the thread-safe invariants were violated within Apple code, causing UAFs, crashes, or hangs. One possible item is to revisit whether that's still necessary in a 10.9+ world, but that'd be for a future Chrome release.

So we need to figure out why TKTokenCreateWithConnection() is hanging (is it waiting for Chrome to do something, or for the token?) in the near-term, and it's definitely releated to (some) WebAuthN devices.

Is there any chance that you've enabled the PIV mode for native login - as per https://www.yubico.com/why-yubico/for-business/computer-login/mac-os-login/ ?

Oh, and activity traces of any of the following daemons:
trustd
secd
identityservicesd
keychaind

I forget which one hosts the TKTokenKit XPC server - and it may be none of them - but that may help trace through further.

Some of the affected users do in fact have either PIV or CCID enabled (some have both). 

I'm now testing with a Yubikey that I know for a fact has all of those functions disabled to see if I can get the behavior to trigger again. 

Will report back once I've successfully repro'd. I appreciate all of the eyes on this!

Oh, and sleeping would resolve this because it'd send a USB power change notification, which would cause the device to register as unplugged ('effectively'... USB power states and all), hence causing the call to fail - until the next time.

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot