Your net-log doesn't include accessing that server. Perhaps you attached the wrong one?

I had done a quick search for webmail.usmc.mil in the JSON and had found it so I assumed it contained the right traffic. I'll loop back with the customer to get a fuller log. Any hints in particular to scan for to ensure it's the right capture?

Use chrome://net-internals and search on the Events tab?

(You found it in the file because there was an already connected, idle socket to that server - meaning it didn't actually show the connection attempt)

Attaching more fleshed out capture. Hopefully this incorporates the needed traffic.

Deleted: net-internals-log.json 1.3 MB

net-internals-log.json 1.3 MB View Download

Another instance of .mil not working. Attached logs for sms.army.mil.

After our latest update, the only reports of failure are coming from .mil domains. Possible that server configurations are not playing nicely with Chrome.

Deleted: net-internals-log.sms.army.mil.json 951 KB

net-internals-log.sms.army.mil.json 951 KB View Download

+Sean, any background on .mil servers where we might be hitting a wall with Chrome?

webmail.us.usmc.mil -- Are we sure that the endpoint for this one is configured correctly?  The server is (at the TLS layer) prompting for client certificates, but is doing so only via root certs.  So if the user isn't prompted to select a certificate but is seeing a 403, it sounds more like the chain simply isn't completing and the endpoint doesn't know it could send one of its smartcard certs.  I get a 500 from the server, after being prompted to select a cert from my card.

www.sms.army.mil -- This one is weird.  Connections from PC / Mac using Chrome work fine, but ChromeOS tank as described.  It is, however, working really hard to make SSL "nice" for most users, so there's some sort of redirection going on.  I don't suppose there's a magic CLI flag for logging the entire TLS negotiation [more detail than normal net-internals]?

Yeah, webmail.us.usmc.mil looks like a client problem and is probably working as intended as far as Chrome is concerned. I'm guessing the intermediates aren't installed on the clients. If we don't know how to reach the names the server sends, there's nothing that can be done.

www.sms.army.mil is happening on a renego, so it's hard to tell what's going on. I do not believe there is currently any option that will log that plaintext. (There may be something to be said for hooking into BoringSSL's msg_callback.) Though the server is sending us an unknown_ca alert and we're only sending one certificate, so I'm guessing this is  issue #579480  again.

Confirmed that user hadn't imported proper root certs for usmc.mil. Running those tests again.


If we can figure a way to collect some more interesting logs from the user for sms.army.mil, please let us know.

I am willing to provide more logs. I only need instructions on how to do it.

For SMS.army.mil

Could you get a log of it working on other operating systems? If that sends intermediate certificates, that would give evidence this is the difference.

Also, this mentions a smartcard. Is this using the Chrome OS smartcard extensions API? I see one of the net-internals logs mentions a US DoD CACKey extension.

If the log of it working confirms, that would suggest this is similar to  issue #579480 , but not quite the same. The extensions API doesn't even let the smartcard provide a chain and it seems sanest to make that the extension's job, rather than have us weave the non-leaf certificates out of the store. (Ryan?)
https://developer.chrome.com/extensions/certificateProvider#type-CertificateInfo

I don't have a log offhand (or the ability to generate it ATM), but can confirm that the site does work in Chrome on other platforms; failing only on Linux / ChromeOS.  I'm going to try an A/B build of Chromium on a working platform to confirm, unless you're willing to say this is a smoking gun in which case I'll abort this really slow compile :).

I get the idea about the extension doing it, but will offer that this could lead to a lot of questionable extensions for all of the use-cases out there during a notional transition period to a post-chain-sending world.  That is, all of the complainants in the original bug thread [with dozens / hundreds of intermediates] might choose to write their own middleware.  That would leave me concerned about (1) code quality / security in each (2) user perception that they should trust any extension which advertises middleware support for their use case and (3) difficulty for users having more than one card - do they need to switch out their middleware to go from a banking to government card?

Frankly for DoD, this is easy enough to fix, though may be a hassle for the middleware developers.  For USG it's probably manageable if the extension can call back into the larger truststore for support.  For everyone else -- "it's complicated" :).

If you can confirm the site works on other platforms, you should be able to generate the log... you do it entirely in Chrome:
https://dev.chromium.org/for-testers/providing-network-details

The reason for making it the extension's responsibility is that, on CrOS, the extension/smartcard-driver is the one supplying the leaf, not the trust store. In that case, it seems pretty silly to expect half of the chain to be meandered through the trust store and half from the extension/smartcard-driver. It's also the only option where getting the intermediates is at all sound. All the path-building stuff has serious problems.

If OS cert stores grouped identities by {key, leaf, intermediates} so we could just ship them all in one bunch, that would solve a lot of problems. Sadly, they do not. But since we get to define what an identity is in CrOS by way of the extensions API, we should get it right.

Of course, by far the best option is for the server to not require the intermediates be sent. OS APIs are far too variable and finicky to rely on.

Oh, logging's easy!  It's just that my smartcard reader has a different card than will be accepted by the site in it right now and I need to keep it active for a couple more hours.  I tested last night and it was working; just didn't think that log would be sufficient.  Can definitely do it a bit later.

So I sort of understand the point about the middleware providing the leaf, but in this particular case, which layer is building the path which informs the user that their cert chains back to the root requested and allows them to attempt authentication to the site?  If it's the middleware, I completely get it.  It just seems a pretty significant change that the middleware would take on this responsibility on all platforms, and one requiring a lot of coordination to avoid breaking more sites.

No question all sites should offer better support, but then that's at least partly a server OS / container coding problem, right?  I'm not aware of a single platform which allows me to configure a truststore which I don't wholly advertise for TLS client authentication -- certainly none of the big ones do -- despite the fact that it would be ideal to (notionally, idealized PKI) advertise just the roots I accept to users while retaining all of the intermediates to validate what they send back.  And I agree with old discussions that sending dozens or hundreds of intermediates isn't good practice for routine use - it's a hard problem.

Will generate and send log as soon as the current process completes.

Log attached for a successful authentication to www.sms.army.mil on OSX.

Also did manage to disable sending the intermediates on OSX and it gave the typical ERR_BAD_SSL_CLIENT_AUTH_CERT output.  I'd send those logs for posterity if I could figure out why my net-internals file dialog fails :/.

Deleted: net-internals-log (6).json 631 KB

net-internals-log (6).json 631 KB View Download

Thanks! That confirms what's going on then.

David: This is fixed with the intermediate fixes, right?

I'm not sure. This is using the CrOS custom key stuff, which adds another layer of messiness. I don't know if those interact with NSS (or if they should...).

Probably Sean or Jamie can best answer this as they can simply re-run the original steps on Chrome OS latest version again?

Independent of whether it's working on accident right now, there is the more important question of whether we need to migrate to a world where it does *not* work and instead the mechanism is by way of the certProvider implementation (to which I think the answer is a strong yes, so we're not further constrained on NSS).

dskaram: Did you ever determine if this was since resolved?

rsleevi: Are you keeping track anywhere of places where we rely on NSS in CrOS but shouldn't? This is one of them. Because certProvider does not supply intermediates, we rely on the NSS-based path-building using the "pile of in-memory intermediates" logic not only when the certificates are stored in chapsd, but also when they come from certProvider. That seems not great. Ideally certProvider would provide the chain it wants, but the API doesn't have that.

I'm not sure I agree with the proposed design solution, but I agree, we should see if this was resolved :)

That is, I'm not sure if the world is better if extensions start doing chain/path-building, and I stand by the statement servers shouldn't rely on it. The stuff emaxx is looking at re: intermediates for client certs I think is closely related - which would at least aim to remove the implicit dependency

FWIW, I can say anecdotally that there are sites which Chrome simply can no longer be used for due to the lack of chain certificates being included with the authentication. I'm not on CrOS frequently enough anymore to say whether or not it is working around this where the normal browser isn't, but we absolutely run into cases where we have to refer users to other browsers in order to be able to get into certain sites (and that's just at the network layer; let's not discuss some sites' affinity for 'compatibility view').

We don't presently keep a list of sites affected by this issue, as from an end user perspective - given how frequently truth changes for some sites - it presents as much opportunity for harm as good. But if it would facilitate discussion around "building better browsers" we can begin track and share as the data grows. Just remember that ours is a low-use case and that you may have better data already from telemetry about failed TLS connections.

Alas, client certificates are rarely-used and their problems are *EXTREMELY* deployment-specific and sensitive both to problems in the browser and problems in the client certificate deployment. Problems will not going to show up in our telemetry. There's a reason every browser developer hates the feature. :-)

We're going to need a clear report from you to diagnose any issues you're running into.

Preaching to the choir on the unholy trinity of PKI-Client-Server diagnostics :).

I'll ask the teams to begin noting each of these cases and see what kind of list we can provide. More to follow here.

dskaram: Any other customers in the current repertoire who might also be able to assist? I may be reaching, but assume that the SCC still gets a lot of questions around this.

friendly ping, dskaram@ can you provide the info requested in c#27 and c#28?

I don't have any at the moment. When we run into this in the field again, we can ping this bug back up.

Please feel free to reopen when you're able to collect the requested information.