Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Issue 579480 client-side certificates do not ship intermediate CA certificates anymore
Starred by 23 users Reported by wouter.v...@fedict.be, Jan 20 2016 Back to list
Status: Fixed
Owner:
Closed: Jul 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux, Chrome
Pri: 2
Type: Bug



Sign in to add a comment
UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0 Iceweasel/43.0.4

Example URL:
https://test.eid.belgium.be/

Steps to reproduce the problem:
1. Set up a root CA, sign an intermediate CA, and sign a client certificate with that intermediate CA
2. Configure a web server to allow the root CA as a trust anchor (do not hand it the intermediate CA)
3. Put the client certificate and the intermediate CA certificate on a smart card
4. Use modutil to hook the PKCS#11 for the smart card into chrome
5. Try to log on to the given website

What is the expected behavior?
Chrome sends the full certificate chain to the web server, and the authentication succeeds.

What went wrong?
Chrome only sends the client certificate rather than the full chain to the web server. As a result, the authentication fails.

The attached pcapng file, created with wireshark, contains an example of a failed session against the above URL (through a proxy).

Did this work before? Yes Chrome 39; possibly later, but 39 is the most recent version of which I am certain that it worked.

Chrome version: Google Chrome	47.0.2526.111 (Officiƫle build) (64-bits)  Channel: stable
OS Version: Debian sid
Flash Version: 

This is a Linux-specific issue; I can't reproduce it on Windows or OS X.
 
chrome-fail.pcapng
11.5 KB Download
I should have noted that the pcapng file was created several weeks ago; it is possible that it was created with a different version of Google Chrome.

Additionally, while the OS version is valid for me personally, I should add that this seems unrelated --  I have several outstanding support requests for the same issue by people running various different Linux distributions.
Labels: -Cr-Internals-Network Cr-Internals-Network-SSL Needs-Bisect
Thanks for filing the bug! Would you be willing to do a bisect to narrow down the regression? Instructions are at: http://dev.chromium.org/developers/bisect-builds-py
You could look up the corresponding revision number at https://omahaproxy.appspot.com/. For instance 47.0.2526.111 is revision 352221. Download the bisect script and then invoke the following command.
python tools/bisect-builds.py -a linux -g good-revision -b 352221

Labels: Needs-Feedback
Before you do #2, could you provide us a netlog dump? Instructions at https://sites.google.com/a/chromium.org/dev/for-testers/providing-network-details . 
here you go.
net-internals-log.json
4.1 MB Download
If the bisect is necessary, let me know and I'll look into it. I'd like to avoid that if possible though.
Actually, apparently bisect-builds does a bisect with *prebuilt* chromium, rather than expecting me to build it myself. That does make it less onerous :-)

I'm on it.
The bisect ends with:

You are probably looking for a change made after 338708 (known good), but no later than 338714 (first known bad).
NOTE: There is a Blink roll in the range, you might also want to do a Blink bisect.
Traceback (most recent call last):
  File "bisect-builds.py", line 1270, in <module>
    sys.exit(main())
  File "bisect-builds.py", line 1266, in main
    PrintChangeLog(min_chromium_rev, max_chromium_rev)
  File "bisect-builds.py", line 1070, in PrintChangeLog
    print ('  ' + CHANGELOG_URL % (GetGitHashFromSVNRevision(min_chromium_rev),
  File "bisect-builds.py", line 1063, in GetGitHashFromSVNRevision
    data = json.loads(url.read())
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Don't think the JSON error is critical, but hey.

Also, when I try to run the script with the -l option (to do a blink bisect), it fails with:

python ./bisect-builds.py -a linux64 -l -g 335443 -b 352221 -- https://test.eid.belgium.be/
Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions)
Traceback (most recent call last):_x64/339009/
  File "./bisect-builds.py", line 1270, in <module>
    sys.exit(main())
  File "./bisect-builds.py", line 1228, in main
    not opts.not_interactive, evaluator)
  File "./bisect-builds.py", line 828, in Bisect
    revlist = context.GetRevList()
  File "./bisect-builds.py", line 501, in GetRevList
    self.good_revision)
  File "./bisect-builds.py", line 1034, in FixChromiumRevForBlink
    blink_deps_rev = GetBlinkDEPSRevisionForChromiumRevision(self, rev)
  File "./bisect-builds.py", line 987, in GetBlinkDEPSRevisionForChromiumRevision
    return int(_GetBlinkRev(url, blink_re))
TypeError: int() argument must be a string or a number, not 'NoneType'
Cc: svaldez@chromium.org
Labels: -Needs-Bisect -Needs-Feedback
Owner: davidben@chromium.org
Status: Assigned
No bisect or feedback needed.

We're aware it changed. That was somewhat intentionally, somewhat unintentionally. There are undesirable performance characteristics to how it behaves, and long term, it's unlikely we'll continue to support sending intermediates (on any platform), but we may restore the functionality for the short-term.

You should configure your server to have the intermediates or know how to properly construct a certificate chain from the supplied client certificate. The onus for certificate path building rests on the thing doing the validation, since it knows more about what it's trying to accomplish. Any reliance on Chrome sending intermediates is inherently flawed, since that requires Chrome running path building algorithm with unknown criteria, and so is at best, best effort, but generally, poor for performance and certificate selection.

David: Tentatively assigning to you, but feel free to punt it from your queue if you're getting overloaded.
Status: Started
I have this CL from earlier that I need to dust off but should do the trick.
https://codereview.chromium.org/1526783002/
Hi Ryan,

Good to know the issue is already known and being dealt with.

As for the long-term plans you're referring to, I'm not sure I follow your reasoning.

Background: this issue was encountered in the context of the Belgian electronic identity card (for which I help maintain the PKCS#11 module), issued to every Belgian citizen over the age of 12 as well as to some foreigners. Each of these cards may have up to two client certificates (but also contains its intermediate and root certificate).

Since there are many citizens, storing them all in a single CA is not feasible, so there are many intermediate CAs (about 10 to 16 for the citizen cards, and about 5 for the foreigner cards, *for every year of issuance*). Older cards were valid for a period of 5 years, so that means that there are currently between 75 and 100 intermediates active. However, in 2014 the card validity was increased to 10 years, so these numbers will double over time.

The server needs to send a list of valid trust anchors to the client to allow the client to determine which client-side certificate to select. I can see only two ways of doing that: The first is that the server has the intermediate certificates and sends them to the client; in our case, that would mean the server would need to send hundreds of intermediate certificates in the "Certificate Handshake" message, which would make it rather large -- and that would also have a significant impact on performance.

Or, the server sends just the root certificates (as it does now); the client needs to build the path anyway (otherwise it can't select the correct certificate), so I don't understand how that can still have a major impact on performance.

What am I missing?
A similar remark applies to chrome.certificateProvider.onCertificatesRequested.
Here you can only pass the end-entity certificate, not the entire certificate chain. Within the context of using a smart card (via chrome.usb), that carries its full certificate chain, it would be interesting if you could pass this full certificate chain as certificateProvider.

In some cases (like an eID smart card), the client-side knows better how to construct the certificate chain instead of the server-side.
The client side does not know how to construct a suitable chain; that is at the servers discretion. Servers should be prepared to receive only a leaf certificate, and should properly implement chain building to handle this case.
Let me draw an analogy here. I drive around in my car. Cops halt me for a check. They ask my identity card. Is it up to the cop to reach into my pocket for some identity proof? No, I -as client- will present what I think my identity should look like at that moment. The cop -server side- will only validate what I present. It's not his job to "reconstruct" my identity as only I know best what's in my pocket as identity proof.

In case of the Belgian eID, the client-side knows perfectly how to construct a suitable chain, as the chain resides on the eID card itself. The server-side on the other hand, does not always have the latest intermediate CA's available (because new intermediate CA's are being put into production constantly). Hence the client-side can certainly make useful "suggestions" here by presenting the accurate full certificate chain.

The above described scenario is a reality.
Any update on this? Please note that this issue impacts 11 million Belgians that use the eid to login into a bunch of government and other website.
 Issue 607544  has been merged into this issue.
The description in comment #13 is incorrect. When you present your drivers license, the cop knows what a legitimate or illegitimate license looks like. He knows that the authority is derived from the license issuing institution, which itself derives its power from the state, from which he in turn trusts and acts on its behalf.

The validation of certificate chains based on incomplete knowledge is wholly the remit and expectation of the server, in that it bears the duty to determine if the license presented to them is valid. If, for example, I presented an international drivers license, then the cop may walk back to their vehicle and call it in for confirmation, which may include determining if my international drivers license is legitimate and bears the correct authorization. That's because the state for which the cop works for has a reciprocality agreement with my country. But it is not my obligation, of the driver, to present the license *and all evidence related to how it was issued* - that's what the cop themselves reconstrusts.

To further explain why this is a difficult problem:
You have a certificate A, signed by Key B/Name 1 (since Keys/Names are a pair).
The client has two different certificates that match Key B/Name 1 - one of which was issued by CA "C", the other which was issued by CA "D". Which certificate should be presented if the server gives no statement about who it trusts?

Now, let's again work with this thought experiment, with certificate A, signed by Key B/Name 1. The client has three certificates that match Key B/Name 1, all issued by the CA "C". The first certificate is revoked, the second is valid according to the user's clock (but invalid according to the server's clock), and the third is valid. The server indicates it trusts CA "C". What chain should the client send?
- If you argue it shouldn't send the first one, then you're saying the client should check revocation. But that's expecting that the client knows what the server will do.
- If you argue it shouldn't send the second one, then you're saying the client should know the server's clock is wrong - that is, it should be psychic.
- If you argue it should send the third one, then you're naturally saying that the client needs some logic to determine how to construct a chain that the server will accept.

And that's the fundamental flaw of the argument - that the client should know what the server will accept. It can't. The client should present what it can be for sure - the user's certificate - and the server should be responsible for handling path building and validation per RFC 5280. The client does not know how to construct a certificate chain out of the possible paths.

The old code was broken for several reasons, performance included, but fundamentally it's something that a client cannot know, and clients should not be expected to know. Well-configured servers (and plenty exist) are able to handle this situation. IIS, for example, knows perfectly well how to construct the certificate chain.
I don't really see how this is related to situation where I have *single* client certificate, signed by intermediate CA which was signed by CA mentioned by server when requesting client certificate. There is only *one* certificate that we have to send, there is no ambiguity, we just fail to send its whole chain.
Because of how Chrome is implemented, the distinction between the two scenarios is indistinguishable. Chrome itself, via the APIs it uses, does not know whether you have a single intermediate or multiple, and across different platforms, behaves differently under all variations of that. One of our goals of Chrome is consistency, which sometimes means not supporting something that may only be supported on a single platform, unless and until such a time as we can support it on all.
Is this different or the same as the changes that occurred when BoringSSL was implemented all the way back in m46? As a result of the BoringSSL implementation we had to create certificate bundles of all the CAs (a very lengthy process) on the server-side to ensure proper authentication.
It is the same.
Hi Ryan,

On 28-04-16 19:33, rsleevi@chromium.org via Monorail wrote:

I understand your argument, and fundamentally this is your decision, not 
mine.

Having said that, the statement that the client "cannot" know something, 
while true, is not entirely relevant to the merit of sending 
certificates along to the server. By the very same argument, the client 
also "cannot" know whether the server will accept its end-entity 
certificate.

In order to filter the client's available end-entity certificates so as 
to select the (set of) certificate(s) which may possibly be acceptable 
to the server, the client will need to build a path anyway. Once it's 
built that path, it seems reasonable to expect it to retain the 
information it gathered to perform said filtering, and send it to the 
server, along with the end-entity certificate.

But I guess I'm missing something here, and it probably means we'll have 
to implement some workarounds for this issue.
Issue 548631 has been merged into this issue.
Cc: davidben@chromium.org
Labels: OS-Chrome
Owner: rsleevi@chromium.org
(Swapping ownership to Ryan since there seems to be some question as to whether this is a good idea. Also adding OS-Chrome since that would also be affected.)
Status: Assigned
Comment 25 by t...@pwnt.be, Jul 27 2016
Forgive my ignorance, but mainly out of curiosity: how much of the following is correct?

1. The fact that other browsers and platforms as well as old Chromes are able to handle this edge case is because their PKCS implementation is too lax. The behavior of newer versions of Linux Chrome is the only correct one, strictly speaking.

2. For this to work, the Belgian government needs to adapt its servers to allow them to construct trust chains which were previously being constructed on the client side.

Thanks!
#1 isn't tied to PKCS (anything). It's not that they were lax, it's just the handling was incidental and not required (by spec or by client)

#2 is correct. Any server expecting client certificates should be prepared to do path building/verification, As the PKI is a directed cyclic graph, only those with full knowledge of the graph can evaluate the trust paths.
Cc: -davidben@chromium.org rsleevi@chromium.org
Labels: M-54
Owner: davidben@chromium.org
Status: Fixed
Oops, attached the CL to the wrong bug.

https://chromium.googlesource.com/chromium/src.git/+/8d569f5901989954c0b39a83f16ebd36375e98b8

If someone affected could try the Dev Channel build in a week or two, that would be great. I'll try to remember to ping this thread again when that happens.

Now hopefully this change won't break something else. But it's client certificates and client certificate deployments are all awful, so I'm sure someone will break. :-(
The latest Dev channel release on Linux (54.0.2816.0) should include the change, if someone wants to confirm that worked.
Seems to work for me. Thanks!
Sign in to add a comment