Issue metadata
Sign in to add a comment
|
All Cloudflare sites fail with ERR_SSL_PROTOCOL_ERROR (Fortinet)
Reported by
kanepy...@gmail.com,
Dec 25 2016
|
||||||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.28 Safari/537.36 Example URL: https://www.cloudflare.com/ Steps to reproduce the problem: 1. Start computer on Christmas morning 2. Try to watch twitch stream 3. why is ffz not loading 4. why is no cloudflare site not loading 5. submit bug What is the expected behavior? Using curl works. There is Fortinet interception hardware on the network, but it is set to SNI blocking only. It should only be blocking or allowing connections; but I included the full bytes logging just in case that was the issue. What went wrong? Attached is a bytes-included log of an attempted connection. Did this work before? Yes I don't have records, but whatever I just updated from. Chrome version: 56.0.2924.28 Channel: beta OS Version: Flash Version: Shockwave Flash 24.0 r0 Curl -v log: $ curl -I -v https://www.cloudflare.com >/dev/null * Rebuilt URL to: https://www.cloudflare.com/ Trying 198.41.215.162... * Connected to www.cloudflare.com (198.41.215.162) port 443 (#0) * found 173 certificates in /etc/ssl/certs/ca-certificates.crt * found 704 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: cloudflare.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: * start date: Fri, 28 Oct 2016 00:00:00 GMT * expire date: Fri, 02 Nov 2018 12:00:00 GMT * issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert ECC Extended Validation Server CA * compression: NULL * ALPN, server accepted to use http/1.1 > HEAD / HTTP/1.1 > Host: www.cloudflare.com > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 OK < Date: Sun, 25 Dec 2016 21:39:58 GMT < Content-Type: text/html < Connection: keep-alive < Set-Cookie: __cfduid=dd0fcd09d4cfb3a711312d0f3b07afcf31482701998; expires=Mon, 25-Dec-17 21:39:58 GMT; path=/; domain=.cloudflare.com; HttpOnly < Last-Modified: Fri, 23 Dec 2016 18:10:24 GMT < link: </vendor/bitdashplayer.min.css>; rel=preload; as=style, </vendor/bitdash-controls.min.css>; rel=preload; as=style, </video/marketing-video/cloudflare-marketing-video.mpd>; rel=preload, </video/marketing-video/cloudflare-marketing-video.m3u8>; rel=preload, </video/marketing-video/video_0_800000/dash/init.mp4>; rel=preload; as=video, </video/marketing-video/audio_0_128000/dash/init.mp4>; rel=preload; as=video < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=31536000 < X-Content-Type-Options: nosniff < X-Frame-Options: SAMEORIGIN < Served-In-Seconds: 0.002 < CF-Cache-Status: HIT < Expires: Mon, 26 Dec 2016 01:39:58 GMT < Cache-Control: public, max-age=14400 < Server: cloudflare-nginx < CF-RAY: 316f72e149a50d85-SJC < * Connection #0 to host www.cloudflare.com left intact
,
Dec 27 2016
Other Chrome computers on the same network do not have an error. Other browsers on this computer do not have an error. My Android phone connected to the network doesn't have an error.
So I suppose this is either
(A) the Fortinet blocking this computer specifically in an unusual way - normally it would serve a bad-certificate 302 to the "access denied" page
This doesn't really make sense, as Firefox and Curl work fine.
(B) Something about the request is causing the Fortinet to think the connection is an attack. E.g. an experiment
I didn't see a TLS 1.3 connection header in the initial request packet. What else would be objectionable?
(C) Something about the response handshake is causing the Fortinet to terminate the connection.
,
Dec 27 2016
This doesn't repro for me on Chrome Stable. So it's definitely either a change in Beta or an experiment trial.
,
Dec 28 2016
I can't repro on Mac Canary. Could you please provide an about:net-internals log of this happening? Instructions: https://sites.google.com/a/chromium.org/dev/for-testers/providing-network-details Make sure to open up about:net-internals before running into the error.
,
Dec 28 2016
It looks like Fortinet is blocking SSL connections that it doesn't understand. Chrome is attempting to negotiate TLS 1.3 with Cloudflare, and is successfully sending the initial message (ClientHello), but is receiving an alert (likely from Fortinet) that shuts down the connection since its unable to inspect the SNI from the server response.
,
Dec 28 2016
Though looking at the timing, its likely that its failing to parse the ServerHello and not even getting to attempting to parse the certificate. Can you try connecting to https://cert-test.sandbox.google.com and seeing if that connection also fails (sending a net-internals log for that connection if so).
,
Dec 28 2016
Removing needs feedback label (Sorry, missed that the original post had a net-internals log)
,
Dec 28 2016
And adding it back, per comment 6 (Just one of those days...)
,
Dec 28 2016
Ugh. Fortinet seems to have a "SSL Certificate Inspection" mode which is the sort of thing TLS 1.3 intentionally breaks. The certificate is now encrypted: http://kb.fortinet.com/kb/documentLink.do?externalID=FD35043 This document suggests their certificate-sniffing mode also checks SNI, which is probably the SNI mode the reporter is referring to. In principle they could just notice we always send SNI and never bother parsing beyond that, but I'm guessing they're trying to parse beyond that and failing. http://kb.fortinet.com/kb/documentLink.do?externalID=FD34661 +awhalley, can you find a contact for Fortinet? We'll need to get them to fix this.
,
Dec 28 2016
Actually +awhalley
,
Dec 28 2016
Ironically, if you turn on the "Full SSL Inspection" mode, I bet that would work around the problem, even though that sort of thing tends to be more buggy.
,
Dec 30 2016
#11: That would require deploying the interception root, which is forbidden by the usage profile of "home network with some narrow site restrictions". Also I can't provide additional testing right now as I came back from Xmas holiday.
,
Dec 30 2016
We've reached out to Fortinet to let them know their software has problems with TLS 1.3. Ultimately they'll need to revise how this feature is implemented. (TLS 1.3 deployment is just starting, which is why you're only seeing a little bit of it now, but it will eventually be deployed widely on both browsers and websites.) No response yet, but I imagine they too are all on holiday. :-)
,
Jan 5 2017
(As an update, my original set of Fortinet contacts didn't get anywhere, but I've managed to reach someone now.)
,
Jan 5 2017
Fortinet was unable to reproduce the problem. They've asked for the following information:
* The Fortinet product model deployed
* The software version used
* A backup of the configuration when the problem can be reproduced or if
they do not want to disclose their configuration:
=> Ensure the traffic hits the expected firewall policy by checking
the 'log&report>forward traffic' logs
=> The UTM profiles associated to the firewall policy with
certificate-inspection
=> Verify that the webfilter can correctly classify the category and
what is the applied action by checking the logs in 'log&report> web filter'
* Ensure there is no other device in transit that could disturb the
TLSv1.3 traffic
I'll also point them at this bug so they can chime in directly.
,
Feb 2 2017
Setting issue as assigned because owner was assigned in comment #5.
,
Feb 21 2017
We have at least one very large customer seeing similar issues against BlueCoat. The connection fails with SSL_HANDSHAKE_ERROR / ERR_CONNECTION_CLOSED. Customer found that restricting to TLS 1.2 via policy resolves the issue for Chrome 56 stable. Net internals logs are at: https://drive.google.com/corp/drive/folders/0B3BtTQPWWixOMk1FNkhMekJnNEU (google.com view only) Other large EDU customers are seeing similar issues and I'm working to gather details from them on proxy / firewall in use. Suspect many are using SSL / TLS inspection which is common among EDUs. Marking this as ReleaseBlock-Stable and P1 as I believe this is breaking Chrome for many customers.
,
Feb 21 2017
Do we have a contact with the EDU customers to figure out what TLS inspection hardware/software they are running, and what particular version of BlueCoat is being run by the customer seeing issues? We expect that SSL_PROTOCOL_ERROR and ERR_CONNECTION_CLOSED errors are a result of issues with non-conformant middleboxes.
,
Feb 21 2017
Bluecoat version is 6.5 for affected customer.
,
Feb 21 2017
jayhlee: Please file a *separate* bug for the Blue Coat issue. Note this bug talks about Fortinet. Removing your label changes here as this original bug is not currently actionable. Feel free to add RBS and Pri-1 to your other ticket. (Strange. We got in touch with Blue Coat ahead of time and they reported it was fine. Blue Coat's implementation strategy for TLS interception is not legitimate and cannot be supported. The customer needs to talk to Blue Coat support.)
,
Feb 21 2017
filed crbug.com/694593 at David's request.
,
Feb 28 2017
We've heard back from Fortinet that the issue is in an older version of their software. They say updating to FortiOS 5.4.0 or later will fix it. They also suggest that a possible workaround would be to disable URL filters and replace with FortiGuard filters. (Since the original reporter for this bug isn't responding, I'll go ahead and close this. But if someone is affected by this and uses Fortinet, confirmation that the instructions work would be much appreciated!)
,
Jun 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3 commit e34d74248235b57ce7c4a9ba05956f5f3f12d8c3 Author: David Benjamin <davidben@chromium.org> Date: Thu Jun 29 20:35:16 2017 Histogram TLS 1.3 interference causes and fix TLS13 experiment set. This histograms a number of TLS 1.3 interference failure modes known to be associated with buggy middleboxes. These metrics are gathered to classify the causes of TLS version interference and better understand the situation. Since each individual report, particularly against closed or reset connections, is indistinguishable from transient network failures, only gather metrics for the TLS 1.3 experiment set of servers. The noise will still be there, but we won't be contending with noise from connections to TLS 1.2 servers. Also fix the TLS 1.3 experiment set. The original list diverged slightly from what was deployed. Bug: 694593, 676969 , 733223 , 658863 Change-Id: I9e78804a4c62318edd8b6fbb170b03afacd86e86 Reviewed-on: https://chromium-review.googlesource.com/550279 Commit-Queue: David Benjamin <davidben@chromium.org> Reviewed-by: Steven Valdez <svaldez@chromium.org> Reviewed-by: Alexei Svitkine <asvitkine@chromium.org> Cr-Commit-Position: refs/heads/master@{#483473} [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket.cc [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket.h [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_impl.cc [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_impl.h [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_pool.cc [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_pool.h [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/tools/metrics/histograms/enums.xml [modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/tools/metrics/histograms/histograms.xml |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by kanepy...@gmail.com
, Dec 27 2016