New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 676969 link

Starred by 4 users

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Feb 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug-Regression



Sign in to add a comment

All Cloudflare sites fail with ERR_SSL_PROTOCOL_ERROR (Fortinet)

Reported by kanepy...@gmail.com, Dec 25 2016

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.28 Safari/537.36

Example URL:
https://www.cloudflare.com/

Steps to reproduce the problem:
1. Start computer on Christmas morning
2. Try to watch twitch stream
3. why is ffz not loading
4. why is no cloudflare site not loading
5. submit bug

What is the expected behavior?
Using curl works.

There is Fortinet interception hardware on the network, but it is set to SNI blocking only. It should only be blocking or allowing connections; but I included the full bytes logging just in case that was the issue.

What went wrong?
Attached is a bytes-included log of an attempted connection.

Did this work before? Yes I don't have records, but whatever I just updated from.

Chrome version: 56.0.2924.28  Channel: beta
OS Version: 
Flash Version: Shockwave Flash 24.0 r0

Curl -v log:

$ curl -I -v https://www.cloudflare.com >/dev/null
* Rebuilt URL to: https://www.cloudflare.com/
Trying 198.41.215.162...
* Connected to www.cloudflare.com (198.41.215.162) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 704 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: cloudflare.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: 
* 	 start date: Fri, 28 Oct 2016 00:00:00 GMT
* 	 expire date: Fri, 02 Nov 2018 12:00:00 GMT
* 	 issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert ECC Extended Validation Server CA
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> HEAD / HTTP/1.1
> Host: www.cloudflare.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sun, 25 Dec 2016 21:39:58 GMT
< Content-Type: text/html
< Connection: keep-alive
< Set-Cookie: __cfduid=dd0fcd09d4cfb3a711312d0f3b07afcf31482701998; expires=Mon, 25-Dec-17 21:39:58 GMT; path=/; domain=.cloudflare.com; HttpOnly
< Last-Modified: Fri, 23 Dec 2016 18:10:24 GMT
< link: </vendor/bitdashplayer.min.css>; rel=preload; as=style, </vendor/bitdash-controls.min.css>; rel=preload; as=style, </video/marketing-video/cloudflare-marketing-video.mpd>; rel=preload, </video/marketing-video/cloudflare-marketing-video.m3u8>; rel=preload, </video/marketing-video/video_0_800000/dash/init.mp4>; rel=preload; as=video, </video/marketing-video/audio_0_128000/dash/init.mp4>; rel=preload; as=video
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Served-In-Seconds: 0.002
< CF-Cache-Status: HIT
< Expires: Mon, 26 Dec 2016 01:39:58 GMT
< Cache-Control: public, max-age=14400
< Server: cloudflare-nginx
< CF-RAY: 316f72e149a50d85-SJC
< 
* Connection #0 to host www.cloudflare.com left intact
 
net-internals-log (3).json
482 KB View Download

Comment 1 by kanepy...@gmail.com, Dec 27 2016

<strike>I was inspecting the log and noticed that the TLS13NegotiationEnabled experiment was Enabled - is that perhaps relevant?</strike>

The bytes show the connection was attempted with TLS 1.0, and the connection is closed with a code 49 "Access denied" TLS error.

Comment 2 by kanepy...@gmail.com, Dec 27 2016

Other Chrome computers on the same network do not have an error. Other browsers on this computer do not have an error. My Android phone connected to the network doesn't have an error.

So I suppose this is either

(A) the Fortinet blocking this computer specifically in an unusual way - normally it would serve a bad-certificate 302 to the "access denied" page
    This doesn't really make sense, as Firefox and Curl work fine.
(B) Something about the request is causing the Fortinet to think the connection is an attack. E.g. an experiment
    I didn't see a TLS 1.3 connection header in the initial request packet. What else would be objectionable?
(C) Something about the response handshake is causing the Fortinet to terminate the connection.

Comment 3 by kanepy...@gmail.com, Dec 27 2016

This doesn't repro for me on Chrome Stable. So it's definitely either a change in Beta or an experiment trial.

Comment 4 by mmenke@chromium.org, Dec 28 2016

Components: -Internals>Network Internals>Network>SSL
Labels: Needs-Feedback
Status: Untriaged (was: Unconfirmed)
I can't repro on Mac Canary.  Could you please provide an about:net-internals log of this happening?  Instructions: https://sites.google.com/a/chromium.org/dev/for-testers/providing-network-details

Make sure to open up about:net-internals before running into the error.
Cc: davidben@chromium.org
Labels: -OS-Linux -Needs-Feedback OS-All
Owner: svaldez@chromium.org
Summary: All Cloudflare sites fail with ERR_SSL_PROTOCOL_ERROR (Fortinet) (was: All Cloudflare sites fail with ERR_SSL_PROTOCOL_ERROR)
It looks like Fortinet is blocking SSL connections that it doesn't understand. Chrome is attempting to negotiate TLS 1.3 with Cloudflare, and is successfully sending the initial message (ClientHello), but is receiving an alert (likely from Fortinet) that shuts down the connection since its unable to inspect the SNI from the server response.
Labels: Needs-Feedback
Though looking at the timing, its likely that its failing to parse the ServerHello and not even getting to attempting to parse the certificate.

Can you try connecting to https://cert-test.sandbox.google.com and seeing if that connection also fails (sending a net-internals log for that connection if so).

Comment 7 by mmenke@chromium.org, Dec 28 2016

Labels: -Needs-Feedback
Removing needs feedback label (Sorry, missed that the original post had a net-internals log)

Comment 8 by mmenke@chromium.org, Dec 28 2016

Labels: Needs-Feedback
And adding it back, per comment 6 (Just one of those days...)
Ugh. Fortinet seems to have a "SSL Certificate Inspection" mode which is the sort of thing TLS 1.3 intentionally breaks. The certificate is now encrypted:
http://kb.fortinet.com/kb/documentLink.do?externalID=FD35043

This document suggests their certificate-sniffing mode also checks SNI, which is probably the SNI mode the reporter is referring to. In principle they could just notice we always send SNI and never bother parsing beyond that, but I'm guessing they're trying to parse beyond that and failing.
http://kb.fortinet.com/kb/documentLink.do?externalID=FD34661

+awhalley, can you find a contact for Fortinet? We'll need to get them to fix this.
Cc: awhalley@chromium.org
Actually +awhalley
Ironically, if you turn on the "Full SSL Inspection" mode, I bet that would work around the problem, even though that sort of thing tends to be more buggy.
#11: That would require deploying the interception root, which is forbidden by the usage profile of "home network with some narrow site restrictions".

Also I can't provide additional testing right now as I came back from Xmas holiday.
We've reached out to Fortinet to let them know their software has problems with TLS 1.3. Ultimately they'll need to revise how this feature is implemented. (TLS 1.3 deployment is just starting, which is why you're only seeing a little bit of it now, but it will eventually be deployed widely on both browsers and websites.)

No response yet, but I imagine they too are all on holiday. :-)
(As an update, my original set of Fortinet contacts didn't get anywhere, but I've managed to reach someone now.)
Fortinet was unable to reproduce the problem. They've asked for the following information:

* The Fortinet product model deployed
* The software version used
* A backup of the configuration when the problem can be reproduced or if
they do not want to disclose their configuration:
    => Ensure the traffic hits the expected firewall policy by checking
the 'log&report>forward traffic' logs
    => The UTM profiles associated to the firewall policy with
certificate-inspection
    => Verify that the webfilter can correctly classify the category and
what is the applied action by checking the logs in 'log&report> web filter'
* Ensure there is no other device in transit that could disturb the
TLSv1.3 traffic

I'll also point them at this bug so they can chime in directly.

Comment 16 by b...@chromium.org, Feb 2 2017

Status: Assigned (was: Untriaged)
Setting issue as assigned because owner was assigned in comment #5.

Comment 17 by jayhlee@google.com, Feb 21 2017

Labels: -Pri-2 ReleaseBlock-Stable Pri-1
We have at least one very large customer seeing similar issues against BlueCoat. The connection fails with SSL_HANDSHAKE_ERROR / ERR_CONNECTION_CLOSED. Customer found that restricting to TLS 1.2 via policy resolves the issue for Chrome 56 stable. Net internals logs are at:

https://drive.google.com/corp/drive/folders/0B3BtTQPWWixOMk1FNkhMekJnNEU (google.com view only)

Other large EDU customers are seeing similar issues and I'm working to gather details from them on proxy / firewall in use. Suspect many are using SSL / TLS inspection which is common among EDUs.

Marking this as ReleaseBlock-Stable and P1 as I believe this is breaking Chrome for many customers.
Do we have a contact with the EDU customers to figure out what TLS inspection hardware/software they are running, and what particular version of BlueCoat is being run by the customer seeing issues?

We expect that SSL_PROTOCOL_ERROR and ERR_CONNECTION_CLOSED errors are a result of issues with non-conformant middleboxes.


Comment 19 by jayhlee@google.com, Feb 21 2017

Bluecoat version is 6.5 for affected customer. 
Cc: jayhlee@chromium.org
Labels: -Pri-1 -ReleaseBlock-Stable Pri-2
jayhlee: Please file a *separate* bug for the Blue Coat issue. Note this bug talks about Fortinet. Removing your label changes here as this original bug is not currently actionable. Feel free to add RBS and Pri-1 to your other ticket.

(Strange. We got in touch with Blue Coat ahead of time and they reported it was fine. Blue Coat's implementation strategy for TLS interception is not legitimate and cannot be supported. The customer needs to talk to Blue Coat support.)

Comment 21 by jayhlee@google.com, Feb 21 2017

filed crbug.com/694593 at David's request.
Status: WontFix (was: Assigned)
We've heard back from Fortinet that the issue is in an older version of their software. They say updating to FortiOS 5.4.0 or later will fix it. They also suggest that a possible workaround would be to disable URL filters and replace with FortiGuard filters.

(Since the original reporter for this bug isn't responding, I'll go ahead and close this. But if someone is affected by this and uses Fortinet, confirmation that the instructions work would be much appreciated!)
Project Member

Comment 23 by bugdroid1@chromium.org, Jun 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3

commit e34d74248235b57ce7c4a9ba05956f5f3f12d8c3
Author: David Benjamin <davidben@chromium.org>
Date: Thu Jun 29 20:35:16 2017

Histogram TLS 1.3 interference causes and fix TLS13 experiment set.

This histograms a number of TLS 1.3 interference failure modes known to
be associated with buggy middleboxes. These metrics are gathered to
classify the causes of TLS version interference and better understand
the situation. Since each individual report, particularly against closed
or reset connections, is indistinguishable from transient network
failures, only gather metrics for the TLS 1.3 experiment set of servers.
The noise will still be there, but we won't be contending with noise
from connections to TLS 1.2 servers.

Also fix the TLS 1.3 experiment set. The original list diverged slightly
from what was deployed.

Bug: 694593,  676969 ,  733223 , 658863
Change-Id: I9e78804a4c62318edd8b6fbb170b03afacd86e86
Reviewed-on: https://chromium-review.googlesource.com/550279
Commit-Queue: David Benjamin <davidben@chromium.org>
Reviewed-by: Steven Valdez <svaldez@chromium.org>
Reviewed-by: Alexei Svitkine <asvitkine@chromium.org>
Cr-Commit-Position: refs/heads/master@{#483473}
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket.cc
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket.h
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_impl.cc
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_impl.h
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_pool.cc
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/net/socket/ssl_client_socket_pool.h
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/tools/metrics/histograms/enums.xml
[modify] https://crrev.com/e34d74248235b57ce7c4a9ba05956f5f3f12d8c3/tools/metrics/histograms/histograms.xml

Sign in to add a comment