New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 822290 link

Starred by 4 users

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Dec 5
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug



Sign in to add a comment

ERR_SSL_VERSION_INTERFERENCE on numerous web sites

Reported by witchyw...@gmail.com, Mar 15 2018

Issue description

Chrome Version       : 65.0.3325.162
OS Version: 6.1 (Windows 7, Windows Server 2008 R2)
URLs (if applicable) : https://www.ivanti.com/  http://www.harbormarine.net/ https://www.jenreviews.com
Other browsers tested:
  Add OK or FAIL after other browsers where you have tested this issue:
     Safari:
    Firefox:
    IE/Edge: (IE on Win7) OK

What steps will reproduce the problem?
1. Try going to the website
2.
3.

What is the expected result? Seeing the site


What happens instead of that?


Please provide any additional information below. Attach a screenshot if
possible.

UserAgentString: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36



 

Comment 1 by mmenke@chromium.org, Mar 15 2018

Components: Internals>Network>SSL
Labels: Needs-Feedback
Could you please provide a chrome://net-export/ log of this happening?  Instructions: https://dev.chromium.org/for-testers/providing-network-details.  Just visiting one or two sites while logging should be fine.  The log won't include cookies, but will include your information about your proxy configuration and any visited websites.

What firewall/proxy products are you using?
Cc: svaldez@chromium.org
Started the logging, opened a tab, tried to go to www.ivanti.com, hit reload, closed the tab and stopped the log. We are a financial institution, so security is pretty tight. We have a Fortigate 200 firewall that NATs traffic from inside to dark fiber connected ISP (Green House Data). Web traffic is aggressively filtered and proxied including deep content inspection which does mess with certificates. The sites I reported work fine in IE on my computer (Windows 7 Pro) and also in Firefox on another computer.
chrome-net-export-log.json
1.2 MB View Download
Project Member

Comment 4 by sheriffbot@chromium.org, Mar 16 2018

Cc: mmenke@chromium.org
Labels: -Needs-Feedback
Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Cc: davidben@chromium.org
Labels: Needs-Feedback
Owner: svaldez@chromium.org
Status: Assigned (was: Unconfirmed)
Can you try getting a net-export log while hitting https://tls.ctf.network/version.php, so that we can try correlating against the server-side.
Accessing this site also failed with the same error.
chrome-net-export-log2.json
841 KB View Download
It looks like something on your network is messing with the session ID in the ClientHello and ServerHello. Trying to reproduce locally we haven't had luck. Do you know what version of FortiOS is being run on the firewall?

Comment 8 by mmenke@chromium.org, Mar 16 2018

Cc: -mmenke@chromium.org
Currently running FortiOS v5.6.2 build1486 (GA)
The firewall rule that governs HTTP and HTTPS traffic has several "Security Profiles" applied to it:
Anti-Virus: Default = full AV scan, treat EXE in attachments like a virus
Web Filtering: fairly loose, only blocking stuff like port, drugs and weapons
DNS Filter: block DNS to known C&C, redirect DNS requests to our ISP DNS server
Application Control: THIS LOOKS INTERESTING --> QUIC is currently set to BLOCK and the help text reads: Allowing QUIC instructs the FortiGate to inspect Google Chrome packets for a QUIC header and logs as a QUIC message. Blocking QUIC forces Google Chrome to use HTTP2/TLS1.2 and the FortiGate logs QUIC as blocked.
IPS: protect client
SSL/SSH inspection: Full Decryption with exceptions for certain web filter categories like Finance and Business; inspects all SSL/SSH standard ports.

Comment 10 by agl@chromium.org, Mar 19 2018

Cc: agl@chromium.org

Comment 11 Deleted

Our previous tests with Fortinet showed that it behaved properly in December, so it looks like a recent change or the 5.6 release vs the 5.4 release is the issue. We'll try getting a version to test locally against and see if we can reproduce.

Tested against https://tls.ctf.network/version.php.
Log1 shows tests against other sites (https://www.androidpolice.com, others).

FortiOS 5.2.11 with inspection enabled.

Chrome 65.0.3325.181 64 bit.

chrome-net-export-log.json
642 KB View Download
chrome-net-export-log1.json
1.3 MB View Download
IE 11.309.16299.0 OK
Firefox 59.0.1 OK
This is caused by a bug in Fortinet's product. It appears their product has non-compliant behaviors and didn't implement TLS 1.2 correctly. Unfortunately, this bug did not trigger until clients and servers started deploying TLS 1.3 (imagine if an HTTP implementation broke on unknown HTTP headers). TLS 1.3 is intended to be a completely backwards-compatible improvement to TLS, but bugs in middleware products can break this compatibility, as with any other such change.

We worked around some of the Fortinet bugs (and bugs in other middleware products) by adding hacks in the protocol, but it now appears they've since introduced a new bug, or this bug was in one of the products or modes we were unable to test.

Do you have a support contract with Fortinet folks? Could you file a ticket with them? We've been sporadically in touch with them for these issues over the past year, but they've been unresponsive.

For now, there is a temporary SSLVersionMax administrative policy which may be set to disable TLS 1.3 in your enterprise. However, note this policy is temporary. Please file a ticket with Fortinet so they can fix these bugs.

https://www.chromium.org/administrators/policy-list-3#SSLVersionMax
I have opened a ticket with Fortinet and included your response. They tried to close the case by stating:

I have confirmed TLS v1.3 is still a working draft, even by IETF organization. So we don't have any deployment document for it yet. You can follow up with IETF about it at https://tools.ietf.org/html/draft-ietf-tls-tls13-23 

Since I think the draft was finalized today? I have escalated the ticket and hopefully can get someone on it. Its frankly ridicules. 

Comment 17 by agl@chromium.org, Mar 26 2018

> They tried to close the case by stating:

Fortigate do not need to support or know about TLS 1.3. They only need to implement TLS 1.2 (from 2008) correctly. TLS 1.3 is designed to be backwards compatible otherwise every web site would need to support it before it could be enabled, and it's backwards compatible for (compliant) proxies too.

Sadly we've had discussions going back over a year with Fortigate about this, but they stopped replying to us. We thought that we had worked around the issue by changing TLS 1.3, but clearly some combination of firmware version and configuration still has issues :(


Comment 18 by lassey@google.com, Apr 27 2018

Labels: -Needs-Feedback
This issue also occurs when using Chrome Version 70.0.3538.110 (Official Build) (64-bit) for Windows to access a virtual server on a Fortigate 800D running FortiOS 5.6.2 when SSL inspection is enabled for the virtual server traffic. 

Fortinet Support Ticket 3020902 has been created for the issue and includes a link to this discussion.

I am in the process of attempting to have Fortinet technical support escalate the issue to the FortiOS developers and have asked that the FortiOS developers reach out to the Chrome Development team.
Status: WontFix (was: Assigned)
TLS 1.3 had been in development for years, with many attempts on our end to reach out to them. They were unresponsive and now the protocol is finalized and deployed in Chrome, so it is now too late to make any changes to it or deployment strategies.

From our experience with other broken middleware, it is almost certainly the case that the Fortigate product did not implement the 10-year-old TLS 1.2 correctly. As noted above, there is no requirement for Fortigate to support the new TLS 1.3, but they must implement the existing TLS 1.2 protocol correctly. (Of course, users of such an SSL inspection device are limited in security by whatever that product supports, so it would also be prudent for them to keep up with security standards.)

If their development team reaches out to us now, we can help them in diagnosing the implementation flaws, but there is nothing actionable to be done in Chrome at this point. The issue must be fixed on the Fortigate end. Thus, I'm closing this ticket on our end.
I agree that Fortinet needs to get on board. However, waiting and closing a bug report in response to status update in an attempt to get traction for a vendor to resolve a problem shows poor form in my opinion. If there was no intention to do anything with this bug report after TLS 1.3 was finalized, why wasn’t it closed back on August 28,2018 when RFC 8446 was published?

From: david… via monorail <monorail+v2.294852074@chromium.org>
Sent: Wednesday, December 5, 2018 12:38 PM
To: JT Moore <JamesT.Moore@chsfl.org>
Subject:  Issue 822290  in chromium: ERR_SSL_VERSION_INTERFERENCE on numerous web sites
(We had a lot of these open and lost track of this one until you pinged it. :-) Though back in August, the protocol was finalized, but the final version hadn't yet been deployed, just a draft. Now, everything has thoroughly stuck and there's nothing to change anymore.)

This bug tracker is to track things that we need to do on our end. We close things when they're not actionable. Like I said, if Fortinet start being responsive, we can certainly do what we can to help diagnose their bugs, but I do not think there's anything here that's actionable on our end right now.
We just started getting the ERR_SSL_VERSION_INTERFERENCE error at 830pm CT last night. We do have a fortigate firewall but nothing on it has changed in over a month.

We use uptime robot and they were able to ping us on these failures as well.
ssl error2.png
27.3 KB View Download
ssl error.png
128 KB View Download
Unfortunately, the Fortinet level 1 and level 2 support staff I worked with were not very familiar with the issue and I had to be rather persistent with them to get the issue escalated. However, after finally getting the issue escalated, the Fortinet escalation engineer was able to quickly provide a resolution which, in our case, was to upgrade to FortiOS 6.0.2. Although it is not explicitly documented in the publically available release notes, FortiOS 6.0.2 contains a new version of the IPS engine that corrects the condition leading to ERR_SSL_VERSION_INTERFERENCE and similar errors occurring when using browsers with TLS 1.3 enabled are used to access a HTTPS virtual server on the FortiGate with SSL inspection enabled. We were also provided an interim update for the just the IPS engine as another possible option to resolve the problem, but elected to upgrade the entire OS instead. If you prefer to only update the IPS engine, you will need to request the specific update for the hardware and version FortiOS you are using from Fortinet support since those updates not readily available for download by end users.

Note: If you are encountering this error in a situation other than when using browsers with TLS 1.3 enabled are used to access a HTTPS virtual server on the FortiGate with SSL inspection enabled, you should open a support case with Fortinet since there are some other scenarios such as using SSL inspection for client traffic passing through the FortiGate to sites on the Internet which may require a different resolution.
From: dpu… via monorail <monorail+v2.3961533212@chromium.org>
Sent: Monday, December 31, 2018 1:12 PM
To: JT Moore <JamesT.Moore@chsfl.org>
Subject:  Issue 822290  in chromium: ERR_SSL_VERSION_INTERFERENCE on numerous web sites
Thanks for the update! Glad to hear they've since fixed the bug.

Sign in to add a comment