Do you see this for ALL websockets (e.g. do secure sockets work on https://websocketstest.com/) or just when using Servlet.com?

Can you attach a network log of the failure? https://dev.chromium.org/for-testers/providing-network-details

Adding on this, issue occurs with websockets served with TLS over non-443 and 8443 ports. Issue on the Jupyter notebook tracker, complete with a site for testing:

https://github.com/jupyter/notebook/issues/1849

rgbkrk: See comment #1. Could you provide a network log? Thanks!

Hi, here is network log

Deleted: net-internals-log.json 911 KB

net-internals-log.json 911 KB View Download

Thank you for providing more feedback. Adding requester "elawrence@chromium.org" for another review and adding "Needs-Review" label for tracking.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

https://github.com/jupyter/notebook/issues/1849#issuecomment-255314646
"This seems related to the Safari won't connect websockets with self-signed certs issue, in that browsers don't seem to reliably use the same configuration for websockets as regular connections."

Starts at #768 in the net-internals-log.json:

t=2718 [st= 0] +REQUEST_ALIVE  [dt=90]
t=2718 [st= 0]    URL_REQUEST_DELEGATE  [dt=0]
t=2718 [st= 0]   +URL_REQUEST_START_JOB  [dt=90]
                  --> load_flags = 18 (BYPASS_CACHE | DISABLE_CACHE)
                  --> method = "GET"
                  --> priority = "LOWEST"
                  --> url = "wss://charts.odkl.ru/reports/socket"
t=2718 [st= 0]      URL_REQUEST_DELEGATE  [dt=0]
t=2718 [st= 0]      HTTP_CACHE_GET_BACKEND  [dt=0]
t=2718 [st= 0]     +HTTP_STREAM_REQUEST  [dt=90]
t=2718 [st= 0]        HTTP_STREAM_REQUEST_STARTED_JOB
                      --> source_dependency = 769 (HTTP_STREAM_JOB)
t=2808 [st=90]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                      --> source_dependency = 769 (HTTP_STREAM_JOB)
t=2808 [st=90]     -HTTP_STREAM_REQUEST
t=2808 [st=90]      URL_REQUEST_DELEGATE  [dt=0]
t=2808 [st=90]      CANCELLED
                    --> net_error = -501 (ERR_INSECURE_RESPONSE)
t=2808 [st=90]   -URL_REQUEST_START_JOB
                  --> net_error = -501 (ERR_INSECURE_RESPONSE)
t=2808 [st=90]    URL_REQUEST_DELEGATE  [dt=0]
t=2808 [st=90] -REQUEST_ALIVE

---------

t=2808 [st=90]   -SOCKET_POOL
                  --> net_error = -202 (ERR_CERT_AUTHORITY_INVALID)

The certificate captured in the log for charts.odkl.ru looks okay to me on Windows. 

This may be dupe of Issue 588789?

Note that https://tmp59.tmpnb.org:9001/ is not using a self-signed certificate, it uses certs from Lets Encrypt. Additionally, this bug is not present on macOS Sierra for Safari or Firefox.

Here's a capture log of going to https://tmp59.tmpnb.org:9001

Deleted: net-internals-log.json 1.9 MB

net-internals-log.json 1.9 MB View Download

Fun things:

Try running on an updated mac:

$ openssl s_client -showcerts -connect apple.com:443

$ openssl s_client -showcerts -connect google.com:443

Hint: Google is A-OK, Apple's site fails with macOS openssl

Has anyone been able to look at the internals log?

I've also reported bugs to Apple and given them more system info. I'm happy to get more information, logs, whatever.

ricea, could you look at this?

The net-internals log is just saying ERR_INVALID_RESPONSE after we get 205 bytes of data out of the socket on one connection. On the other, the socket hangs.

Looking at the the log for charts.odkl.ru, the CERT_VERIFIER step is failing for wss: but not for https:. For https:, two additional certificates appear in the verified_cert list, and cert_status = 0. For wss:, the verified_cert list is the same as the certificates list, and cert_status = 4.

For tmp59.tmpnb.org, the CERT_VERIFIER step appears to not even happen for the HTTPS connections, only for the WSS connections.

I know of few differences between WebSocket SSL connections and HTTPS SSL connections:

A different socket pool is used → impact on SSL handshake unlikely
Different connection throttling logic is used → impact on SSL handshake unlikely
The user is not given the opportunity to bypass certificate errors → does not seem to be happening here
HTTP/2 is not advertised → changes the handshake, but I don't know why it would change certificate verification behaviour

The WebSocket SSL config is created from the HTTPS SSL config here:

https://cs.chromium.org/chromium/src/net/http/http_stream_factory_impl_job.cc?q=websocket_server_ssl_config

I should be able to get an MacOS Sierra environment set up tomorrow, and then I will see whether I can reproduce the issue or not.

I think tmp59.tmpnb.org is a different issue. I can reproduce it on Linux.

I captured a net log with "Include the actual bytes sent/received." checked, and so was able to see the response headers which were returned:

HTTP/1.1 400 Bad Request
Server: nginx/1.11.4
Date: Wed, 09 Nov 2016 08:18:15 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 34
Connection: keep-alive

Can "Upgrade" only to "WebSocket".

Since the request headers clearly contain a correct "Upgrade" header, and it is the same regardless of the port, this looks like a server bug.

davidben, I'm sending this back to you to see if you know why the CERT_VERIFIER step differs for SSL WebSocket connections and not for HTTPS SSL connections. See the netlog in #3 and my comment #14.

Although there is also a CERT_VERIFIER difference in the netlog in comment #10, it doesn't appear to be related to the failure to connect, so I'm happy to ignore it.

I'm at IETF right now and won't be able to look into anything until the week after next.

Missing cert verify jobs is a red herring. Those results are cached and don't get reported in the logs if cached. There is no difference between how the certificates are verified. Though I wouldn't be surprised if the WebSockets layers above didn't route error click-throughs up properly.

Thanks for looking at this.

There's no way to click-through a WebSocket SSL error. What you can do is visit the origin first over HTTPS and click through. Then the certificate policy is reused. We have a test for this:

https://cs.chromium.org/chromium/src/chrome/browser/ssl/ssl_browser_tests.cc?q=TestWSSInvalidCertAndGoForward

All the sites in this bug are being accessed over HTTPS first. No SSL errors are clicked through in the attached netlogs, and I can see no reason to believe that a click through was required before the netlog started.

If the CERT_VERIFIER thing is a red-herring then I don't have anything to go on. I can't connect at all to charts.odkl.ru. servlet.com:443 responds with plain unencrypted HTTP to my attempts to speak SSL. tmp59.tmpnb.org:9001 has an unrelated problem.

If someone can reproduce the issue, please provide a byte-level capture as described in https://dev.chromium.org/for-testers/providing-network-details being very careful to avoid including private information. I suggest using an incognito window or fresh profile and closing all other windows.

I have confirmed that https://tmp59.tmpnb.org:9001/ also doesn't work in Firefox on Linux.

After selecting "Welcome to Python.ipynb", it shows "Connecting to kernel" in the top right but the connection never succeeds.

This strongly implies that the issue is not caused by Chrome.

I also see this bug on Chrome 54 with the same error message:
WebSocket connection to '~~~' failed: WebSocket opening handshake was canceled

This error only happens on Chrome running under the MacOS Sierra; very unusual and hope that this bug will be identified soon. 
FYI for websocket Server we are using engine.io. 

ultravistor, I think your problem may be unrelated. The "WebSocket opening handshake was canceled" error corresponds to the internal ERR_ABORTED error code. This is different from the ERR_INSECURE_RESPONSE seen in the charts.odkl.ru netlog or the bad handshake seen in the tmp59.tmpnb.org netlog.

WebSocket error codes are reported in the Net.WebSocket.ErrorCodes histogram. There's no significant different between MacOS 10.11 and MacOS 10.12 in ERR_ABORTED frequency, so I don't this the problem is widespread. The rate is fairly constant at around 0.01% of WebSocket connections on MacOS. There's a peak to 0.016% around November 10th, but I think it's just noise as there have been similar, larger peaks in the past.

Interestingly, ERR_INSECURE_RESPONSE *is* significantly higher for MacOS 10.12. It was steady at about 0.05% for 10.11, but is now at 1%. Something is going on, but I don't yet have enough information to determine what it is. I need to find a publically available server that has the issue.

I have changed the title of this issue to be more specific about the problem.

Lately I've received reports of technical issues with some of our services. Although I don't have much information, I suspect it's related to this bug. Shortly after the release of macOS Sierra 10.12.0, some users have reported problems with our services that uses websockets. Later those same users reported that the issue had been fixed, which I suspect might coincide with the release of 10.12.1.

I received a single screenshot from a user with the following error visible in the Chrome debug console:
"Websocket connection to 'wss://sub.domain.com/socket.io/1/websocket/mvNn8jvgQMssUhRP6rRc' failed: WebSocket opening handshake was canceled" (where sub.domain.com is a node.js server secured with wildcard certificate provided from COMODO CA)

I'm unable to reproduce the problem on my testbench running 10.12.1. It seems obvious to me that Apple did something in 10.12.0 that resulted in Chrome being unable to use secure websockets under certain conditions. While the issue may now appear to have been fixed, I'm concerned as to why only the Chrome browser was affected.

I hope this will be of help in this investigation. I'll answer any questions to the best of my abilities.

fataldevoverflow: Thank you for your input. It would be useful to find patterns in the people who have the issue, for example that they are/aren't running in corporate environments where certificate selection is changed via an admin policy.

My best guess (and this is purely a guess) is some kind of bad interaction between Apple's TCP/IP implementation and Chrome's SSL implementation. Other browsers (except Opera) use a different SSL implementation. As for why HTTPS is unaffected, the bug may be sensitive to the exact size of the initial handshake.

The rate of ERR_INSECURE_RESPONSE seems steady since October 24th so unfortunately I think it's most likely not fixed in 10.12.1. On the positive side, if there actually is a bug in Chrome it gives us more time to fix it.

If anyone can reproduce, either a netlog with full byte information, or a packet capture might enable us to reproduce the exact SSL configuration that exhibits the issue.

Hi Ricea,

I'm having this bug on my chrome too. How can i get this log for you?

I also have this issue after upgrade to Sierra 10.12.1. It worked perfectly before the upgrade.  People who have not upgraded in my team do not have this problem. 

kernel.js:455 WebSocket connection to 'wss://dev.aktiun.com:9443/user/jorge3/api/kernels/e2485328-8f94-48ba-b357-d515c9ae3647/channels?session_id=183695F385994165851BA81EAA273D37' failed: WebSocket opening handshake was canceled

Does anyone have macOS Parental Controls enabled?

maximillian.forasteiro: Yes please! Please follow the instructions at https://dev.chromium.org/for-testers/providing-network-details including the "Byte-level captures" part.

You can email the log to me directly at ricea@chromium.org if the log might contain non-public information.

jorge: Would it be possible for you to provide a network log as well?

I have reproduced the issue on Sierra 10.12.1. I am currently dissecting packet traces.

What I've learnt so far:

The Client Hello message only differs by the presence of the ALPN extension in the https: trace. Due to padding the length is 512 bytes for both protocols.

The certificates sent by the server are identical for both protocols.

The Server Hello also only differs in the presence of the ALPN extension. The length is different because it's not padded.

There's nothing noticeably odd at the TCP/IP level. In the case of WSS, Chrome cleanly closes the connection 1 millisecond after receiving the Encrypted Handshake Message.

My current working hypothesis is that it's a problem with Authority Information Access. The site I've reproduced with is using Authority Information Access, doesn't send an intermediate certificate and doesn't work in Firefox.

Unfortunately, I currently have a sample size of 1. It would help me quickly discover whether or not this is a dead end if I could find some other sites that use AIA and don't send an intermediate certificate. If anyone knows how to find such sites I would appreciate it. They don't need WebSocket support - I only need to verify whether an SSL connection is established or not.

Hi Ricea,

I got this log while I was trying to load a internal site that uses websockets.

If there is anything that I could help you guys to solve this problem, please, let me know!

Deleted: net-internals-log (1).json 5.9 MB

net-internals-log (1).json 5.9 MB View Download

I have the same error.

"websocket connection to 'wss://my-domain.com:8000/' failed: websocket opening handshake was canceled" only for Chrome (54.0.2840.98) at macOS Sierra (10.12)

I try browsers:
Windows: 'Firefox', 'Chrome', 'IE', 'Opera' - works fine.
Linux: 'Firefox', 'Chrome' - works fine.
Mac: 'Firefox', 'Safary' - works fine.

Error only at Chrome at Mac.

Now I found this topic and see suggestion about changing port to 443 or 8443. I will try it now.

Change port to 8443 not helps.

elfanlide, thank you for the very thorough report. That seems to rule out Authority Information Access as the cause, although it is still possible for Firefox to connect to a server that uses Authority Information Access if it has the intermediate certificate cached.

The port number really shouldn't make any difference. I think that is a red herring.

The port number is a red herring, we're experiencing this issue even on the standard 443 port. Websockets are only a problem in Chrome on macOS currently.

I just discovered that it works fine in Canary. I am running a bisect to try to find the change which fixed it.

I found the change that fixed it. It was landed on Nov 10:

https://chromium.googlesource.com/chromium/src/+/1a282f54af2ea79cf9aa0b78234e3b4053b4d73b

The change will be in version 55 of Chrome. I will ask whether it can be backported to the current version, 54.

Adam: I don't believe we want to merge it to M54. It is a large CL that significantly affects how we verify certificates on OS X.

Considering these sites also wont work in Firefox or on Chrome for Android, due to it's lack of AIA fetching, we should acknowledge that these sites can be reconfigured to work correctly.

rsleevi, I was afraid of that.

So, in principle, adding the intermediate certificates to the server configuration as described in https://degreesofzero.com/article/how-to-fix-missing-intermediate-ssl-certificate-errors-in-apache.html or https://www.godaddy.com/help/iis-7-install-a-certificate-4801 should work? Do you know of a better page that describes this issue and the fix?

I've reproduced this issue and commented in #21 last week; I've confirmed that we can avoid the error by adding intermediate certificates (tested on Chrome 54 & macOS Sierra).

Also confirmed that Chrome 55 works without adding ca. Maybe some acknowledgements would be helpful to the site owners/admins. Thanks

Sorry, the version is Chrome 57, not 55. 
> Also confirmed that Chrome 55 works without adding ca. Maybe some acknowledgements would be helpful to the site owners/admins. Thanks

ultravistor: thank you for the confirmation.

Do you know of any good documentation on how to configure intermediate certificates for engine.io?

To clarify, versions 55.0.2883.51 and newer contain the fix. So current versions of Chrome from the beta channel should not have this problem.

versions of Chrome from the beta channel should not have this problem.

Thanks, I got it.

Following pages doesn't specifically states intermediate certificates but
contains some examples (Nodejs based modules, including socket.io/engine.io,
ws etc, uses "https" so we just need to pass some parameters to https when
we create new object)

http://nategood.com/nodejs-ssl-client-cert-auth-api-rest

https://certsimple.com/blog/a-plus-node-js-ssl?client=ms-android-sonymobile

2016/11/22 午後10:03 "ri… via monorail" <monorail+v2.1555508240@chromium.org>:

ricea: In general, the advice has been for server operators to contact their CAs for assistance on installing intermediates, as CAs maintain the most comprehensive support documentation for the intermediates necessary to install and how to install them for the various services. There's enough pages on the Internet to provide guidance, but something like http://serverfault.com/questions/628062/what-is-wrong-with-my-ssl-trust-chain or https://community.qualys.com/docs/DOC-1931 are sufficient to explain the issue.

I'am site owner and admin.
I can share link to server for testing at private message.

If needs additional configuration at nginx for solve this problem - it is okay. But provided links - only theoretical information.

I have 2 domains with my service with valid https. So one I can use for config testing. Please say what should I change.

Current config part of websocket:
----------------------------------------------------
upstream websocket {
    server 127.0.0.1:8000;
}
server {
    listen my.ip.ad.dr:8443 ssl;
    server_name my-domain.com;
    proxy_read_timeout 65m;

    #ssl on;
    ssl_certificate      /etc/nginx/ssl/my-domain.com.crt;
    ssl_certificate_key  /etc/nginx/ssl/my-domain.com.key;
    ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers  "RC4:HIGH:!aNULL:!MD5:!kEDH";
    add_header Strict-Transport-Security 'max-age=31536000';
    keepalive_timeout 70;

    location / {
        proxy_pass http://websocket;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
    }
}
----------------------------------------------------

elfanlide: See http://nginx.org/en/docs/http/configuring_https_servers.html#chains

Briefly, you replace the /etc/nginx/ssl/my-domain.com.crt file with one that also includes the intermediate certificates.

Thanks a lot! It works!

But I think this issue for "chrome mac" should be fixed anyway, but who needs urgent fix - can use provided link at comment #46.

-------------------------------------------
When I try at local computer "openssl s_client -connect my-domain.com:443":
...
Verify return code: 21 (unable to verify the first certificate)

After command at server "cat my-domain.com.crt CAbundle.crt > my-domain.com.chained.crt" and use new cert at nginx config (then nginx reload) - the same local command return:
...
Verify return code: 0 (ok)

After that I go to chrome mac (no updates for os or browser) and try use wss service - and it works without cache flushing.

Additional I try return to my port #8000 - it works too.

Lowering the priority as it can be fixed on the server side and the client-side fix will be released with version 55.

 Issue 664738  has been merged into this issue.

I have confirmed that the rate of ERR_INSECURE_RESPONSE is back down to 0.05% in M55. Thanks everyone.