I'll provide any additional info on demand

In firefox nightly (49.0a1 и 50.0a1) everything works as expected on machine with above-mentioned problem.

after update 52.0.2743.41 beta-m (64-bit) yhe issue still exist

pthatcher, can you help with an owner?

Our customer faced with this issue in Chrome version 52.0.2743.82 m. Is there any chance to speed this up?

The symptoms you describe (inconsistent repro across machines, consistent on one machine) suggest this behavior might be triggered by a Chrome experiment to make ECDSA certificates the default for WebRTC. We've seen a similar issue here: https://bugs.chromium.org/p/webrtc/issues/detail?id=5863.

On an affected machine, try going to chrome://flags/#enable-webrtc-ecdsa and choosing "Disabled". Let us know if that stops the problem from occurring.

As for why TLS negotiation is failing with an ECDSA certificate, a network capture (e.g. Wireshark) would be useful. Previously, for example, we've seen servers that advertise ECDH but are only partially configured for it, so we try to fall back to kDHE+aECDSA, which we don't actually support.

Thank you, I've tested this solution on 2 affected machines and the problem is solved in both cases!
So, these issues are really about one thing and should be merged.

The issue in https://bugs.chromium.org/p/webrtc/issues/detail?id=5863 was that the WebRTC peer had a noncompliant implementation of DTLS. The solution was a fix in the server software, rather than a fix in Chrome.

Have you concluded the bug here is also in your server? If so, great. Otherwise, we'll need a network capture of the DTLS handshake in order to continue the investigation.

Am I right that in future releases ECDSA will be set in "Enabled" state and I should update my Gateway to work with such certificates?

Oh, I didn't mention your comment, sorry, I'll check dump.

Yes, Chrome -- and WebRTC in general -- are planning on moving to ECDSA certificates by default. If you need to interoperate with peers that only support RSA, you can change your Javascript client code to create RSA certificates explicitly. See http://w3c.github.io/webrtc-pc/#sec.cert-mgmt.

I set enable-webrtc-ecdsa to "Enabled" and after Chrome restart I see the strange behaviour: with one server dtls setup finished successfully and with other server dtls setup finished with failure.

Recently I thought that problem should be persistent on both servers, because there is same Gateway software and same browser. So I make pcap file with successfull and failed dtls setup (I omited some packets, because I think they are not nessassary).

Deleted: dtls_success_failure.pcapng 2.5 KB

dtls_success_failure.pcapng 2.5 KB Download

Yep, this is consistent with what we saw in https://bugs.chromium.org/p/webrtc/issues/detail?id=5863. Specifically, the DTLS ClientHello (e.g. packet 3 in the attachment in #15) advertises support for EC key agreement protocols like TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, but since it doesn't include the elliptic_curves extension (https://tools.ietf.org/html/rfc4492#section-5.1.1), BoringSSL rejects those ECDH ciphers.

If there are inconsistencies here, it may be because the client/server roles switch. This problem will only occur if Chrome is acting as the server in DTLS.

Yes, thank you, I see it now!

Also I see some kind of "magic" in my setup.
I have same versions of my Gateway on 130.167.246.139 and 255.171.130.2 (so the code working with OpenSSL is same too), also I have same version of OpenSSL - 1.0.1e. Also I have one chrome browser calling both gateways in the same way (so there is no role switching). And now I can't figure out why they produce different "Client Hello" messages: the first one contains ECDSA and eliptic_curves extensions and the second one doesn't.
Maybe there is some system wide config or something else? May be you have some ideas, because I'm totally stuck:(

Any chance one or both of the gateways is behind a load balancer or reverse proxy?

For curiosity's sake, here's the fix to OpenSSL that started adding these extensions in the ClientHello for DTLS 1.0: https://github.com/openssl/openssl/commit/2054eb771ea29378f90d3a77c2f4015b17de702d

It was first included in OpenSSL 1.0.1i:
https://github.com/openssl/openssl/compare/OpenSSL_1_0_1h...OpenSSL_1_0_1i

Since we've established this is a bug with the gateway (admittedly exacerbated by conservative behavior in BoringSSL), I'm going to close this as WontFix.

I'll do more investigatesions, thank you for your time and patience.

May be it will help somebody else:
Inspite of the same version of openssl (which is determined from command "openssl version") there were different openssl packages on machines:
1) On problem machine openssl*   1.0.1e-16.el6_5.7
2) On good machine openssl*      1.0.1e-48.el6_8.1
So, after update to 1.0.1e-48.el6_8.1 problem machine start sending good "Client Hello" messages with full set of nessessary headers.

Thanks for circling back with details! Glad you were able to fix your setup.