Issue metadata
Sign in to add a comment
|
TLS ClientHello to fefe.de domains results in intermittent TCP/IP RST
Reported by
winroot...@gmail.com,
Feb 22 2018
|
||||||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.18 Safari/537.36 Steps to reproduce the problem: 1. visit https://blog.fefe.de 2. the server sends a RST after timeout What is the expected behavior? What went wrong? The Server does not support TLS 1.0 and somehow upgrades fail. Firefox and CURL work fine. Did this work before? Yes previous dev build Chrome version: 65.0.3325.18 Channel: stable OS Version: Ubuntu 17.10 Flash Version:
,
Feb 22 2018
the issue persists with Version 66.0.3346.8 (Official Build) dev (64-bit)
,
Feb 22 2018
There could be any number of bugs in the server implementation that would cause it to reject a valid TLS 1.2 ClientHello.
,
Feb 22 2018
Interesting. In Windows, it works for me in 63.0.3239.132 and 65.0.3325.88, but fails in 66.0.3350.0.
,
Feb 22 2018
Chrome sends a "TSLv1" Hello not 1.2, at least according to wireshark. Curl and firefox use 1.2 right away.
,
Feb 22 2018
No, you're looking at the record-layer version there, which is not actually meaningful. (In fact, you're looking at a TLS 1.3 ClientHello there.)
,
Feb 22 2018
You don't have any kind of antivirus, proxy, middlebox, or other similar sort of device on your machine or network, do you? I wasn't able to reproduce this on my machine, or on my local build of BoringSSL.
,
Feb 22 2018
FWIW, I'm encountering this on Corp with BeyondCorp proxies disabled.
,
Feb 22 2018
Oh huh. Okay, then probably it's not some kind of middlebox then. Weird. Is it correlated with toggling chrome://flags/#tls13-variant between "Disabled" and "Enabled (Draft 23)"?
,
Feb 22 2018
Re #10: Toggling the setting between those two values seems to have no effect.
,
Feb 22 2018
Hrm, well that's a relief at least. I don't suppose you have time to run a bisect-builds.py? :-)
,
Feb 22 2018
Re #12 - I'm trying. Windows bisects crash every tab in every build and this has happened for months, so I'm digging into that now.
,
Feb 22 2018
It seems to work as late as 66.0.3326.0 {#530538}, but bisect builds all fail after that due to an unrelated issue.
,
Feb 22 2018
On my Linux box, 66.0.3343.3 loaded the page. Upon updating to 66.0.3346.8, the page no longer loaded (after hard refresh).
,
Feb 22 2018
Odd. I can't repro on 66.0.3346.8 on my Linux box, and I don't see anything in that range that would have changed our ClientHello.
,
Feb 22 2018
FWIW, there's a Boring roll [1] in that range[2]: [1] https://boringssl.googlesource.com/boringssl/+log/7e5dd25d47face6deb452823ae4f40c4c7ca375a..61dedd681501616de5928fe5c0eac6640d4de0c1 [2] https://chromium.googlesource.com/chromium/src/+log/66.0.3343.0..66.0.3346.0?pretty=fuller&n=10000 Interestingly, the server claims to be "Gatling", which is advertised from a similar hostname; https://www.fefe.de/gatling/
,
Feb 22 2018
Right, there's nothing in that range that would have changed the ClientHello. (We even have change detector tests that trip whenever we unexpectedly change our ClientHello. :-) )
,
Feb 22 2018
I do not know what causes the "error" but it is gone for me now, working on Version 66.0.3346.8 (Official Build) dev (64-bit) again o.O This seems still very weird to me, but I cannot even reproduce the behavior now anymore. But it is good to know that I was not alone with that :-P seems like some oddity though.
,
Feb 22 2018
And on the topic of antivirus, nope, none, did not work at work and then did not work at home either, completely different IPSs so I would not count any connection issues in. Very weird.
,
Feb 22 2018
https://www.fefe.de/gatling/ seems to have the same behavior. I cannot load this site in Canary 64bit 66.0.3352.0 but I can in Chromium 32bit 66.0.3351.0 compiled yesterday. Is there a mechanism a user can use to turn off GREASE to eliminate that as one source of problems?
,
Feb 22 2018
No, we don't add random knobs like that. But GREASE is not new, and it's quite easy to confirm this via the bssl command-line tool.
,
Feb 22 2018
*to confirm that GREASE is not the cause. Connecting repeatedly with the command-line tool with GREASE enabled does not reproduce the issue. Also because of auto-reload, if it were a very low-probability event from GREASE, auto-reload would have saved you.
,
Feb 23 2018
OK, now https://lieferungen.orrs.de/ is also affected. All fine in Firefox but only a RST/Timeout in Chrome.
,
Feb 23 2018
It is not a consistent fail, but it is very weird o.O
,
Feb 23 2018
RE #24: That's an Apache server, so it seems less likely to be related.
David-- I saw this at home as well; the pages on fefe.de loaded on Canary 66.3344 but after upgrading to 66.3353, I get the timeouts. I was thinking that perhaps this is a "revisit" issue (something with Session tickets or whatever?), but the behavior persists even in a Guest account.
Interestingly, the handshake looks quite different in Chrome 63:
t=4301 [st=151] SSL_HANDSHAKE_MESSAGE_SENT
--> hex_encoded_bytes =
01 00 00 BE 03 03 C2 12 33 76 DB 0D 20 2A 60 E6 . .....3v.. *`.
D9 5F C0 4F 72 26 94 99 4F CE 20 23 87 9C 77 3C ._.Or&..O. #..w<
55 35 2B 22 B9 96 00 00 1C CA CA C0 2B C0 2F C0 U5+".. ....+./.
2C C0 30 CC A9 CC A8 C0 13 C0 14 00 9C 00 9D 00 ,.0........ . .
2F 00 35 00 0A 01 00 00 79 3A 3A 00 00 FF 01 00 / 5 .. y:: ..
01 00 00 00 00 10 00 0E 00 00 0B 77 77 77 2E 66 . . . .www.f
65 66 65 2E 64 65 00 17 00 00 00 23 00 00 00 0D efe.de . # .
00 14 00 12 04 03 08 04 04 01 05 03 08 05 05 01 . .............
08 06 06 01 02 01 00 05 00 05 01 00 00 00 00 00 ...... . ..
12 00 00 00 10 00 0E 00 0C 02 68 32 08 68 74 74 . . . ..h2.htt
70 2F 31 2E 31 75 50 00 00 00 0B 00 02 01 00 00 p/1.1uP . ..
0A 00 0A 00 08 1A 1A 00 1D 00 17 00 18 7A 7A 00 . . ... . . .zz
01 00 .
--> type = 1
t=4301 [st=151] SOCKET_BYTES_SENT
--> byte_count = 199
Vs. the failing Chrome 66:
t= 2081 [st= 152] SSL_HANDSHAKE_MESSAGE_SENT
--> hex_encoded_bytes =
01 00 05 78 03 03 BE 92 E3 FE BF D8 43 AD 7F 7E . .x........C..~
94 80 10 21 6E 49 E8 EB C0 A0 A6 07 C1 BC 36 7E ...!nI........6~
B2 E7 65 F4 EB E7 20 2A 63 FA DA 21 C1 E3 38 75 ..e... *c..!..8u
E1 10 1B 2C E5 2B 26 04 A6 5D 64 3A DA 85 19 BE ...,.+&..]d:....
8C 00 50 20 F2 BD 01 00 22 CA CA 13 01 13 02 13 . P ... ".......
03 C0 2B C0 2F C0 2C C0 30 CC A9 CC A8 C0 13 C0 ..+./.,.0.......
14 00 9C 00 9D 00 2F 00 35 00 0A 01 00 05 0D 5A . . . / 5 .. ..Z
5A 00 00 FF 01 00 01 00 00 00 00 10 00 0E 00 00 Z .. . . .
0B 77 77 77 2E 66 65 66 65 2E 64 65 00 17 00 00 .www.fefe.de .
00 23 00 00 00 0D 00 14 00 12 04 03 08 04 04 01 # . . .......
05 03 08 05 05 01 08 06 06 01 02 01 00 05 00 05 ............ . .
01 00 00 00 00 00 12 00 00 00 10 00 0E 00 0C 02 . . . . ..
68 32 08 68 74 74 70 2F 31 2E 31 75 50 00 00 00 h2.http/1.1uP
0B 00 02 01 00 00 33 00 2B 00 29 5A 5A 00 01 00 . .. 3 + )ZZ .
00 1D 00 20 5A 4C 82 49 77 2C 7A 19 8F EB E4 DD . ZL.Iw,z.....
91 EC A9 AA 92 27 60 5D F4 E8 E9 C6 F7 6B 73 D8 .....'`].....ks.
41 A2 42 3C 00 2D 00 02 01 01 00 2B 00 0B 0A FA A.B< - ... + ...
FA 7F 17 03 03 03 02 03 01 D5 09 04 4C 50 A1 30 ............LP.0
03 39 E7 1B 22 F6 EE C0 DF 8F B2 9E 50 D5 10 5E .9..".......P..^
06 E9 B2 59 C5 D6 84 F3 BC F6 99 A9 EA FF 7E 1C ...Y..........~.
F0 56 66 06 72 4F 3A D5 B2 8E 9C 7A 8F 9F 58 95 .Vf.rO:....z..X.
60 32 32 BD 81 CA F5 AC 1E 48 6F 77 9A A5 87 A6 `22......How....
39 04 64 92 07 4C 60 72 32 98 4D 69 93 4B BE B2 9.d..L`r2.Mi.K..
B4 0B CC 53 A9 FD 32 71 DE BB 58 BF 13 F2 F8 AE ...S..2q..X.....
67 EB B9 F7 D0 C9 09 DB 93 58 8B 54 25 E4 A8 7E g........X.T%..~
C6 CB 75 24 E2 54 69 C1 B2 28 4C 17 E8 10 27 E1 ..u$.Ti..(L...'.
96 80 AE 08 B0 EF 12 B6 5C 53 50 19 90 76 76 7B ........\SP..vv{
1D 09 2C 72 A1 EF 47 4A C9 02 76 AD 46 A4 39 FA ..,r..GJ..v.F.9.
CA 81 BA 7D 6E 5A 6B AA BC 3D 8F 37 9F C8 68 83 ...}nZk..=.7..h.
49 D7 76 B3 97 05 FB 3D 2B F4 BC 8B 13 F1 7D A6 I.v....=+.....}.
30 E9 0B 6C 5A 99 F3 6C 08 11 59 15 BA F2 FF D3 0..lZ..l..Y.....
17 07 A4 EE 3B F6 22 8C 48 AA 00 E0 72 C4 79 4D ....;.".H. .r.yM
C7 D4 BB CE 6D FF 0F CA BA 14 10 D3 C8 85 4A 3E ....m.........J>
D3 5C D3 9E 63 01 25 99 2F 93 87 05 1F C0 FD AF .\..c.%./.......
14 93 4F 0D 59 35 96 38 96 A6 48 EF 3E 52 D4 0A ..O.Y5.8..H.>R..
87 7F AB FE A1 5E D5 66 03 A0 25 9B AA 5D F1 1C .....^.f..%..]..
3A 49 03 04 AD 12 1A D1 2C 22 39 9E 78 CC 19 76 :I......,"9.x..v
49 38 CA D6 AE C9 3C A8 F3 A5 D8 0A 18 A1 2B 4E I8....<.......+N
01 03 50 89 A4 B1 E6 80 10 3B AD BB 59 DF 84 E4 ..P......;..Y...
B9 AA 25 04 62 B5 D7 01 B2 9E F1 6B 63 DA 53 FE ..%.b......kc.S.
42 64 82 65 1F BA 2F 40 79 13 55 54 98 9D 12 76 Bd.e../@y.UT...v
B4 D6 72 6E B9 CF 2B 82 AD B8 17 52 CD 93 3A 96 ..rn..+....R..:.
35 2A 64 EE 84 7D 3F A0 C2 A5 7D CF 7B 62 66 36 5*d..}?...}.{bf6
3E 4E 24 F8 41 1E C0 A2 72 FE E7 C2 C2 5C 9E 62 >N$.A...r....\.b
F6 3E 4E 94 00 12 8B EB ED AA 90 5A 13 07 B5 D5 .>N. ......Z....
DD 6F 5A 56 39 49 6A DE 7D 89 E9 B8 F4 FF AC 65 .oZV9Ij.}......e
D0 5C 09 30 43 F1 3D 95 EB 62 FB B5 71 3B CE F3 .\.0C.=..b..q;..
EB AB C7 41 63 5D A8 63 A0 26 9B 2F D2 78 46 6A ...Ac].c.&./.xFj
12 CC 84 66 5C D7 C8 99 82 D0 A1 F6 EC 1B 56 42 ...f\.........VB
A4 86 DD F7 70 1D A0 E5 C0 3C 8E 8B 94 9D DE 82 ....p....<......
60 A2 ED 25 2C CF 86 D0 77 8C 90 9F 0C 33 63 F8 `..%,...w....3c.
C8 F5 DB 18 4C 4C 4F 8C DB 06 97 D1 85 BF A8 E0 ....LLO.........
B5 C9 6D 97 AC 56 EE 59 09 2D 44 F8 9B C5 5F BE ..m..V.Y.-D..._.
E1 96 A0 94 D1 B2 EC ED FB 45 CE 49 80 26 10 87 .........E.I.&..
B4 28 5C 22 7E 41 75 D7 45 36 8E D8 77 25 D2 9A .(\"~Au.E6..w%..
27 59 49 2C 13 3B 98 70 1E 2B A6 59 DB 4A 09 2C 'YI,.;.p.+.Y.J.,
FB B0 1D BB 34 F5 9C 3C 6E 1A C0 78 83 BA B2 9D ....4..<n..x....
56 3B A5 67 D7 CB 8F 58 A6 70 C1 4E F3 EB DD 54 V;.g...X.p.N...T
6E E9 1E 8A 85 2F D3 C7 DA D7 98 A2 27 CC DA E6 n..../......'...
3C 36 9A 3A 3D 5C E8 84 D9 17 CB 02 B6 C1 A4 B5 <6.:=\..........
4D E3 79 36 21 90 EE F8 A6 10 C8 A6 FF 01 D4 93 M.y6!...........
07 74 A8 8E 85 71 96 E1 71 8A 1E 0B F9 A1 E1 42 .t...q..q......B
68 9E DB 24 8D 25 69 86 20 B4 8D 93 92 72 9F 2B h..$.%i. ....r.+
37 E8 AC 13 18 D4 9C D4 DF 32 DF 27 0F 76 93 F1 7........2.'.v..
E2 79 E6 83 C8 49 2E 6E 86 5B 68 9B 30 7E FA 06 .y...I.n.[h.0~..
B6 25 6A 4C 5B 7E 74 5A CC BE A1 2C 2F BD D1 1B .%jL[~tZ...,/...
ED DA 8C 2E 8F 1B A6 31 3C 35 D1 14 6A C5 FF 98 .......1<5..j...
00 82 20 F3 D9 DF 2D 06 F4 A5 9A 57 C2 BA 59 A3 . ...-....W..Y.
71 5C 32 D7 21 12 3D 26 8C 3C B3 CD 85 71 FD C7 q\2.!.=&.<...q..
3E 01 B4 C6 31 A4 21 D0 74 C2 AF 2B A4 D1 39 23 >...1.!.t..+..9#
25 77 C7 9E C9 AC F2 3C 87 23 43 A2 14 8C 3E CA %w.....<.#C...>.
98 B8 2D 13 92 8B 10 86 1C F2 35 85 2F BA 72 B1 ..-.......5./.r.
53 21 8A 3B 28 77 36 20 E3 0E C6 F1 73 FB 54 CD S!.;(w6 ....s.T.
A3 DB F2 91 4A 63 DD 55 BC DD F3 36 4F 39 3A 1B ....Jc.U...6O9:.
24 13 EE 21 F8 C8 E2 CD E6 F3 BE 41 AD B9 B8 74 $..!.......A...t
B4 DB 7C EE 99 C7 1B 0A EF 44 F6 8A 9E E7 FA 04 ..|......D......
0D 10 31 66 F3 BB 06 5B 0D 48 90 AA 1C 3C 2B D9 ..1f...[.H...<+.
59 2E 99 79 5A 70 56 96 1C 17 3F 59 65 0F 98 67 Y..yZpV...?Ye..g
64 9C 3E E9 78 77 E6 32 19 BD CB AD D2 49 7B 69 d.>.xw.2.....I{i
6B 68 89 E2 52 0C 5F 30 CA D3 42 21 A9 F9 59 AF kh..R._0..B!..Y.
FD 9C 3D 61 0A 03 92 AD BA 1D A0 92 E3 23 28 06 ..=a.........#(.
07 AD ED AB 42 DB B9 80 1B 5D E9 2E FB 84 96 11 ....B....]......
CF B0 EE BC 22 BB 7B F4 B1 CD AD EB F4 A5 25 63 ....".{.......%c
83 C1 4F 31 C5 36 C2 22 C2 10 3B 6D 83 54 03 03 ..O1.6."..;m.T..
F1 4C A1 B0 A4 24 8A 0D 55 69 CF 15 68 0F 85 D7 .L...$..Ui..h...
EB A1 17 64 82 BD F4 A1 FC EB 39 2F A8 D1 36 66 ...d......9/..6f
E5 A8 6E B0 3F BA D7 99 30 00 0A 00 0A 00 08 5A ..n.?...0 . . .Z
5A 00 1D 00 17 00 18 9A 9A 00 01 00 Z . . ... .
--> type = 1
t= 2081 [st= 152] SOCKET_BYTES_SENT
--> byte_count = 1409
Is there an easy way to decode the bytes of the latter message, as they seem quite a bit longer than I would expect?
,
Feb 23 2018
first 3 times is Chrome and then follows firefox
,
Feb 23 2018
Ah, the huge handshake in #26 has a 1100 byte extension of type 0xd509 which appears to be DummyPQPadding.
,
Feb 23 2018
If the problem really is due to long handshake intolerance (which has been a problem for OpenSSL in the past), then https://chromium.googlesource.com/chromium/src/+/074164133fddd6c98ef26cb157996ca553261427 is a plausible explanation of why we would see this issue only from some clients.
,
Feb 23 2018
That would be in-line with when that code was landed (right before the M65 branch).
,
Feb 23 2018
This doesn't still need feedback, right?
,
Feb 23 2018
Thanks for the report. This server appears to have problems with any ClientHello message larger than 768 bytes. If you happen to know what software might be causing this, that would be interesting to know because I suspect that, within a couple of years, browsers may exceed that regularly. But, for now, this problem can be resolved by either using a stable release or waiting until March (when this experiment expires). Unless this issue is widespread (and the stats don't suggest that it is) then I don't think this is cause to end the experiment early.
,
Feb 23 2018
Re #32: The server claims to be "Gatling", and the readme.TLS implies that it uses OpenSSL. https://github.com/zg/gatling/blob/master/README.tls
,
Feb 23 2018
Gatling attempts to "tarpit" traffic that does not look like valid HTTP/HTTPS requests.
The Gatling do_sslaccept() contains the following, which fails on a ClientHello over 768 bytes.
/* Apparently the packets look radically different depending on
* whether it's TLS or SSLv2. GREAT! */
/* first packet must be handshake */
if (buf[0]!=0x16 || /* content type: handshake */
buf[1]>3 || /* version major */
buf[2]>3 || /* version minor */
buf[3]>2) { /* length > 0x200 */ <-- That's problematic.
if (buf[0]!=0x80 ||
buf[2]!=1 || /* Client Hello */
buf[3]>3 || /* version major */
buf[4]>3) { /* version minor */
tai6464 tarpit;
/* this does not look like an SSL packet. */
if (logging) {
char tmp[100];
buffer_puts(buffer_1,"close/not_ssl_traffic ");
buffer_putulong(buffer_1,sock);
buffer_putspace(buffer_1);
buffer_put(buffer_1,tmp,fmt_ip6c(tmp,h->peerip));
buffer_putnlflush(buffer_1);
}
,
Feb 23 2018
Thanks for the analysis, Eric. I'll contact the Gatling authors to see about fixing that.
,
Feb 23 2018
(I believe OpenSSL's ClientHello size limit is 16k. It's also ours, though we can freely change ours.)
,
Feb 23 2018
I think he will fix it quite soon and thx for the effort and work put into the issue. I think others might have taken short cuts in fighting DoS too, but maybe mostly on layer3-7 firewalls so there might be other side effects due to this change. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by winroot...@gmail.com
, Feb 22 201817.3 KB
17.3 KB Download