davidben@chromium.org

Thanks! The error seems to be that some record from the server did not decrypt correctly. Interestingly, we did manage to decrypt some of the data. I'm going to cross-reference the data in your NetLog with your keys and see if I can figure out where it went wrong.

I poked around id:757. The second application data record (the one containing the bulk of the gzipped data) is wrong, so we were right to reject the record. Though how that happened, I'm not sure.

The authenticator on the record is *almost* right. The last two bytes are wrong.

Want: 080e8908cfcc9e7d1e29b35c7088c7a8
Got:  080e8908cfcc9e7d1e29b35c70889e7d

If I remove the auth check, the rest of the record seems to be valid. At least, if I feed all that into the gzip decoder, gzip is happy and I get something which looks like HTML. I can't compare it against the actual one because that URL is dynamically generated and has since changed.

id:760 is similar.
Want: a303bbc2f6ccf858d9c5c573230ca945
Got:  a303bbc2f6ccf858d9c5c573230cf858

id:770
Want: 469fa5496d782792bcfeff164cd87ed7
Got:  469fa5496d782792bcfeff164cd82792

id:798
Want: 8b786c7b02ad94c81c648f3153928f8a
Got:  8b786c7b02ad94c81c648f31539294c8

This one managed to get several requests through without failing. When it does fail, it seems to still fail at the same resource, however.

I'm not sure what to make of this. It seems someone somewhere (Chrome, your network, or the server) is managing to corrupt the last couple of bytes of records when the HTML file is fetched.

The page loads fine for me, but it has changed since. Does it still occur for you? If so, you mentioned in the other bug report that incognito mode didn't have this problem. Is this still the case?

These are all resumptions, but that doesn't mean much in itself; if there were any connections to the host before starting to gather NetLogs, it's expected that subsequent ones would be resumptions. Though incognito would use a different session cache, so maybe there's something related there?

I can't reproduce the issue right now. In the original issue, I was able to reproduce it in incognito (it doesn't happen every time). I asked the site to see if there's anything interesting on server side.

Can extensions modify authenticator on the record?
With NetLog, sslkeylog.txt and pcap, we can verify if the bytes were correct (decrypt pcap) over the wire?

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I don't believe Chrome extensions can modify that, no. (Your network and any proxiesa or random networking middleware can.)

To verify it, I wrote a lot of custom code to check stuff. :-) In principle, one can do that, yeah. You might be able to capture a pcap and feed it into Wireshark. Hopefully Wireshark will notice this. Though it probably wouldn't be able to tell you that the last two bytes were what was broken.

If you can't reproduce it right now, it may have something to do with the sizes of everything, since the resource into question (that HTML listing) is dynamic. Perhaps someone somewhere has a memory error and is only noticing when the sizes are just right.

Thanks for filing the issue!

This issue seems to be similar to Issue:  893958 , hence merging into it and marking it as Duplicate.
Note: Feel free to Un-Dupe it if not the case,

Thanks!

This one is not a new issue and looks quite different  from 893958 (different site, error type) . 

Agreed. Sorry about that, some of our triages were too over-eager in merging things this morning and we've had to clean up after them.

I was able to reproduce this issue again with https://www.mail-archive.com/cisco-nsp@puck.nether.net/
. pcap also included
https://www.dropbox.com/s/89hrqjj2wqf7zhb/crbug_889805_20181016.zip?dl=0

ISP Comcast

>Ubuntu 16.04.5 LTS

Google Chrome	69.0.3497.100 (Official Build) (64-bit)
Revision	8920e690dd011895672947112477d10d5c8afb09-refs/branch-heads/3497@{#948}
OS	Linux

Firefox ESR 52.9.0 64bit
getting SSL_ERROR_BAD_MAC_READ

>android 8.0.0, Chrome 69.0.3497.100
ERR_SSL_PROTOCOL_ERROR on wifi
unable to repro on Verizon LTE

>Windows10 x64
69.0.3497.100 unable to repro

Firefox ESR 52.9.0 64bit unable to repro

Hrm. If you're able to repro this in Firefox too, that suggests something outside of Chrome. So that leaves something on your computer, something on your ISP, or mail-archive.com itself.

Are the Windows and Ubuntu machines the same network? Were they tested at the same time? (I.e. is it possible it's triggered by a particular state of mail-archive.com and that page had changed in between the two tests?)

If both are yes, that suggests something on the Ubuntu machine in particular.

I guess it is a server side issue.

I can reproduce this issue with curl from different hosts/networks

$ curl 'https://www.mail-archive.com/ausnog@lists.ausnog.net/msg02881.html'  -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H 'Upgrade-Insecure-Requests: 1' -H 'DNT: 1' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7' --compressed -v
*   Trying 72.52.77.8...
* Connected to www.mail-archive.com (72.52.77.8) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 596 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: www.mail-archive.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: OU=Domain Control Validated,CN=www.mail-archive.com
*        start date: Sat, 22 Apr 2017 23:52:00 GMT
*        expire date: Wed, 22 Apr 2020 23:52:00 GMT
*        issuer: C=US,ST=Arizona,L=Scottsdale,O=GoDaddy.com\, Inc.,OU=http://certs.godaddy.com/repository/,CN=Go Daddy Secure Certificate Authority - G2
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /ausnog@lists.ausnog.net/msg02881.html HTTP/1.1
> Host: www.mail-archive.com
> Connection: keep-alive
> Cache-Control: max-age=0
> Upgrade-Insecure-Requests: 1
> DNT: 1
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
> Accept-Encoding: gzip, deflate, br
> Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7
> 
< HTTP/1.1 200 OK
< Date: Fri, 19 Oct 2018 06:55:06 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Link: <https://www.mail-archive.com/ausnog@lists.ausnog.net/msg02881.html>; rel="canonical"
< Content-Encoding: gzip
< Vary: Accept-Encoding
< Content-Length: 2242
< Keep-Alive: timeout=5, max=500
< Connection: Keep-Alive
< Content-Type: text/html; charset=utf-8
< 
* GnuTLS recv error (-24): Decryption has failed.
* Closing connection 0
curl: (56) GnuTLS recv error (-24): Decryption has failed.

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Weird. I'll go ahead and close this then, and send the mail-archive.com folks a note that something may be off with their server.

I'm the admin for mail-archive.com. This is a total mystery to me. I can 100% reproduce the failure in comment #15. If I change the SSL config to remove ECDHE-RSA-AES256-GCM-SHA384 the cipher changes to ECDHE-RSA-AES256-SHA384 and starts working for me.

I've had a few user reports on this topic. One was from August 9, 2018 on Firefox and the user is no longer able to reproduce the problem after the SSL change, but this is not definitive. This user says "BTW, while looking at the certificate analysis for your server, I've noticed one issue: Under Certification Path #2, it shows that the root certificate is sent by the server. https://www.ssllabs.com/ssltest/analyze.html?d=mail-archive.com From what I've read, sending the root server certificate in the chain is considered to be incorrect, and presumably might trigger some type of problems."


I have another user who originally reported on Sept 27,2018 against Chromium. They are still reproducing the problem with ECDHE_RSA_AES_128_GCM_SHA256 on curl. Their curl trace is attached.

Finally, I asked the best expert I know on this topic who speculated on SSL build errors  or perhaps a hardware flaw in the CPU's carryless-mult unit. It's a little hard for for me to imagine stock Ubuntu 18.04 would ship with a busted SSL. It's also a little hard for me to imagine we'd only see a CPU hardware flaw now, rather than years ago. This server has been in service for quite a few years

For what it is worth, this is my local curl version on a Linux laptop.

curl 7.60.0 (x86_64-pc-linux-gnu) libcurl/7.60.0 OpenSSL/1.0.2o zlib/1.2.8 libidn2/2.0.2 libpsl/0.19.1 (+libidn2/2.0.2) libssh2/1.8.0 nghttp2/1.31.1 librtmp/2.3

Ideas or suggestions for experiments are welcome.

Deleted: success-config.txt 560 bytes

success-config.txt 560 bytes View Download

Deleted: success.txt 2.3 KB

success.txt 2.3 KB View Download

Deleted: fail-config.txt 558 bytes

fail-config.txt 558 bytes View Download

Deleted: fail.txt 2.6 KB

fail.txt 2.6 KB View Download

Deleted: user-fail-report.txt 2.4 KB

user-fail-report.txt 2.4 KB View Download