Issue metadata
Sign in to add a comment
|
Unaable to load login for https://my.screenname.aol.com/
Reported by
mtolle.m...@gmail.com,
Dec 10 2017
|
||||||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36 Steps to reproduce the problem: 1. attempt to log in to mail.aol.com 2. 3. What is the expected behavior? chrome presents log in page What went wrong? chrome presents "This site can’t be reached" Did this work before? Yes 62.0.3202.94 Chrome version: 63.0.3239.84 Channel: stable OS Version: Linux Mint 17.3 Rosa 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Flash Version: 27.0.0.187 /home/max/.config/google-chrome/PepperFlash/27.0.0.187/libpepflashplayer.so Still works properly in Ubuntu
,
Dec 11 2017
My network is fine - i reported this issue - and i am sending this reply - from the machine - using chrome - in the OS - stated in the report
,
Dec 11 2017
In other words - There seems to be some interaction [that i am unable to identify], between the browser update, [in this machine's environment], and the web page [or server], that causes [only] mail.aol.com [that i have observed so far] to fail to connect and/or load the login page properly ...
,
Dec 11 2017
I am guessing that something in the way chrome 63.0.3239.84 handles the redirect called by mail.aol.com has changed from the way chrome 62.0.3202.94 handles the redirect - as stated in the original report - that page loaded properly before the update. Tested on two multi-boot [Mint 17, Ubuntu 14, Windows 7, and Windows10] laptops with the same hardware and software. Chrome still loads the page properly in all those environments except Mint 17. Also - Firefox 57 continues to load the page properly ...
,
Dec 11 2017
If non-Mint 17 Linux environment is working fine, then I'd suggest filing a bug against Mint Linux first if other users are seeing the same error.
,
Dec 11 2017
So it's of no concern that the redirect worked properly in 62.0.3202.94 - but fails since the update to 63.0.3239.84 ???
,
Dec 11 2017
And exactly what do you think the maintainers of Mint 17 could do about a bug introduced by a change to chrome ???
,
Dec 11 2017
Again - Firefox 57 continues to load the page properly in Mint 17 !!!
,
Dec 11 2017
Again - my online banking site allowed me to log on using 62.0.3202.94 Now - since updating to 63.0.3239.84 - I find that my online banking site returns a page that says: "This site can't provide a secure connection" "onlinebanking.my_bank.com sent an invalid response" ERR_SSL_PROTOCOL_ERROR
,
Dec 11 2017
+thomasanderson@, could you ptal?
,
Dec 11 2017
,
Dec 11 2017
Tested on Linux Mint 18.3 and unable to reproduce
,
Dec 12 2017
Upon further investigation it seems like this problem could be related to Symantec / Digecert Certificates ? https://security.googleblog.com/2017/09/chromes-plan-to-distrust-symantec.html Many of the sites that we use to manage our lives seem to be affected - Banking - including : Credit card accounts Deposit accounts Investment accounts Health Care - including : Insurance Diagnostic reports Appointments Utilities, etc .... Can someone please test on the same OS as the one I've reported : Linux Mint 17.3 Rosa 4.4.0-45-generic #66~14.04.1-Ubuntu x86_64 GNU/Linux I'm not eager to install Mint 18 - so I guess we'll have to use Firefox for now. Hopefully the Chrome team can re-mediate this problem soon - as we actually prefer Chrome to FF Thanks
,
Dec 14 2017
I used Synaptic Package Manager to remove 63.0.3239.84 Then,I located google-chrome-stable_62.0.3202.94-1_amd64.deb in /var/cache/apt/archives to reinstall chrome using GDebi Package Installer Subsequently, I am able to access those sites that failed to connect after the update to google-chrome-stable_63.0.3239.84-1_amd64.deb, which as you know includes includes 37 security fixes I can only speculate as to which of the fixes prevented my Mint 17 install from connecting properly .... Could AOL Mail have possibly been affected by CVE-2017-15419: Cross origin leak of redirect URL in Blink ??? Did my banking and similar sites failed to connect due to CVE-2017-15423: Issue with SPAKE implementation in BoringSSL ??? Please remediate these issues before the next stable release Thanks - I appreciate your efforts
,
Dec 14 2017
PS I guess I will have no choice but to use Firefox until the chrome team can properly address the issues described
,
Dec 14 2017
I still cannot reproduce the issue. OP could you bisect for us? We have a utility that makes this very easy, you don't need to do any builds or even have a chromium checkout. $ curl -s --basic -n "https://chromium.googlesource.com/chromium/src/+/master/tools/bisect-builds.py?format=TEXT" | base64 -d > bisect-builds.py $ python tools/bisect-builds.py -a linux64 -g 400000
,
Dec 14 2017
I used the link provided https://chromium.googlesource.com/chromium/src/+/master/tools/bisect-builds.py?format=TEXT It downloaded the file bisect-builds.txt What do I do next?
,
Dec 14 2017
$ python tools/bisect-builds.py -a linux64 -g 400000 python: can't open file 'tools/bisect-builds.py': [Errno 2] No such file or directory
,
Dec 14 2017
please change $ python tools/bisect-builds.py -a linux64 -g 400000 to $ python bisect-builds.py -a linux64 -g 400000
,
Dec 14 2017
Fetching revisions ...
,
Dec 14 2017
Meanwhile - a new browser window - Welcome to Chromium - just launched WTF ????
,
Dec 14 2017
Should I leave Chromium open? OR close it? Terminal running python bisect-builds.py seems stalled .... Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions) Downloading revision 465723...inux_x64/98544// Received 92002024 of 92002024 bytes, 100.00% Bisecting range [400004 (good), 524201 (bad)], roughly 16 steps left. Trying revision 465723...
,
Dec 14 2017
What should happen next ???
,
Dec 14 2017
I'm about to Ctrl-C kill python bisect-builds .... Unless you can explain why i should let it continue !
,
Dec 14 2017
Closed Chromium window and terminal came back with this: Revision 465723 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: What next ?????
,
Dec 14 2017
Guess I'll select q ...
,
Dec 15 2017
The bisect-builds script launches different versions of chromium. You can test if you're able to login, then you close the chromium window. In the prompt, if you were able to login, enter 'g' for good, else 'b' for bad. bisect-builds will then launch another chromium window (with a different version) for you to test. In total, you'll have to test about 15 versions. What the script is doing is narrowing down to a specific version that introduced the bug. At the end, you'll see a message with a url that lists revisions. Post that when you're done.
,
Dec 15 2017
Re comment#8, I guessed Chrome 63 on Ubuntu 14 is working fine for you (from comment#5), thus the difference would be the difference between Ubuntu and Mint. If a popular site like mail.aol.com doesn't work with all Chrome M63, I'd suppose we get a lot more bug reports.
,
Dec 15 2017
Issue 795129 has been merged into this issue.
,
Dec 15 2017
You are probably looking for a change made after 500605 (known good), but no later than 500612 (first known bad). CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/60c34f75f2cc489c71f57a181467ee85d0ed9db0..2385249c97edb9e910fab75a101a71997937f425
,
Dec 15 2017
@thomasanderson Thanks for explaining how to use the bisect-builds script @kochi I'm sorry to say that I have the same problem[s] in Ubuntu 14
,
Dec 15 2017
Also - here is the output when i launch chrome from the terminal and tried mail.col.com then my banking site: /usr/bin/google-chrome-stable %U --no-referrers [13252:13289:1215/030917.080466:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030917.131148:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030917.613623:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030917.790204:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030922.911148:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030922.968573:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030953.104535:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030953.335670:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 [13252:13289:1215/030956.574401:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -107 [13252:13289:1215/030956.780127:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -107
,
Dec 15 2017
c#33: Thanks for the bisect, though that range looks rather benign from my POV. Could you try another bisect to see if you get the same result? You can probably speed up the process if you use: $ python bisect-builds.py -a linux64 -g 400000 -- mail.aol.com +net OWNERS do you know what the error codes in c#35 mean?
,
Dec 15 2017
-101 just means the connection was reset. Please attach a NetLog per the following instructions of the issue. Thanks! https://dev.chromium.org/for-testers/providing-network-details
,
Dec 16 2017
python bisect-builds.py -a linux64 -g 400000 -- mail.aol.com Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions) Downloading revision 465938...inux_x64/98334/e631b58a0c6b3bc7a0cfc5d1143e49b52b/ Received 91994030 of 91994030 bytes, 100.00% Bisecting range [400004 (good), 524573 (bad)], roughly 16 steps left. Trying revision 465938... Revision 465938 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 494306... Received 96493297 of 96493297 bytes, 100.00% Bisecting range [465938 (good), 524573 (bad)], roughly 15 steps left. Trying revision 494306... Revision 494306 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 507699... Received 100847143 of 100847143 bytes, 100.00% Bisecting range [494306 (good), 524573 (bad)], roughly 14 steps left. Trying revision 507699... Revision 507699 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 501118... Received 97222460 of 97222460 bytes, 100.00% Bisecting range [494306 (good), 507699 (bad)], roughly 13 steps left. Trying revision 501118... Revision 501118 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 497821... Received 97008878 of 97008878 bytes, 100.00% Bisecting range [494306 (good), 501118 (bad)], roughly 12 steps left. Trying revision 497821... Revision 497821 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 499539... Received 97088472 of 97088472 bytes, 100.00% Bisecting range [497821 (good), 501118 (bad)], roughly 11 steps left. Trying revision 499539... Revision 499539 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500328... Received 97158446 of 97158446 bytes, 100.00% Bisecting range [499539 (good), 501118 (bad)], roughly 10 steps left. Trying revision 500328... Revision 500328 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500769... Received 97248147 of 97248147 bytes, 100.00% Bisecting range [500328 (good), 501118 (bad)], roughly 9 steps left. Trying revision 500769... Revision 500769 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 500543... Received 97086215 of 97086215 bytes, 100.00% Bisecting range [500328 (good), 500769 (bad)], roughly 8 steps left. Trying revision 500543... Revision 500543 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500612... Received 97342819 of 97342819 bytes, 100.00% Bisecting range [500543 (good), 500769 (bad)], roughly 7 steps left. Trying revision 500612... Revision 500612 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 500575... Received 97093821 of 97093821 bytes, 100.00% Bisecting range [500543 (good), 500612 (bad)], roughly 6 steps left. Trying revision 500575... Revision 500575 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500590... Received 97098355 of 97098355 bytes, 100.00% Bisecting range [500575 (good), 500612 (bad)], roughly 5 steps left. Trying revision 500590... Revision 500590 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500597... Received 97098240 of 97098240 bytes, 100.00% Bisecting range [500590 (good), 500612 (bad)], roughly 4 steps left. Trying revision 500597... Revision 500597 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500602... Received 97098487 of 97098487 bytes, 100.00% Bisecting range [500597 (good), 500612 (bad)], roughly 3 steps left. Trying revision 500602... Revision 500602 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500604... Received 97098503 of 97098503 bytes, 100.00% Bisecting range [500602 (good), 500612 (bad)], roughly 2 steps left. Trying revision 500604... Revision 500604 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500605... Bisecting range [500604 (good), 500612 (bad)], roughly 2 steps left. Trying revision 500605... Revision 500605 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g You are probably looking for a change made after 500605 (known good), but no later than 500612 (first known bad). CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/60c34f75f2cc489c71f57a181467ee85d0ed9db0..2385249c97edb9e910fab75a101a71997937f425 Every time the page re-directed properly to my.screenname.aol.com and loaded the page I selected (g)ood I selected (b)ad every time the browser opened then closed before even any expected error displayed
,
Dec 16 2017
python bisect-builds.py -a linux64 -g 400000 Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions) Downloading revision 465938...inux_x64/98331// Received 91994030 of 91994030 bytes, 100.00% Bisecting range [400004 (good), 524573 (bad)], roughly 16 steps left. Trying revision 465938... Revision 465938 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 494306... Bisecting range [465938 (good), 524573 (bad)], roughly 15 steps left. Trying revision 494306... Revision 494306 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 507699... Received 100847143 of 100847143 bytes, 100.00% Bisecting range [494306 (good), 524573 (bad)], roughly 14 steps left. Trying revision 507699... Revision 507699 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 501118... Received 97222460 of 97222460 bytes, 100.00% Bisecting range [494306 (good), 507699 (bad)], roughly 13 steps left. Trying revision 501118... Revision 501118 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 497821... Bisecting range [494306 (good), 501118 (bad)], roughly 12 steps left. Trying revision 497821... Revision 497821 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 499539... Bisecting range [497821 (good), 501118 (bad)], roughly 11 steps left. Trying revision 499539... Revision 499539 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500328... Bisecting range [499539 (good), 501118 (bad)], roughly 10 steps left. Trying revision 500328... Revision 500328 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500769... Received 97248147 of 97248147 bytes, 100.00% Bisecting range [500328 (good), 501118 (bad)], roughly 9 steps left. Trying revision 500769... Revision 500769 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 500543... Received 97086215 of 97086215 bytes, 100.00% Bisecting range [500328 (good), 500769 (bad)], roughly 8 steps left. Trying revision 500543... Revision 500543 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500612... Received 97342819 of 97342819 bytes, 100.00% Bisecting range [500543 (good), 500769 (bad)], roughly 7 steps left. Trying revision 500612... Revision 500612 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 500575... Received 97093821 of 97093821 bytes, 100.00% Bisecting range [500543 (good), 500612 (bad)], roughly 6 steps left. Trying revision 500575... Revision 500575 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500590... Bisecting range [500575 (good), 500612 (bad)], roughly 5 steps left. Trying revision 500590... Revision 500590 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500597... Received 97098240 of 97098240 bytes, 100.00% Bisecting range [500590 (good), 500612 (bad)], roughly 4 steps left. Trying revision 500597... Revision 500597 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500602... Bisecting range [500597 (good), 500612 (bad)], roughly 3 steps left. Trying revision 500602... Revision 500602 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500604... Bisecting range [500602 (good), 500612 (bad)], roughly 2 steps left. Trying revision 500604... Revision 500604 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 500605... Bisecting range [500604 (good), 500612 (bad)], roughly 2 steps left. Trying revision 500605... Revision 500605 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g You are probably looking for a change made after 500605 (known good), but no later than 500612 (first known bad). CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/60c34f75f2cc489c71f57a181467ee85d0ed9db0..2385249c97edb9e910fab75a101a71997937f425 Every time the page loaded my banking website properly I selected (g)ood I selected (b)ad every time the browser closed before any error displayed
,
Dec 16 2017
Looks like you're getting the same revision range every time. Thanks for repro'ing again. The only thing I see that could cause the issue is the clang update. We've since updated clang again, so the behavior might be fixed in Chrome beta or dev. OP, could you install the beta and unstable channels and test to see if the issue is fixed in either of those? $ sudo apt install google-chrome-beta $ google-chrome-beta mail.aol.com $ sudo apt remove google-chrome-beta $ sudo apt install google-chrome-unstable $ google-chrome-unstable mail.aol.com $ sudo apt remove google-chrome-unstable
,
Dec 16 2017
I admit I'm no expert - but could either of the following changes from M62 to M63 be relevant to the issue[s] we're attempting to correct? I don't have permission to view the pages referenced at : https://chromereleases.googleblog.com/2017/12/stable-channel-update-for-desktop.html#gpluscomments https://crbug.com/780312 CVE-2017-15419: Cross origin leak of redirect URL in Blink https://crbug.com/778101 CVE-2017-15423: Issue with SPAKE implementation in BoringSSL
,
Dec 16 2017
@thomasanderson If I install beta / dev channel versions - will I still be able to use my stable version ? Will my bookmarks be preserved ?
,
Dec 16 2017
> If I install beta / dev channel versions - will I still be able to use my stable version ? yes > Will my bookmarks be preserved ? yes
,
Dec 16 2017
Looks like M64 stable is scheduled to release late Jan 2018 I guess I'll have to tell my wife to continue to use FF till then ...
,
Dec 16 2017
64.0.3282.24 (Official Build) beta (64-bit) loads both pages properly I guess I can leave the beta version installed without any ill effects ?
,
Dec 16 2017
> 64.0.3282.24 (Official Build) beta (64-bit) loads both pages properly Cool, glad to hear it's (somehow) fixed. To give further insight into how exactly it got fixed, could you do another bisect-builds? This time like this: $ python bisect-builds.py -a linux64 -g 520840 -b 499098 -- mail.aol.com > When will M64 be released to the stable channel? I can't give an exact date, but there's a new major release about every 6 weeks, and version 63 hit stable 2017-12-06. > I guess I can leave the beta version installed without any ill effects ? That's correct.
,
Dec 16 2017
running python bisect-builds.py -a linux64 -g 520840 -b 499098 -- mail.aol.com now ....
,
Dec 16 2017
python bisect-builds.py -a linux64 -g 520840 -b 499098 -- mail.aol.com Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions) Downloading revision 508562...inux_x64/98314// Received 100898224 of 100898224 bytes, 100.00% Bisecting range [499100 (bad), 520836 (good)], roughly 13 steps left. Trying revision 508562... Revision 508562 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 514032... Received 99917292 of 99917292 bytes, 100.00% Bisecting range [508562 (bad), 520836 (good)], roughly 12 steps left. Trying revision 514032... Revision 514032 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 517086... Received 100057225 of 100057225 bytes, 100.00% Bisecting range [514032 (bad), 520836 (good)], roughly 11 steps left. Trying revision 517086... Revision 517086 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 515561... Received 100012786 of 100012786 bytes, 100.00% Bisecting range [514032 (bad), 517086 (good)], roughly 10 steps left. Trying revision 515561... Revision 515561 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 516235... Received 100561618 of 100561618 bytes, 100.00% Bisecting range [515561 (bad), 517086 (good)], roughly 9 steps left. Trying revision 516235... Revision 516235 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 516673... Received 100022860 of 100022860 bytes, 100.00% Bisecting range [516235 (bad), 517086 (good)], roughly 8 steps left. Trying revision 516673... Revision 516673 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 516443... Received 100595390 of 100595390 bytes, 100.00% Bisecting range [516235 (bad), 516673 (good)], roughly 7 steps left. Trying revision 516443... Revision 516443 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 516594... Received 100568904 of 100568904 bytes, 100.00% Bisecting range [516443 (bad), 516673 (good)], roughly 6 steps left. Trying revision 516594... Revision 516594 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b Downloading revision 516649... Received 100016573 of 100016573 bytes, 100.00% Bisecting range [516594 (bad), 516673 (good)], roughly 5 steps left. Trying revision 516649... Revision 516649 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 516630... Received 100451709 of 100451709 bytes, 100.00% Bisecting range [516594 (bad), 516649 (good)], roughly 4 steps left. Trying revision 516630... Revision 516630 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 516618... Received 100450509 of 100450509 bytes, 100.00% Bisecting range [516594 (bad), 516630 (good)], roughly 3 steps left. Trying revision 516618... Revision 516618 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g Downloading revision 516605... Received 100456821 of 100456821 bytes, 100.00% Bisecting range [516594 (bad), 516618 (good)], roughly 2 steps left. Trying revision 516605... Revision 516605 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g You are probably looking for a change made after 516594 (known bad), but no later than 516605 (first known good). CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/463c09ce2821765018dd3de8a9492bc0be936847..1f8efcca81bd1cbdc9edd78ea67ddd4902abc909
,
Dec 16 2017
Does that NetLog help any ??
,
Dec 16 2017
I defer to the net folks for parsing the netlog.
,
Dec 16 2017
+ajwong This issue regressed somwhere here: https://chromium.googlesource.com/chromium/src/+log/60c34f75f2cc489c71f57a181467ee85d0ed9db0..2385249c97edb9e910fab75a101a71997937f425 And was fixed here: https://chromium.googlesource.com/chromium/src/+log/463c09ce2821765018dd3de8a9492bc0be936847..1f8efcca81bd1cbdc9edd78ea67ddd4902abc909 Do you think your frame pointer change was what fixed this? I'm guessing the clang roll is what caused the regression.
,
Dec 16 2017
I'm logging out of Mint 17 to boot into Ubuntu 14 and install beta to see if it works there too I'll update you on the result
,
Dec 16 2017
No joy for Ubuntu 14.04.5 LTS in chronme-stable or beta : You are probably looking for a change made after 516594 (known bad), but no later than 516605 (first known good). CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/463c09ce2821765018dd3de8a9492bc0be936847..1f8efcca81bd1cbdc9edd78ea67ddd4902abc909 Same bisect as Mint 17 - but it's curious that beta worked in Mint but not Ubuntu
,
Dec 16 2017
update to chrome 64.0.3282.24 beta now works in Ubuntu 14
,
Dec 18 2017
,
Dec 18 2017
Fascinating. The NetLog suggests the server is resetting the connection after we send our second handshake flight. This suggests that whatever broke, it's quite low-level. That does indeed suggest the compiler changes (for lack of anything else plausible-seeming), which is a little distressing. Or one of those changes stomped on memory somewhere. This server is using the legacy static RSA ciphers, so that's a ClientKeyExchange (encrypted premaster secret), and then an encrypted Finished message. If I mangle the contents of the Finished message, this server sends back decrypt_error. If I mangle the record encryption, the server resets the connection. That suggests that the breakage is there. This server is using TLS_RSA_WITH_AES_256_CBC_SHA, so it'd be something in the AES code or the HMAC-SHA1 code. BoringSSL has lots of implementations of these primitives based on hardware capabilities. Finding out which got run might help point at things... what does cat /proc/cpuinfo give?
,
Dec 19 2017
cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 20 model : 1 model name : AMD C-50 Processor stepping : 0 microcode : 0x5000025 cpu MHz : 800.000 cache size : 512 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 6 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt hw_pstate vmmcall arat npt lbrv svm_lock nrip_save pausefilter bugs : fxsave_leak sysret_ss_attrs bogomips : 1994.93 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate processor : 1 vendor_id : AuthenticAMD cpu family : 20 model : 1 model name : AMD C-50 Processor stepping : 0 microcode : 0x5000025 cpu MHz : 800.000 cache size : 512 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 6 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt hw_pstate vmmcall arat npt lbrv svm_lock nrip_save pausefilter bugs : fxsave_leak sysret_ss_attrs bogomips : 1994.93 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate
,
Dec 19 2017
Thanks! Given that, I think you would be using the bsaes-x86_64.pl implementation and the SSSE3 SHA-1 implementation. Though both are in assembly, so it's quite puzzling how either would start breaking on the C compiler...
,
Dec 19 2017
@#53: The framepointer change *shouldn't* have caused any noticeable behavior difference. If it did, it's because the shifts in codegen either masked a real compiler bug, or shifting things such that some sort of memory or memory stomp doesn't blow things up. @#63: It's not 100% correct assumption that the assembly implementation won't interact badly with the C-compiler. You could imagine a bug in the assembly that didn't correctly save a register or otherwise violated the x86_64 ABI such that it is sensitive to the specific codegen at the call site.
,
Dec 19 2017
That's a good point. Both of those files have been in OpenSSL forever, so hopefully they've had those sorts of ABI bugs ironed out, but it wouldn't be the first time OpenSSL's assembly's been buggy...
,
Dec 19 2017
Okay, well, let's try this... IMPORTANT: These instructions will log all the traffic Chrome's net stack sends and receives over the network, the corresponding TLS session keys that decrypt it, and the unencrypted data. Obviously, we don't want to log this from your main profile, which contains all kinds of state, so I'm going to have you pass --user-data-dir to Chrome. That points it at a different profile directory entirely, so it won't load cookies, etc. You can even run both instances at the same time. My goal is to confirm whether or not something got encrypted wrong and, if it did, which part was wrong. So I'm going to check the crypto and make sure it's all correct. 1. Run google-chrome --user-data-dir=/tmp/whatever --ssl-key-log-file=sslkeylog.txt 2. Don't sign in to anything from that browser session, of course. :-) 3. Go to chrome://net-export as before, but turn on "Include raw bytes (will include cookies and credentials)" before enabling logging. 4. Reproduce the issue. Just visiting https://my.screenname.aol.com should suffice. 5. Exit Chrome. 6. Email me both the NetLog and sslkeylog.txt. Would you be willing to do that? This log is a bit more invasive than what we'd previously tried, so I understand if you would rather not.
,
Dec 19 2017
> I think you would be using the bsaes-x86_64.pl implementation and the SSSE3 SHA-1 implementation. Correction: encryption would be vpaes-x86_64.pl. That file never saves and restores anything, but it also never touches any callee-saved registers. :-) I also attempted to transcribe the capabilities from /proc/cpuinfo to the OpenSSL thing (running M63 with OPENSSL_ia32cap=36066180540922879:0, because apparently we broke expressing that in hex at some point!), but that didn't reproduce it. So either I transcribed that wrong or something else is going on.
,
Dec 20 2017
@davidben @#66: NetLog and sslkeylog.txt should be in your email
,
Dec 21 2017
Thanks you very much for that. It was extremely helpful! This bug is fascinating. I checked the crypto and that does confirm the theory that something got encrypted wrong. Specifically, of the three blocks in the ciphertext, the middle one is garbage, but first and third are correct! This is particularly interesting because, to compute the third block, you need to have correctly computed the second block. Thus the AES code encrypted things correctly, and even passed it as IV to the third block correctly, but somehow the output got messed up. We did appear to have a small typo in e_tls.c (which I'll go fix), but it wouldn't have broken things. We were merely missing out on a tiny optimization. (Though it could have exacerbated whatever's going on here.) In particular, the second block gets encrypted to a buffer on the stack and then copied to the output buffer, rather than being encrypted straight to the output buffer. The incorrect middle block is interesting, because it is the same for all three sockets in the NetLog and has what appears to be a return address on it. The last three hex digits (remaining would have been randomized) match the return address for this memcpy call, the one that was supposed to write the block out! https://boringssl.googlesource.com/boringssl/+/3239/crypto/cipher_extra/e_tls.c#206 Thus it appears that, for whatever reason, buf (which is normally rsp+0x30) is being passed as rsp-8. My guess is something is causing one of the other registers that computation depends on to go wrong. Exactly what went wrong is unclear. The two compiler changes probably changed things around enough to tickle it. Honestly, given that we're not seeing widespread breakage (both in bug reports and our metrics) and that mimicking your CPU's capabilities locally did not reproduce this issue, I suspect it may be something odd with your CPU or machine. Could you try the following: 1. Update to the latest stable Chrome, 63.0.3239.108, if you haven't already. (If you can't update to a later version for whatever reason, let me know what version you've got installed and I'll adjust the instructions. The number in step 3 was computed for 63.0.3239.108.) 2. Install gdb if you don't have it, and run "gdb --args /opt/google/chrome/chrome --user-data-dir=/tmp/blah" 3. Type "b *_start+0x1eb038f" and press enter. (This sets a breakpoint to shortly before the call to memcpy in question.) 4. Run "r" to start Chrome with the breakpoint. 5. Load https://my.screenname.aol.com. At this point, it should freeze because you've tripped the breakpoint. 6. Go back to your gdb session. Run "disassemble $rip,$rip+50". This is just to sanity-check that we got the right address. 7. Run "info reg". This is the command I'm interested in seeing the output of. 8. Run "q" to exit gdb. Thanks!
,
Dec 21 2017
Er, after you've run all that, could you paste the gdb session from your terminal? I'm interested in the output of disassemble and info reg, but may as include the whole thing in case there are some errors from gdb of note somewhere.
,
Dec 21 2017
@davidben @#71+72: gdb output for 63.0.3239.108 should be in your email
,
Dec 21 2017
PS I do appreciate all the effort by all the chrome team ... Happy Holidays to each of you and those you love
,
Dec 22 2017
@davidben @#71: Have you had a chance to look over the gdb output I emailed? Did it reveal any thing interesting?
,
Dec 27 2017
I believe davidben is out for the holidays until next week. Unfortunately we haven't had much luck figuring out what's wrong, as near as we can tell, those sites use a specific cipher suite that's tripping up crypto operations on your device. Unfortunately we haven't been able to reproduce locally. Is the laptop hardware a custom build or some standard configuration? If its a standard configuration, can you provide the make/model to see if we can track down the corresponding hardware and see if we can reproduce on that.
,
Dec 27 2017
The laptops are Gateway branded model NV51B08u [Acer][manufactured in China in 2011]
,
Dec 27 2017
FWIW - when i launch chrome from the terminal and try mail.aol.com in the browser - i get this in the terminal: [13252:13289:1215/030917.080466:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -101 when i launch chrome from the terminal and try my_bank.com in the browser - i get this in the terminal: [13252:13289:1215/030956.574401:ERROR:ssl_client_socket_impl.cc(1084)] handshake failed; returned -1, SSL error code 1, net_error -107 different net_error numbers - 101 in the first case - 107 in the second
,
Jan 10 2018
The websites referenced will open in 64.0.3282.71 beta They still won't open in 63.0.3239.132 stable
,
Jan 11 2018
@davidben @#71: Have you had a chance to look over the gdb output I emailed? Did it reveal any thing interesting?
,
Jan 12 2018
@mtolle: Just do you know, davidben@'s OOO at the moment--I hope he'll get back to you soon.
,
Jan 22 2018
Does the default for the chrome://flag[s] setting for the TLS 1.3 variant used in M63 stable match the default setting for the 1.3 variant used in M64 beta? I ask because the problem began for me after a TLS 1.3 experiment was apparently implemented in M63 stable - and seems to be fixed in M64 beta. Would one of the experiments available in the drop-down possibly remediate the issue in M63?
,
Jan 22 2018
I can only guess that the debugger output i emailed to @davidben either revealed something horribly broken or is totally unremarkable. Clearly since i don't understand the debugging output - it's possible i wouldn't understand an attempt at explanation for the layperson anyway. Perhaps i should just wait and see what happens when M64 is promoted to stable.
,
Jan 22 2018
This does not have anything to do with TLS 1.3. The cipher you're hitting problems with is a legacy insecure CBC mode cipher, which have all been removed in TLS 1.3. Yeah, the debugger output was puzzling as it didn't show any of the wrong values I was expecting. I'll see if we can figure out another gdb command to try after I've caught up on my post-vacation email, but honestly it seems quite likely to just be a CPU bug somewhere.
,
Jan 26 2018
The sites that failed to load properly in M63 stable now load properly in M64 stable.
,
Mar 20 2018
Oops, we never got around to following up on this, but I guess it's since been resolved by way of the various other changes. :-/ I'll go ahead and close this. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by kochi@chromium.org
, Dec 11 2017