New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 31 users

Issue metadata

Status: Fixed
Owner:
Email to this user bounced
Closed: Sep 2012
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Restricted
  • Only users with EditIssue permission may comment.



Sign in to add a comment
link

Issue 69813: Very slow connection to gmail and slow uploads over SSL

Reported by rvargas@chromium.org, Jan 16 2011 Project Member

Issue description

The initial connection to gmail is very slow form time to time (to get to the login page).

I guess I screwed up the log, but here's what I have:

The request:

t=1295145179046 [st=    0] +REQUEST_ALIVE                             [dt=54765]
t=1295145179046 [st=    0]     URL_REQUEST_START_JOB                  [dt=    6]
                               --> load_flags = 1114240 (ENABLE_LOAD_TIMING | MAIN_FRAME | VERIFY_EV_CERT)
                               --> method = "GET"                   
                               --> priority = 0                     
                               --> url = "http://gmail.google.com/mail/"
t=1295145179052 [st=    6]    +URL_REQUEST_START_JOB                  [dt=   78]
                               --> load_flags = 1114240 (ENABLE_LOAD_TIMING | MAIN_FRAME | VERIFY_EV_CERT)
                               --> method = "GET"                   
                               --> priority = 0                     
                               --> url = "http://gmail.google.com/mail/"
t=1295145179056 [st=   10]        HTTP_CACHE_GET_BACKEND              [dt=    0]
t=1295145179056 [st=   10]        HTTP_CACHE_OPEN_ENTRY               [dt=    1]
                                  --> net_error = -2 (FAILED)       
t=1295145179057 [st=   11]        HTTP_CACHE_CREATE_ENTRY             [dt=    1]
t=1295145179058 [st=   12]        HTTP_CACHE_ADD_TO_ENTRY             [dt=    0]
t=1295145179058 [st=   12]       +PROXY_SERVICE                       [dt=    0]
t=1295145179058 [st=   12]           PROXY_SERVICE_RESOLVED_PROXY_LIST  
                                     --> pac_string = "DIRECT"      
t=1295145179058 [st=   12]       -PROXY_SERVICE                       
t=1295145179058 [st=   12]        TCP_CLIENT_SOCKET_POOL_REQUESTED_SOCKET  
                                  --> host_and_port = "gmail.google.com:80"
t=1295145179058 [st=   12]       +SOCKET_POOL                         [dt=    3]
t=1295145179058 [st=   12]           SOCKET_POOL_REUSED_AN_EXISTING_SOCKET  
                                     --> idle_ms = 78               
t=1295145179061 [st=   15]           SOCKET_POOL_BOUND_TO_SOCKET      
                                     --> source_dependency = {"id":10838,"type":5}
t=1295145179061 [st=   15]       -SOCKET_POOL                         
t=1295145179061 [st=   15]       +HTTP_TRANSACTION_SEND_REQUEST       [dt=    4]
t=1295145179061 [st=   15]           HTTP_TRANSACTION_SEND_REQUEST_HEADERS  
                                     --> GET /mail/ HTTP/1.1        
                                         Host: gmail.google.com     
                                         Connection: keep-alive     
                                         Referer: http://gmail.google.com/
                                         Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
                                         User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.634.0 Safari/534.16
                                         Accept-Encoding: gzip,deflate,sdch
                                         Accept-Language: en-US,en;q=0.8,es;q=0.6
                                         Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
                                         Cookie: ...
t=1295145179065 [st=   19]       -HTTP_TRANSACTION_SEND_REQUEST       
t=1295145179065 [st=   19]       +HTTP_TRANSACTION_READ_HEADERS       [dt=   63]
t=1295145179065 [st=   19]           HTTP_STREAM_PARSER_READ_HEADERS  [dt=   63]
t=1295145179128 [st=   82]           HTTP_TRANSACTION_READ_RESPONSE_HEADERS  
                                     --> HTTP/1.1 302 Moved Temporarily
                                         Content-Type: text/html; charset=UTF-8
                                         Cache-Control: no-cache, no-store, max-age=0, must-revalidate
                                         Pragma: no-cache           
                                         Expires: Fri, 01 Jan 1990 00:00:00 GMT
                                         Date: Sun, 16 Jan 2011 02:32:57 GMT
                                         Location: https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1eic6yu9oa4y3&scc=1&ltmpl=default&ltmplcache=2
                                         Content-Length: 409        
                                         X-Content-Type-Options: nosniff
                                         X-Frame-Options: SAMEORIGIN
                                         X-XSS-Protection: 1; mode=block
                                         Server: GSE                
t=1295145179128 [st=   82]       -HTTP_TRANSACTION_READ_HEADERS       
t=1295145179128 [st=   82]       +HTTP_CACHE_WRITE_INFO               [dt=    2]
t=1295145179130 [st=   84]           URL_REQUEST_REDIRECTED           
                                     --> location = "https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1eic6yu9oa4y3&scc=1&ltmpl=default&ltmplcache=2"
t=1295145179130 [st=   84]    -URL_REQUEST_START_JOB                  
t=1295145179131 [st=   85]     URL_REQUEST_START_JOB                  [dt=    3]
                               --> load_flags = 1114240 (ENABLE_LOAD_TIMING | MAIN_FRAME | VERIFY_EV_CERT)
                               --> method = "GET"                   
                               --> priority = 0                     
                               --> url = "https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1eic6yu9oa4y3&scc=1&ltmpl=default&ltmplcache=2"
t=1295145179134 [st=   88]    +URL_REQUEST_START_JOB                  [dt=54649]
                               --> load_flags = 1114240 (ENABLE_LOAD_TIMING | MAIN_FRAME | VERIFY_EV_CERT)
                               --> method = "GET"                   
                               --> priority = 0                     
                               --> url = "https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1eic6yu9oa4y3&scc=1&ltmpl=default&ltmplcache=2"
t=1295145179135 [st=   89]        HTTP_CACHE_GET_BACKEND              [dt=    0]
t=1295145179135 [st=   89]        HTTP_CACHE_OPEN_ENTRY               [dt=    0]
                                  --> net_error = -2 (FAILED)       
t=1295145179135 [st=   89]        HTTP_CACHE_CREATE_ENTRY             [dt=    1]
t=1295145179136 [st=   90]        HTTP_CACHE_ADD_TO_ENTRY             [dt=    0]
t=1295145179136 [st=   90]       +PROXY_SERVICE                       [dt=    0]
t=1295145179136 [st=   90]           PROXY_SERVICE_RESOLVED_PROXY_LIST  
                                     --> pac_string = "DIRECT"      
t=1295145179136 [st=   90]       -PROXY_SERVICE                       
t=1295145179136 [st=   90]        SPDY_SESSION_POOL_FOUND_EXISTING_SESSION  
                                  --> session = {"id":78,"type":6}  
t=1295145179136 [st=   90]        HTTP_TRANSACTION_SEND_REQUEST       [dt=    1]
t=1295145179137 [st=   91]        HTTP_TRANSACTION_READ_HEADERS       [dt=54480]
t=1295145233617 [st=54571]       +PROXY_SERVICE                       [dt=    0]
t=1295145233617 [st=54571]           PROXY_SERVICE_RESOLVED_PROXY_LIST  
                                     --> pac_string = "DIRECT"      
t=1295145233617 [st=54571]       -PROXY_SERVICE                       
t=1295145233617 [st=54571]       +SOCKET_POOL                         [dt=   60]
t=1295145233677 [st=54631]           SOCKET_POOL_BOUND_TO_CONNECT_JOB  
                                     --> source_dependency = {"id":10955,"type":4}
t=1295145233677 [st=54631]           SOCKET_POOL_BOUND_TO_SOCKET      
                                     --> source_dependency = {"id":10960,"type":5}
t=1295145233677 [st=54631]       -SOCKET_POOL                         
t=1295145233677 [st=54631]        SPDY_SESSION_POOL_IMPORTED_SESSION_FROM_SOCKET  
                                  --> session = {"id":10961,"type":6}
t=1295145233678 [st=54632]        HTTP_TRANSACTION_SEND_REQUEST       [dt=    1]
t=1295145233679 [st=54633]       +HTTP_TRANSACTION_READ_HEADERS       [dt=  104]
t=1295145233783 [st=54737]           HTTP_TRANSACTION_READ_RESPONSE_HEADERS  
                                     --> HTTP/1.1 200 OK            
                                         cache-control: no-cache, no-store
                                         content-encoding: gzip     
                                         content-length: 6721       
                                         content-type: text/html; charset=UTF-8
                                         date: Sun, 16 Jan 2011 02:33:52 GMT
                                         expires: Mon, 01-Jan-1990 00:00:00 GMT
                                         pragma: no-cache           
                                         server: GSE                
                                         set-cookie: GoogleAccountsLocale_session=en; Secure
                                         set-cookie: GALX=HSz10cmg7r4;Path=/accounts;Secure
                                         status: 200 OK             
                                         version: HTTP/1.1          
                                         x-auto-login: realm=com.google&args=service%3Dmail%26continue%3Dhttp%253A%252F%252Fmail.google.com%252Fmail%252F%253Fui%253Dhtml%2526zy%253Dl
                                         x-content-type-options: nosniff
                                         x-xss-protection: 1; mode=block
t=1295145233783 [st=54737]       -HTTP_TRANSACTION_READ_HEADERS       
t=1295145233783 [st=54737]        HTTP_CACHE_WRITE_INFO               [dt=    0]
t=1295145233783 [st=54737]    -URL_REQUEST_START_JOB                  
t=1295145233783 [st=54737]     HTTP_TRANSACTION_READ_BODY             [dt=    2]
t=1295145233785 [st=54739]     HTTP_TRANSACTION_READ_BODY             [dt=    7]
t=1295145233792 [st=54746]     HTTP_TRANSACTION_READ_BODY             [dt=   16]
t=1295145233808 [st=54762]     HTTP_TRANSACTION_READ_BODY             [dt=    2]
t=1295145233811 [st=54765]     HTTP_TRANSACTION_READ_BODY             [dt=    0]
t=1295145233811 [st=54765] -REQUEST_ALIVE                             

And the speedy session referenced above (id:78):

t=1295145178868 [st=    0]  SPDY_SESSION_SEND_RST_STREAM  
                            --> status = 5
                            --> stream_id = 9
t=1295145179137 [st=  269]  SPDY_SESSION_SYN_STREAM  
                            --> flags = 1
                            --> accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
                                accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
                                accept-encoding: gzip,deflate,sdch
                                accept-language: en-US,en;q=0.8,es;q=0.6
                                cookie: __utmx=173272373.; __utmxx=173272373.; __utmz=173272373.1288374289.46.1.utmcsr=gmail.google.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utma=173272373.1727144503.1212354590.1292206081.1294897954.51; rememberme=false; PREF=ID=038c731835e5e5fb:U=73b673aaf2f09571:TM=1247892941:LM=1281239789:GM=1:GC=1:S=qXdHl6UEAVcUe5a1; NID=42=PwIYh4eYlBsZV3QSkC7Vbvnb7ggJS4_CN6pKvqZH0kUPOEnry6eVxZeihD3Myr2xFmKYYgZMkCLuAk3ZdzupSwOw4UMiRqbNYk_lxMatm8LlFqgn9CRTaDggs24Hbblw
                                host: www.google.com
                                method: GET
                                referer: http://gmail.google.com/
                                scheme: https
                                url: /accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1eic6yu9oa4y3&scc=1&ltmpl=default&ltmplcache=2
                                user-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.634.0 Safari/534.16
                                version: HTTP/1.1
                            --> id = 11
t=1295145233617 [st=54749]  SPDY_SESSION_CLOSE  
                            --> status = -101
t=1295145233617 [st=54749]  SPDY_SESSION_POOL_REMOVE_SESSION  
                            --> session = {"id":78,"type":6}
t=1295145233620 [st=54752] -SPDY_SESSION
 

Comment 1 by willchan@chromium.org, Jan 16 2011

Status: Assigned
I don't have a full diagnosis, but afaict, what's happening is we're trying to use a SPDY session, but that session may have been timed out or something.  In fact, we get a ERR_CONNECTION_RESET after we 54 (!) seconds.  I have no idea what is taking so long to get the TCP RST.  This is causing the slow load.  After it failed with an ERR_CONNECTION_RESET, we try a new connection (since we have logic to retry on ERR_CONNECTION_{RESET|ABORTED|CLOSED), which succeeds quickly.

The main question is, why is it taking us so long to get a TCP RST on the socket?  Or are we getting it early, but the SPDY code isn't propagating it up?  It'd be nice if the net-internals had the full SPDY_SESSION information.  It appears to have been truncated from earlier.  It'd be nice to see if there were previous successful transactions on the SPDY session.  It'd also be nice to get a packet trace to see if the TCP RST is just taking a long time to arrive, or if it arrived earlier and the Chrome SPDY code isn't delivering it up the stack properly.

Ricardo, are you able to repro this bug?  If not, I suspect we may have to just chalk this up to one of those unexplained mysteries.  It may also be that the ERR_CONNECTION_RESET is the server terminating the connection.  Indeed, I think the GFEs are probably using 60s idle timers or so.  But I'm not entirely sure.

Comment 2 by rvargas@chromium.org, Jan 18 2011

I cannot reproduce it consistently, but I saw it a couple of times last week. I'll try to get a better log next time.

Comment 3 by jrsta...@gmail.com, Jul 11 2011

We consistently get this in my lab. How would one go about logging this so that I might be able to help by providing more data?

Comment 4 by rvargas@chromium.org, Jul 11 2011

jrsta...@gmail.com:

1. Navigate to about:net-internals and leave that tab open 
2. Repro the bug
3. Use data/dump from the net-internals page

It is possible to have some data going to about:net-internals after noticing a problem, but the log is more comprehensive if it's enabled beforehand.

Comment 5 by jrsta...@gmail.com, Jul 12 2011

Here's my log.  At least it is consistent in my lab. The first connection to gmail after opening chrome will take an excruciating long time.  In this case, I initiated the connection at 16:58:15, and it took until about 16:59:38 until I actually saw my list of messages.

I figured that completeness was of more value that simplicity, so I attached my log.
net_internals (4).log
340 KB View Download

Comment 6 by rvargas@chromium.org, Jul 12 2011

Cc: -willchan@chromium.org mbelshe@chromium.org
Owner: willchan@chromium.org
I don't see anything obviously wrong here (URL_REQUEST (id=192), SPDY_SESSION (id=208)), except that it is very slow.

Will: any idea?

Comment 7 by willchan@chromium.org, Jul 13 2011

I took a look, it doesn't seem obviously broken to me either. Things just look slow. In your lab, doesn't this problem only occur with Chrome, or does it happen to other browsers too?

Comment 8 by jrsta...@gmail.com, Jul 13 2011

Not only does it seemed localized to chrome, but a colleague found that using the flags "--use-spdy=off  --use-system-ssl" caused the problem to go away entirely.

Comment 9 by willchan@chromium.org, Jul 13 2011

Interesting, I wonder if it's SPDY related or NSS related. Please try just use-spdy=off without --use-system-ssl and let me know how it performs then.

Comment 10 by willchan@chromium.org, Jul 13 2011

Cc: agl@chromium.org
I took a look at the net-internals dump more closely. I'm not sure, but it looks like there's a pretty high SSL overhead. I wonder if there's something going on with the server we're connecting to. Let me add agl@ to see if see this too.

I think we need a wireshark trace too if you know how to get one. Please provide both a new net-internals dump and a wireshark trace when you repro again and post both, so we can corroborate data in them.

Comment 11 by agl@chromium.org, Jul 13 2011

From looking at the SSL_CONNECT times in that log, nothing seems too terrible but I'm happy to look at packet dumps with more detail.

Comment 12 by willchan@chromium.org, Jul 13 2011

I was comparing SSL_SOCKET_BYTES_RECEIVED vs SOCKET_BYTES_RECEIVED. Sometimes I see over a thousand bytes read with only 8 bytes read from the ssl socket. Not sure if that's normal.

Comment 13 by agl@chromium.org, Jul 13 2011

The SSL socket cannot return data until it has received a whole record as the MAC comes at the end of the record. If the server has a packing problem, then that would cause it. I could confirm with a packet trace.

Comment 14 by jrsta...@gmail.com, Jul 13 2011

Do you want the full wireshark? There seems to be a really high amount of noise here (even after turning off most background programs) is there any filtering you'd like me to do while capturing?

Yesterday I changed the setup on my machine. I'm now running 13.0.782.55 beta, and 14.0.821.0 canary. Beta seems to be able to connect to Gmail fine now, but canary just gets stuck on the loading page with the message "This is taking longer than usual. Try reloading the page". I've tried reloading, and disabling labs, doesn't seem to make a difference.

Comment 15 by willchan@chromium.org, Jul 14 2011

Yes, the full wireshark dump. We're pretty good at filtering, especially since the interaction is solely on a single TCP connection. Remember to give us both the wireshark dump and net-internals so we can corroborate them. And it is only necessary to do it for a buggy version of chrome, probably the canary.

Comment 16 by jrsta...@gmail.com, Jul 14 2011

I also happen to be following  issue 84704 , due to some problems with Google Chat on Google+. There were reports that Adblock happens to interfere with Google Chat, so I disabled Adblock in canary, now I can get into Gmail without issue. 

Although I don't believe this is what caused the original bug, I can no longer reproduce it.

Comment 17 by willchan@chromium.org, Jul 14 2011

OK, thanks for keeping us posted. Please let us know if this happens again. We'd love to get to the bottom of this.

Comment 18 by jrsta...@gmail.com, Sep 1 2011

Today the issue with connecting to Gmail reared it ugly head. But it was not just gmail, it was also google App engine, google plus, google docs. 

As mentioned before, I tried the solution my colleague mentioned, with adding the flag to the command -use-spdy=off and -use-system-ssl. With these flags, the problem completely disappeared, but without them, it was slow as molasses.

I am running Google Chrome 15.0.859.0 canary and 14.0.835.126 beta-m. Both have this issue currently. I ran it in canary and capture the net-internals log and a wireshark capture. Hope that helps.
297d14e2-7f87-4409-b4e5-dfe342635d75
951 KB View Download
gmail-capture.pcap
735 KB Download

Comment 19 by willchan@chromium.org, Sep 1 2011

Thanks for the logs! It looks like server side issues. At the TCP level, ACKs are coming back promptly, but the server isn't sending responses. I don't think this is a Chrome issue, I'm going to talk to some Google server folks to see what's up.

Comment 20 by jrsta...@gmail.com, Sep 2 2011

Glad we could finally nail it down to something at least. I'd love to know what the problem ended up being, so I could tell my colleagues it's safe to turn spdy back on.

Comment 21 by jrsta...@gmail.com, Sep 2 2011

Hello Will,

I understand that this isn't a problem with Chrome per say, but from a naive user's point of view, as this problem will exhibit itself in Chrome (which has spdy enabled by default) and not in other browsers that may not, it will appear as a Chrome problem. 

I know this is a very naive request, but would there be any way that chrome could try spdy first, and if the connection seems really slow, suggest to the user to try connecting without spdy? As I mentioned before, by adding to flags to my command execution, the problem goes away entirely, but this is not something that an 'average' user could probably due, and thus reflects badly on Chrome. May be there is some other way that this could be exposed to the user so that they could turn it off if there is a problem with it.

Comment 22 by willchan@chromium.org, Sep 2 2011

Thanks for your comment here. I'm a bit hesitant to do that since slow connections could be caused by many things, not just SPDY. And while we've seen a few other issues we're working on wrt SPDY client-side, this is the first issue we've seen with it server-side. This happens for all Chrome<=>Google https sites, which has pretty high visibility, so if this were a bigger issue, I'd assume we'd have seen more bug reports or complaints on our support forums. Given this, I'm not sure extra workarounds are necessary.

If you know more users who see this, I'd appreciate hearing from them too and getting their logs as well. I'm also curious if there's something specific to your networks or whatever that causes you guys to trigger this server behavior. So, if you can find others, especially folks in different regions, who have the same issue, I'd appreciate hearing from them and getting their logs.

Comment 23 Deleted

Comment 24 by Deleted ...@, Sep 7 2011

My Gmail /gchat was also very slow, only in Chrome (13.0.782.220) on MacOS.  Disabling spdy as described here: http://www.jamiestarke.com/2011/09/02/are-googles-services-running-slow-in-chrome-try-disabling-spdy/
fixed the issue for me.

Comment 25 by sop...@gmail.com, Sep 7 2011

I tried both -use-spdy=off and --use-spdy=off in my chrome 15, neither made the SPDY tab on net-internals say SPDY was disabled, and I still had troubel with gmail. Has SPDY-disabling changed?

I see this problem at least 10 times per day!

Comment 26 by willchan@chromium.org, Sep 7 2011

@24: Can you post the net-internals and wireshark traces like Jamie did in comment 18? Note it's important that you start the net-internals and wireshark traces *before* you reproduce the bug (before you establish the connection to Google).

@Jamie: Can you update your blogpost so you include those instructions when updating the bug report?

@25: Are you sure Chrome is slowing gmail down for you? Have you tried other browsers? I just want to make sure you've identified Chrome as the culprit, as opposed to Gmail or your ISP or whatever being at fault.

Comment 27 by jrsta...@gmail.com, Sep 7 2011

Sure thing, I'll try to make a much more conclusive version tonight including trying other browsers to verify that it is chrome, checking spdy, and performing log collection. 

I guess before I invested all the time into perfecting, I wanted to make sure I wasn't alone first.

Comment 28 by willchan@chromium.org, Sep 7 2011

Great! I appreciate your help in narrowing down the specific issue here. I'm also discussing with Google server folks to see if they know anything about this.

Comment 29 by jrsta...@gmail.com, Sep 8 2011

I put together a tutorial on how to collect network logs for Chrome Issues that I hope helps: http://www.jamiestarke.com/2011/09/08/how-to-collect-a-network-log-for-google-chrome/

Comment 30 by d...@tapad.com, Sep 9 2011

We're seeing the same problem here. Over the past 2 months, most of our people have moved to Firefox as all Google services suffer from the same issues (all other services are fine, and there are no issues with other browsers). Attaching a network log where GMail did not respond timely.
GMail SSL network capture.log.zip
112 KB Download

Comment 31 by willchan@chromium.org, Sep 13 2011

Jamie, have you narrowed this down any more?

#30: I looked at your log and it looks like primarily GMail is at fault in this one. All other services seem fine.

Honestly, at this point, I'm a bit suspicious it's a Chrome/SPDY issue here. I think you guys are getting unlucky and hitting bad datacenters for those services. Are these all temporary issues that go away eventually? In this log you posted it's taking a whopping 40s to get a response from Google via GMail, other Google services respond in milliseconds. Therefore I doubt it's specific to SPDY, since Chrome is using SPDY for a number of those services but it's only GMail in this log. If these are all sporadic issues, I'm more inclined to blame Google backends. Let me know if the delays actually go away when people switch to Firefox.

Comment 32 by jrsta...@gmail.com, Sep 13 2011

I haven't narrowed it down any further.  I haven't been in the lab the last few days, but when I was there Thursday it was still an issue. If you'd like, I can try to grab a log like #30 did including docs, and plus, as I found them to be just as bad offenders.

Comment 33 by jrsta...@gmail.com, Sep 16 2011

Well, I have good news and bad news. I just spent the last hour and a half or so, but I think I narrowed the problem down, and it looks like it's likely NSS, and not SPDY (which I'm sure makes you happy Will).

I was going to take wireshark captures of a number of other browsers to have a comparison and baseline to try to help narrow it down. What I found was that when I tried to run Firefox 6.0, it seemed to be just as slow as Chrome (to my surprise)

I also ran a number of Chrome captures, and what I found was the largest difference seemed to come from --use-system-ssl rather than --use-spdy=off. While turning off spdy did help (slightly), the load times seemed almost unnoticable when I turned off NSS (through --use-system-ssl). I believe that this would also explain why firefox had the same problem because I believe NSS is the default SSL in firefox, as NSS is a mozilla project as well, is it not?

Perhaps this should be opened as a bug in NSS as well?
chrome logs normal.zip
5.6 MB Download

Comment 34 by jrsta...@gmail.com, Sep 16 2011

Here are the captures with various parameters (marked) set
chrome logs with parameters.zip
5.6 MB Download

Comment 35 by jrsta...@gmail.com, Sep 16 2011

and for comparison, here are Firefox, safari, and Internet Explorer. Firefox seemed to run as slowly as Chrome with no parameters, and safari and IE seemed to run as fast as chrome with NSS disabled.
other browers logs.zip
9.9 MB Download

Comment 36 by jrsta...@gmail.com, Sep 16 2011

For comparison, here are Firefox (which ran as slow as chrome no params) and safari and Internet explorer (which seemed to run as fast as chrome with nss disabled)
other browers logs.zip
9.9 MB Download

Comment 37 by jrsta...@gmail.com, Sep 16 2011

For comparison, here is Firefox (which ran as slow as chrome no params)
firefox log.zip
2.3 MB Download

Comment 38 by jrsta...@gmail.com, Sep 16 2011

As well as IE and Safari, which seem to run as fast as chrome with NSS disabled.

Sorry for splitting this into 5 comments, but the 10 MB cap doesn't work well for wireshark captures or logs
safari and internet explorer logs.zip
7.1 MB Download

Comment 39 by jrsta...@gmail.com, Sep 16 2011

Will, I'm assuming here (perhaps incorrectly) Chrome's NSS is Firefox's NSS, can you confirm? If so I can report the issue at mozilla and cross reference.

Comment 40 by willchan@chromium.org, Sep 16 2011

Cc: -mbelshe@chromium.org rsleevi@chromium.org willchan@chromium.org
Labels: -Internals-Network Internals-Network-SSL
Owner: wtc@chromium.org
Thanks Jamie! This is really useful information. wtc is the resident NSS expert. Chromium's version of NSS is the same as Firefox's, mod a few patches, so yes, it's probably an upstream issue. I'll let wtc speak to that though.

Comment 41 by rsleevi@chromium.org, Sep 16 2011

Not to add fuel to the fire, but Jamie, could you also try a capture with Opera?

Chrome + Firefox use the same underlying SSL/TLS stack (NSS - again, modulo some patches), and adopt many similar networking strategies with regards to connections, aggressively pushing the TCP window sizes, etc.

IE + Safari use the same underlying SSL/TLS stack (CryptoAPI), and tend to use a much more conservative networking approach.

Opera has their own SSL/TLS stack, and the team over there are very aggressive about strict standards compliance, so it would offer another meaningful data point to understand what is the outlier here - Is NSS behaving badly, or is CryptoAPI being prioritized?

Comment 42 by Deleted ...@, Sep 16 2011

I don't have much time to dedicate to this, but my own experiences with very slow google services in chrome led me here. I tried running chrome with SPDY disabled as per http://www.jamiestarke.com/2011/09/02/are-googles-services-running-slow-in-chrome-try-disabling-spdy/, and it seems to have fixed the problem.

andrew

Comment 43 by jrsta...@gmail.com, Sep 16 2011

@rsleevi: Sure thing, here is the wireshark log for Opera (couldn't find a more specific log), following the same procedure (gmail.com, docs.google.com, plus.google.com). It seems to run in similar time to Safari and IE.

So would this point more to NSS behaving badly, as opera's is there own implementation as well, and wouldn't get prioritized?
opera.pcap
5.6 MB Download

Comment 44 Deleted

Comment 45 by kieran.s...@gmail.com, Dec 5 2011

I too cannot add much to this, as I don't have the time. However, I get very strange behavior uploading from Chrome to AArnet Cloudstor (based on the Filesender project  -- see http://www.assembla.com/wiki/show/file_sender) via HTML5.

I have had major issues getting upstream transfers working at a decent speed beyond 4.5Mbps upload on a Win7 machine, where I got 60-80Mbps from the same machine live-booting into linux and doing the transfers from there with FF4.0. Weird. Anyway, I was put onto Jamie's issue, and applied the switches --use-spdy=off --use-system-ssl as per http://www.jamiestarke.com/2011/09/02/are-googles-services-running-slow-in-chrome-try-disabling-spdy/

Now I get 60MBps! I am also within a University network, and I can't help but wonder if it is something to do with a similar system to what Jamie encountered.

Sorry, but just thought I'd mention -- the same issue, but *not* Gmail server related. I have some wireshark grabs of the issue but they are massive, so can't post them. However, I saw "bursts" of packet transfers with a delay equivalent to the round trip time from my workstation to the server (about 23ms). Where I'd get 23ms worth of tranfers, then a pause for 23ms, then transfer for 23ms, then pause for 23ms.. etc. I'm not sure if this is the behavior that has become evident from Jamie's logs.

Comment 46 by Deleted ...@, Dec 18 2011

I am also experiencing this problem.  I had thought it was my flakey DSL connection but now that I've upgraded to a 10+ Mbps connection it became quite severe.  Turning off SPDY as suggested has solved all my problems and gmail, etc. are all quite fast now.

I wonder if this all has something to do with the networking technology used by certain ISPs or routes on the internet.

Comment 47 by jrsta...@gmail.com, Dec 18 2011

My experience would be that Yes, the network technology has a lot to do with it. The problem that I was experiencing went away immediately after my university had our traffic shaper fixed. Before that, there was a number of problems with SSL.

Comment 48 by willchan@chromium.org, Dec 19 2011

@46 (alex): Does the problem happen with Firefox too? Jamie noted that the problem seemed correlated with NSS.

I'm quite curious what could be the cause of this difference of traffic shaping for SSL implementations. Perhaps it's based on something like TLS extensions or ciphersuites or other kind of stuff that may differ slightly across implementations? I am speaking out of my element here. But as we move to more and more websites being served over SSL, it's in our interest to figure out what is going on in these different networks.

Comment 49 by Deleted ...@, Dec 19 2011

It doesn't happen in Firefox or Safari for some reason.  I just upgraded the firmware on my local DLink RV082 router and things seem to have remarkably improved.  I'll keep an eye on this today but it looks like Chrome has become usable again.

Also, just to be correct, turning off SPDY didn't really help completely.  That was a red-herring for me.  It helped a bit but I still had problems with certain services when I was logged into my gmail account.  When I wasn't logged in, things worked fine.  As such, certain services (e.g. plusone.google.com or accounts.google.com) would hang.

It really has something to do with the network hardware between you and wherever you are trying to go.  My firmware was really out of date.  I might even consider getting a different firewall for my network considering the age of the hardware.

Comment 50 by Deleted ...@, Dec 19 2011

...and if it is really SSL/network hardware related, I could see that being related to why I have more problems when I'm logged into my gmail account as those services would use SSL.  As I understand it, SPDY also encrypts data as part of the protocol and that may have similar issues.

Comment 51 by jrsta...@gmail.com, Dec 19 2011

I happened to do testing on a number of conditions when I was having issues with Chrome, and the speed up by disabling SPDY was only marginal, the speed up by going to the system SSL was significant, and the speed up of doing both was only marginally better than system ssl. 

Yes, SPDY does use an encrypted channel, and as far as I understand it uses a single persistent connection for all services at google that use SPDY (correct me if I'm wrong Will), so what I experienced was gmail, docs and google plus all behaving consistently bad. If you experience something different, it could be a potentially different bug. A trace might help, but might not. What do you think Will? There is a step by step guide linked from my article above on how to perform one.

Comment 52 by dharmady...@gmail.com, May 15 2012

I also often encounter this problem.

For me, I can't access any of Google service, not only Gmail. I can't even open the google.com front page. I don't have this problem if using other web browser, only when using Chrome.

Attached is my Wireshark capture. Please let me know if you need other thing.

Thanks.
chrome problem.pcap
329 KB Download

Comment 53 by agl@chromium.org, May 15 2012

dharmady.lim: thank you for the pcap.

I'm afraid that your network connection appears to be broken and I can't see any evidence that Chrome is doing anything wrong.

Something in the network is spoofing the TCP SYN+ACK from Google and rewriting the MSS option to 1218 bytes. This isn't unknown for devices that are tunneling traffic and need to add a layer of wrapping around packets.

However, the device appears to be broken because any packets from your computer to the outside world that actually the full 1218 bytes of payload are dropped. The connections establish correctly because the packets are small, but as soon as any actual data transfer happens a large packet is sent and the connection stalls and dies.

This may be occurring only with SPDY because SPDY is better packing the frames. However I suspect that operations like file uploads, even over regular HTTP, will encounter problems.

Comment 54 by dharmady...@gmail.com, May 16 2012

Thank you for your detailed explanation.

I'll try to troubleshoot from my side.

Comment 55 by ja...@southwell.net, Jul 4 2012

This happens to people connecting to google services from Chrome from within Belize on BTL internet connections.  BTL has aggressive traffic shaping algorithms going on and blocks services such as Skype and VPNs.  I suspect it is identifying SPDY over SSL as a vpn connection of some sort and causing it to hang or killing it prematurely.

Comment 56 by rtenneti@chromium.org, Aug 3 2012

Cc: wtc@chromium.org eroman@chromium.org rtenneti@chromium.org rch@chromium.org
 Issue 139132  has been merged into this issue.

Comment 57 by Deleted ...@, Aug 24 2012

I'm having the problem also - slow start and run speed of chrome for the last 10 days

Comment 58 by hi...@jayasundara.org, Aug 30 2012

When I tried the workaround in following site; which speeded up the gmail.

http://www.jamiestarke.com/2011/09/02/are-googles-services-running-slow-in-chrome-try-disabling-spdy/

Comment 59 by Deleted ...@, Aug 31 2012

My gmail is very slow in Chrome (have tested in other browsers and they're fine), I'm using Windows 7. It's mainly when loading emails so when you've opened a mail it takes forever before you can click reply or file it.

Comment 60 by Deleted ...@, Aug 31 2012

Why is it that all of a sudden it seems that Google+ doesn't want to run on Google Chrome?  For example, notifications, updates, and just about every feature hangs or loads forever, but never actually loads.  I have included a screenshot of the issue.
googleissuesscreenshot.docx
260 KB Download

Comment 61 by jrsta...@gmail.com, Aug 31 2012

What version of Chrome are you running?  Who is your Internet provider?

Comment 62 by bugdroid1@chromium.org, Sep 11 2012

Project Member
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=155889

------------------------------------------------------------------------
r155889 | wtc@chromium.org | 2012-09-11T00:59:41.509434Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/base/nss_memio.c?r1=155889&r2=155888&pathrev=155889
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/socket/ssl_client_socket_nss.cc?r1=155889&r2=155888&pathrev=155889
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/base/nss_memio.h?r1=155889&r2=155888&pathrev=155889
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/socket/ssl_server_socket_nss.cc?r1=155889&r2=155888&pathrev=155889

Increase the sizes of the circular buffers used by SSLClientSocketNSS
and SSLServerSocketNSS.

Larger buffers result in fewer Read() and Write() calls, improving
performance.

R=rsleevi@chromium.org,agl@chromium.org
BUG= 69813 
TEST=none

Review URL: https://chromiumcodereview.appspot.com/10919167
------------------------------------------------------------------------

Comment 63 by nathans...@gmail.com, Sep 24 2012

My wife and I are still experiencing ~10 second request latency when accessing google services like google.com and gmail on both Canary and version 21.0.1180.89. I hope this gets resolved!

Comment 64 by willchan@chromium.org, Sep 24 2012

Labels: Restrict-AddIssueComment-Commit
I'm disabling comments here, since this thread is no longer useful. Please file a new bug report by following the instructions on https://sites.google.com/a/chromium.org/dev/for-testers/providing-network-details. If you email me directly (willchan at chromium.org) with the bug id, I'm happy to triage the bug and do a first pass if the appropriate information is included.

Note that wtc's changelist did not make it into Chrome 21. Therefore, if that fixed the problem, then you won't get a fix until a later release of Chrome. That said, ~10s request latency sounds very high, and I suspect wtc's change won't fix that.

Comment 65 by wtc@chromium.org, Sep 25 2012

Labels: Mstone-23
Status: Fixed
Summary: Very slow connection to gmail and slow uploads over SSL
I also suggest we open new bugs. The problem described in this bug changed
at least once. My checkin in comment 62 (r155889) most likely fixed (or at
least reduced) the problem that jrstarke and kieran.short reported. But I
am not sure if that problem is the same as the very slow connection to
gmail problem that rvargas originally reported.

Also note that on Windows, --use-system-ssl also turns off SPDY, so if that
command-line flag helped, the underlying problem may have been SPDY flow
control if the server was a Google server.

Comment 66 by bugdroid1@chromium.org, Mar 10 2013

Project Member
Labels: -Area-Internals -Internals-Network-SSL -Mstone-23 Cr-Internals-Network-SSL Cr-Internals M-23

Comment 67 by bugdroid1@chromium.org, Mar 14 2013

Project Member
Labels: -Restrict-AddIssueComment-Commit Restrict-AddIssueComment-EditIssue

Sign in to add a comment