New issue
Advanced search Search tips

Issue 681477 link

Starred by 13 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 2
Type: Bug



Sign in to add a comment

HTTP/2 streams aborted by GOAWAY sent be server are not retried

Reported by antti.ri...@gmail.com, Jan 16 2017

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.98 Safari/537.36 Vivaldi/1.6.689.40

Example URL:
https://d889ne7b3rd4a.cloudfront.net/cloudfront-http2-test4/test2.html

Steps to reproduce the problem:
1. Visit the page
2. See that only 1000 of the 1100 images load

What is the expected behavior?
All the images load properly.

What went wrong?
Quick analysis suggests that the Cloudfront CDN sends a GOAWAY to HTTP/2 connection after the client has requested 1000 resources over it. Chrome not retrying those resources on another connection results in failed GETs.

Did this work before? N/A 

Chrome version:  55.0.2883.95 (64-bit)  Channel: stable
OS Version: OS X 10.11.6
Flash Version: 

What's the actual expected HTTP/2 behaviour in this case? Safari seems to fail sometimes, Firefox never (Wireshark capture for Firefox suggests that it opens a second HTTP/2 connection by chance without receiving a GOAWAY on the first connection). If the browsers function correctly, I'll take up this issue with Cloudfront.
 
net-internals-log.json.zip
560 KB Download
Labels: Needs-Triage-M55

Comment 2 by rch@chromium.org, Jan 17 2017

Owner: b...@chromium.org

Comment 3 by b...@chromium.org, Jan 17 2017

Status: Assigned (was: Unconfirmed)
RFC 7540 Section 6.8 says "The receiver of the GOAWAY frame can treat the streams as though they had never been created at all, thereby allowing those streams to be retried later on a new connection.".  This does not require the client to retry the requests, but I agree that it would be nice.

I see streams up to ID 2125 being opened up by Chromium before it receives the GOAWAY with error code = 0 (NO_ERROR) and last stream ID = 1999.  This signals that the server will not process streams above this ID, so they can safely be retried.  I image having to create a new net error code for this, failing requests corresponding to those streams with that particular error code, and adding that code to the already existing retry logic.

Comment 4 by mmenke@chromium.org, Jan 17 2017

Components: -Internals>Network Internals>Network>HTTP2
I assume we may also get and process these errors before the old session has been closed...And since we don't support multiple sessions to the same server, this may just be a WontFix.

Comment 5 by b...@chromium.org, Jan 17 2017

Yes, after GOAWAY is received, outstanding streams with ID not exceeding the GOAWAY frame's last-stream-id value are still processed.  During this period the stream is unavailable, SpdySessionPool::FindAvailableSession() does not find it.  But it is still active, and is only closed later, normally after all those streams are closed.  So yes, if we decided to fix this, we would not to support multiple sessions to the same server, but not multiple active sessions.
Mainline Nginx has http2_max_requests default at 1000. Making 1000+ requests is not an unreasonable scenario at this day and age. 

Having then some requests fail by choice (as hinted by WontFix comment above) feels to me like undermining the HTTP/2 infrastructure.

To cross-link issues and add a little bit of background: here is the original ticket that lead to the introduction of `http2_max_requests`: https://trac.nginx.org/nginx/ticket/1102
Also note that nginx will send GOAWAY not only after `http2_max_requests`, but also during the reload/graceful shutdown process.

The original problem was that clients could reuse h2 connection indefinitely, which was impossible in h1 where server could reply with `Connection: close` at any time without any problems(except for "pipelined" case, but that is another story). With multiple requests inflight setting `Connection: close` is not deterministic anymore(nor supported by h2), therefore sending `GOAWAY`(which has `last_accepted_stream_id` information built in) seems like a logical and fully spec-compliant thing.
hey all - fyi this was also reported to the firefox bug tracker and you can watch that at https://bugzilla.mozilla.org/show_bug.cgi?id=1050329 starting at comment 2 (an open bug was hijacked, so the first couple entries are not relevant).

I've identified a problem with the test case (the server is generating TCP RSTs, making the GOAWAY in the stream impossible to read given race conditions) - but I did want to add my support for how he describes the mechanism ought to work.

This initially went into 7540 as a google design, graceful transition between nodes was a very important property to roberto and it would be great if this could work across the web.. and while I think a max-requests parameter on the server is rather counter productive, you could certainly see this coming into play based on timeouts, or just load rebalancing.

Comment 9 by b...@chromium.org, Apr 13 2017

Cc: yhirano@chromium.org
 Issue 677828  has been merged into this issue.

Comment 10 by b...@chromium.org, Apr 17 2017

Cc: b...@chromium.org
 Issue 696808  has been merged into this issue.
Project Member

Comment 11 by bugdroid1@chromium.org, Apr 19 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4769d412d1b89724db8eb42577cd975fd85f559

commit c4769d412d1b89724db8eb42577cd975fd85f559
Author: bnc <bnc@chromium.org>
Date: Wed Apr 19 19:57:54 2017

Retry request upon GOAWAY frame with NO_ERROR.

Retry request upon receiving a GOAWAY frame with error code NO_ERROR
and with Last-Stream-ID lower than stream id corresponding to request.

BUG= 681477 

Review-Url: https://codereview.chromium.org/2820163004
Cr-Commit-Position: refs/heads/master@{#465717}

[modify] https://crrev.com/c4769d412d1b89724db8eb42577cd975fd85f559/net/base/net_error_list.h
[modify] https://crrev.com/c4769d412d1b89724db8eb42577cd975fd85f559/net/spdy/spdy_network_transaction_unittest.cc
[modify] https://crrev.com/c4769d412d1b89724db8eb42577cd975fd85f559/net/spdy/spdy_session.cc

Comment 12 by b...@chromium.org, Apr 19 2017

Status: Fixed (was: Assigned)
Is this bug fixed in Version 60.0.3112.90?

Comment 14 by b...@chromium.org, Aug 11 2017

Yes, it is fixed in 60.0.3112.90.
In which version was it fixed ?

Comment 16 by b...@chromium.org, Aug 15 2017

The fix was first introduced in 60.0.3076.0.

Sign in to add a comment