I can repro this on canary.  It looks like we keep on retrying the request for "https://www.123-reg.co.uk/" (With a single URLRequest), and keep on getting ABANDONED / ERR_SPDY_SERVER_REFUSED_STREAM.  We keep on doing this every 300 milliseconds without end.

So looks like an HTTP2 issue of some sort.

Confirmed in Version 62.0.3174.0 (Official Build) canary (64-bit)

Tested the issue on windows 7 , ubuntu 14.04 and Mac OS 10.12.5 using chrome M62 #62.0.3174.0 and issue is reproduced.

This is a regression issue broken in M62.

Using the per-revision bisect providing the bisect results,
Good build: 62.0.3165.0 (Revision:488876).
Bad build: 62.0.3167.0 (Revision:489499).

You are probably looking for a change made after 489129 (known good), but no later than 489130 (first known bad).

CHANGELOG URL:

The script might not always return single CL as suspect as some perf builds might get missing due to failure.

  https://chromium.googlesource.com/chromium/src/+log/269d83f247e1e2ec199ee4c4787e2eed80d1417b..48c4ddfd57b22f94ceaa778c0456d9e328f278d0

From the CL above, assigning the issue to the concern owner 

@Bence Béky- Could you please check whether this is caused with respect to your change, if not please help us in assigning it to the right owner.

Review URL : https://chromium.googlesource.com/chromium/src/+/48c4ddfd57b22f94ceaa778c0456d9e328f278d0

Note : Adding RB-Beta for now as it is recent regression , Please feel free to edit/remove this.

Thanks!

Friendly ping to get an update on this.

This is a server bug.  Things are working as intended on Chrome's side, and I'm closing the bug accordingly.  I was able to reproduce locally, and verify that regression happened at my CL blamed in comment #4.  Then I compiled tip of the tree and experimented:

If Chrome does not send a SETTINGS_MAX_HEADER_LIST_SIZE field in the initial settings frame when opening an HTTP/2 connection to www.123-reg.co.uk, or if the value of the field is 1000, 30000, 32767, or 32768 (the last being 32 * 1024), then the request succeeds.

If Chrome sends a SETTINGS_MAX_HEADER_LIST_SIZE value of 32769, 50000, 90000, or 262144 (the last being the default behavior since the CL mentioned above), then the server closes the connection after about 100 ms, which can conceivably be 1 RTT.  It does so by sending a GOAWAY frame with last_accepted_stream_id = 0 (telling Chrome that the request can be retried safely), error_code = NO_ERROR (which is not quite appropriate in this case), and no debug data (which could be included to explain the reason of closing the connection).  This is, however, unwarranted: Chrome sending a SETTINGS_MAX_HEADER_LIST_SIZE value tells the server how long response headers Chrome is willing to accept at maximum.  Larger value means Chrome is willing to accept larger responses.  It does not ask or require the server to use more memory.

Whenever I set the value of SETTINGS_MAX_HEADER_LIST_SIZE so that the request succeeds, response headers include "server: nginx".  I filed https://trac.nginx.org/nginx/ticket/1351.

As for the infinite retry loop, it is working as intended.  Chrome does it specifically to deal with graceful shutdowns.  The root of the problem is the NO_ERROR code sent by the server.  It might be nice to implement a limit on the number of retries, but that would not solve this problem.

I'm working with a Samsung engineer interested in starting working on Chrome's network stack on the infinite retry issue.

Hi (I'm the user who raised this issue on the product forums thread initially). I have raised this issue with 123-reg and directed them to this ticket for further information. 

bnc: nice analysis!

We are getting similar error reports from some of our Google Chrome users for all our HTTP/2 enabled sites.

Thanks to bnc (comment #7) I was able to track down the problem and can confirm that the new SETTINGS_MAX_HEADER_LIST_SIZE value is causing the problem.

In our case our sites are all behind a F5 BIG-IP load balancer running latest firmware (v12.1.2 Build 1.0.271 Hotfix HF1). So it is the F5 system which has some troubles with the new value.

I'll update the bug report once we got a ticket ID from F5 support.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/45a41721a89124db629b9f1bca5d432a9df2ada0

commit 45a41721a89124db629b9f1bca5d432a9df2ada0
Author: Biljith Jayan <billy.jayan@samsung.com>
Date: Wed Aug 16 18:43:14 2017

Add max retries to an HttpNetworkTransaction when an HTTP2 or QUIC related
IO Error occurs

Before this patch, HttpNetworkTransaction keeps resending the request in
an infinite loop if an HTTP2 or QUIC related IO Error occurs. We try to
add a limit to the number of retries (3 tries, including the first request)

BUG:  751642 
Bug: 
Change-Id: I3ab29dede0be0516f8706ec7e807f9febd56fed8
Reviewed-on: https://chromium-review.googlesource.com/602389
Commit-Queue: Matt Menke <mmenke@chromium.org>
Reviewed-by: Matt Menke <mmenke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#494878}
[modify] https://crrev.com/45a41721a89124db629b9f1bca5d432a9df2ada0/net/http/http_network_transaction.cc
[modify] https://crrev.com/45a41721a89124db629b9f1bca5d432a9df2ada0/net/http/http_network_transaction.h
[modify] https://crrev.com/45a41721a89124db629b9f1bca5d432a9df2ada0/net/http/http_network_transaction_unittest.cc

OK, if you are facing the problem like we did and your sites are behind a F5 load balancer you have to request the patch for

> BZID 677119
> Title: HTTP2 enforces unsupported settings: SETTINGS_MAX_HEADER_LIST_SIZE and
>        SETTINGS_HEADER_TABLE_SIZE

from your F5 support contact.

In my case we have upgraded to v12.1.2 Build 1.307.271 Engineering Hotfix HF1 and the error is gone -- latest Chrome Canary build (62.0.3192.0 at the moment) can now connect to our HTTP/2-enabled sites again.


For references: https://github.com/golang/go/issues/20979#issuecomment-320361343

 Issue 757459  has been merged into this issue.

I understand that this is caused by a server issue, but shouldn't Chrome try to fallback to HTTP1 when there's an HTTP2 connection issue? The server still supports it, and from my point of view, it's way easier to add a work around to Chrome than going out asking all the sysadmins around the world to fix their setups. 

At the end, users will blame Chrome, as the affected websites work with all other major browsers out there.

A server rejecting a stream request is a valid thing for a server to do - when they do that, it's not at all clear that it's because they don't fully support HTTP/2.  So we'd be ignoring the rejection message from the server to put more load on the server that rejected the request (Overloading may, in fact, be the reason the server rejected the stream request in the first place).

More generally, hiding bugs tends to be much worse, long term, than exposing them.  In this case, covering it up would increase server load and page load time, in a way that server operators would be unlikely to notice.  Any other HTTP/2 client that runs into this issue would then also have to introduce this behavior to work with buggy servers.  And then presto, this becomes a de facto standard that all clients need to support.

Much better to go through some short term breakage, rather than needlessly make the web platform more complicated indefinitely.

Adding to what mmenke said, even if the server emitted garbage HTTP/2 we would still not fallback to HTTP/1.1. In the same way that if a server emitted garbage HTTP/1.1 we would not fallback to HTTP/1.0 (or 0.9).

Understood. Those are very good points.

 Issue 760921  has been merged into this issue.

Hi, should this issue be fixed and/or resolved by reporters server software / load-balancer middleware (per trac.nginx ticket referenced in #c7)

I suffer from this bug currently, with latest Chrome on Linux x86_64
version: Version 62.0.3202.9 (Official Build) dev (64-bit)

Sites that do not work for me:
- https://www.123-reg.co.uk/
- https://www.czc.cz/

In network log I can see 3 tries to connect to given site.
Disabling the QUIC protocol in chrome://flags does not resolve the issue

 Issue 761632  has been merged into this issue.

The following websites are now working properly with version 63.0.3214.0:
https://www.mercadolivre.com.br/
https://www.otto.de/

The following are still buggy:
https://www.123-reg.co.uk/
https://www.hoeffner.de/
https://www.czc.cz/

https://www.goo.ne.jp/ and many *.goo.ne.jp sites cause ERR_SPDY_SERVER_REFUSED_STREAM for me.
How can I know whether or not the same problem? Thanks.

gotoken:  If they work on Chrome stable, it's almost certainly the same issue.  Short of specifically modifying chromium to send a different SETTINGS_MAX_HEADER_LIST_SIZE header, though, and seeing if that fixes/breaks things, it's difficult to be completely positive this is the issue.

Same issue here.

Windows 10 64bit
Chrome 62.0.3202.18 (Official Build) beta (64-bit)

I checked all the websites mentioned in this issue and the other three issues merged into this one.  First of all, http scheme will never result in this problem, because HTTP/2 is only supported over https.  So I tested the https counterpart of all http links.  Here are the results.

The following sites all work me with Chrome 63.0.3218.0.  If they ever were broken, they are fixed by now.  Note that three of them give ERR_CERT_COMMON_NAME_INVALID certificate error, which is a red flag, but I clicked through the interstitial for the sake of testing.

https://www.ahaonline.cz
https://www.fitweb.cz
https://www.orbion.cz
https://www.abicko.cz
https://www.auto.cz
https://www.autorevue.cz
https://www.blesk.cz
https://www.dama.cz
https://www.e15.cz
https://www.info.cz
https://www.maminka.cz
https://www.mercadolibre.com
https://www.mercadolibre.com.ar
https://www.mercadolivre.com.br
https://www.mobilmania.cz
https://www.otto.de
https://www.recepty.cz
https://www.reflex.cz
https://www.zive.cz

These other sites below, however, are broken.  I tried Chrome 63.0.3218.0, and I also compiled trunk (be29ef382f).  I also modified the SETTINGS_MAX_HEADER_LIST_SIZE value to 64 kB, and to 32 kB + 1 B, and confirmed that all these sites are still broken.  However, if I modify SETTINGS_MAX_HEADER_LIST_SIZE to 32 kB, then all of them work.  This indicates that the issue is caused by a single bug (in server or middlebox) for all of them.

https://depot.sbroker.de
https://www.123-reg.co.uk
https://www.azet.sk
https://www.czc.cz
https://www.goo.ne.jp
https://www.hoeffner.de
https://www.mojezdravi.cz
https://www.rockhard.de
https://www.sportmaster.ru
https://www.zeny.cz

Among these, there are two sites that are featured in the Alexa top 50 within their respective countries: www.azet.sk is 5th in Slovakia, goo.ne.jp is 23th in Japan.

 Issue 766683  has been merged into this issue.

This problem is now also affecting Chrome Beta, now that it's upgraded to 62.0.3202.18.

Tagging as assigned, since investigation is still in progress.

Out of the 10 broken websites mentioned in comment #29, I sent an e-mail to three, filled out a message webform (that they call "e-mail") on four, and did not find any contact info for three.

M62 is already released as Beta, this bug is now blocking Stable release.

another domain that fails with the current official beta release, was working before: https://finanzblick.de/ 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673

commit b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673
Author: Bence Béky <bnc@chromium.org>
Date: Thu Sep 21 19:22:15 2017

Do not send SETTINGS_MAX_HEADER_LIST_SIZE value.

Do not send SETTINGS_MAX_HEADER_LIST_SIZE value in initial HTTP/2
SETTINGS frame, because it is known to break some websites.

I locally verified that the following websites do not load
(ERR_SPDY_SERVER_REFUSED_STREAM) without this change but they do load
with this change:

https://depot.sbroker.de
https://www.123-reg.co.uk
https://www.azet.sk
https://www.czc.cz
https://www.goo.ne.jp
https://www.hoeffner.de
https://www.mojezdravi.cz
https://www.rockhard.de
https://www.sportmaster.ru
https://www.zeny.cz

BUG= 751642 

Change-Id: I5bc9c28b7369a5667a4d1c77833c28dd0d192701
Reviewed-on: https://chromium-review.googlesource.com/677304
Reviewed-by: Ryan Hamilton <rch@chromium.org>
Commit-Queue: Bence Béky <bnc@chromium.org>
Cr-Commit-Position: refs/heads/master@{#503530}
[modify] https://crrev.com/b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673/net/http/http_network_session.cc
[modify] https://crrev.com/b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673/net/http/http_network_session.h
[modify] https://crrev.com/b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673/net/spdy/chromium/spdy_network_transaction_unittest.cc
[modify] https://crrev.com/b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673/net/spdy/chromium/spdy_session.cc
[modify] https://crrev.com/b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673/net/spdy/chromium/spdy_session_unittest.cc

So are we going to change the HTTP2 spec for all time not to allow this?

Re comment #37: the plan is to revert and merge that into M62 for now, because there is a number of broken sites, while the advantage of sending SETTINGS_MAX_HEADER_LIST_SIZE is marginal.  However, I reached out to most site operators to understand what device is non-compliant, and I'll reach out to the device vendor for more information.  As soon as there is a confirmed way for site operators to fix their servers, I'll change Chrome back to continue sending SETTINGS_MAX_HEADER_LIST_SIZE values.  It is very important for the ecosystem to make sure servers do not incorrectly reset the stream on spec-compliant and reasonable settings.

https://crrev.com/c/677304 has been picked up by this morning's Canary 63.0.3222.0.

Requesting merge approval for https://crrev.com/c/677304 (comment #36 above) to M62 (3202).

63.0.3222.0 has been out for three days, I do not see any crash reports related to this change.

List of impacted websites is in comment #29 (plus one more, https://www.liportal.de, reported at  https://crbug.com/757459#c16 ).  These websites currently do not load at all, https://crrev.com/c/677304 fixes that.

Thank you.

This bug requires manual review: Less than 18 days to go before AppStore submit on M62
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Thanks bnc@ for the fix. Can you please confirm how safe this merge is overall? Well tested? 

As discussed over chat with bnc@, fix is safe overall. Approving merge to M62. Branch:3202

bnc:  Does the mean we're never going to ship this feature?  Or is there some path forward here?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0a609b8af206f1db614fa97877372f8e1134571c

commit 0a609b8af206f1db614fa97877372f8e1134571c
Author: Bence Béky <bnc@chromium.org>
Date: Mon Sep 25 18:08:55 2017

Do not send SETTINGS_MAX_HEADER_LIST_SIZE value.

Do not send SETTINGS_MAX_HEADER_LIST_SIZE value in initial HTTP/2
SETTINGS frame, because it is known to break some websites.

I locally verified that the following websites do not load
(ERR_SPDY_SERVER_REFUSED_STREAM) without this change but they do load
with this change:

https://depot.sbroker.de
https://www.123-reg.co.uk
https://www.azet.sk
https://www.czc.cz
https://www.goo.ne.jp
https://www.hoeffner.de
https://www.mojezdravi.cz
https://www.rockhard.de
https://www.sportmaster.ru
https://www.zeny.cz

BUG= 751642 

Change-Id: I5bc9c28b7369a5667a4d1c77833c28dd0d192701
Reviewed-on: https://chromium-review.googlesource.com/677304
Reviewed-by: Ryan Hamilton <rch@chromium.org>
Commit-Queue: Bence Béky <bnc@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#503530}(cherry picked from commit b8be85f12b3d2fdb6ff3b68ac85d6ae726f70673)
Reviewed-on: https://chromium-review.googlesource.com/682374
Reviewed-by: Bence Béky <bnc@chromium.org>
Cr-Commit-Position: refs/branch-heads/3202@{#431}
Cr-Branched-From: fa6a5d87adff761bc16afc5498c3f5944c1daa68-refs/heads/master@{#499098}
[modify] https://crrev.com/0a609b8af206f1db614fa97877372f8e1134571c/net/http/http_network_session.cc
[modify] https://crrev.com/0a609b8af206f1db614fa97877372f8e1134571c/net/http/http_network_session.h
[modify] https://crrev.com/0a609b8af206f1db614fa97877372f8e1134571c/net/spdy/chromium/spdy_network_transaction_unittest.cc
[modify] https://crrev.com/0a609b8af206f1db614fa97877372f8e1134571c/net/spdy/chromium/spdy_session.cc
[modify] https://crrev.com/0a609b8af206f1db614fa97877372f8e1134571c/net/spdy/chromium/spdy_session_unittest.cc

Re #44: Good question.  Yes, there is a path forward.  Most importantly, there is already field trial support for making Chrome send this SETTING value in an experiment group, so that the impact of broken servers can be monitored.  Also, I reached out to most server operators informing them about the issue and asking for information about whether they use the load balancer that is suspected in comment #11.  After sufficient confirmation I can reach out to the vendor to inquire about their firmware upgrade policy and to coordinate the re-launch of this SETTING with them.

In the most fortunate situation, if it turns out that all affected sites use the same product and thus suffer from the same bug, and if the vendor provides automatic upgrades or at least encourages clients to upgrade and provides an easy way to do so, then I anticipate that this SETTING can be re-launched in about one quarter from now.

Android: On latest M62 the pages loads properly without any error. Thanks 

F5 developers confirmed what was posted above in comment #13: their customers can request a patch for bug number 677119, and this fix will be included in future releases.

For the time being, I am closing this ReleaseBlock bug as Fixed, since Chrome does not send this setting value any more neither on tip of tree nor in M62 (and never did on M61), therefore the imminent issue is mitigated.

This problem seems to be reproduced for me with 65.0.3294.5 (Official Build) dev (64-bit)
on https://www.123-reg.co.uk and some sites including https://www.goo.ne.jp

It looks like this may now be fixed as it was broken on version 65.0.3306 but the browser has just updated to 65.0.3307 (latest dev version) and all the sites listed now work.

Out of the sites that were known to be broken at one point, the following are now fixed (tested with SETTINGS_MAX_HEADER_LIST_SIZE value of 256k):
https://www.123-reg.co.uk
https://www.abicko.cz
https://www.ahaonline.cz
https://www.auto.cz
https://www.autorevue.cz
https://www.blesk.cz
https://www.czc.cz
https://www.dama.cz
https://www.e15.cz
https://www.fitweb.cz
https://www.hoeffner.de
https://www.info.cz
https://www.liportal.de
https://www.maminka.cz
https://www.mercadolibre.com
https://www.mercadolibre.com.ar
https://www.mercadolivre.com.br
https://www.mobilmania.cz
https://www.orbion.cz
https://www.otto.de
https://www.recepty.cz
https://www.reflex.cz
https://www.rockhard.de
https://www.zive.cz

However, the following are still broken (tested with 256k, 32k + 1, and they all work with 32k, so it's the exact same issue as before):
https://depot.sbroker.de
https://www.azet.sk
https://www.goo.ne.jp
https://www.mojezdravi.cz
https://www.zeny.cz
https://www.sportmaster.ru

It is very promising to see that many site operators have upgraded their buggy load balancer firmware (or moved to a different solution).  However, unfortunately goo.ne.jp is still in top 50 for Japan (31th according to https://www.alexa.com/topsites/countries/JP), and it auto-redirects to https so it probably negotiates HTTP/2 for all Chrome users, so the workaround cannot be reverted in Chrome quite yet.

Looks like it is broken again.

Re comment #52: I do not get any protocol errors on the domains above.  Can you please describe exactly which website is broken, what the error message is, and what Chrome version are you using?  Thank you.

I'm on dev 67.0.3396.10 and disable QUIC protocol in chrome://flags/ solve this problem

Re comment #54: I'm on 67.0.3396.18 and do not see protocol errors on any of the sites listed in comment #51.  What website fails to load for you and what is the exact error message you are getting?

Over millions of samples in an ongoing experiment, ERR_SPDY_SERVER_REFUSED_STREAM is recorded for Net.ErrorCodesForMainFrame3 with a frequency of 0.01% as opposed to 0.00% for the control group (rounded to this many digits), reported to be a more than 200-fold increase.  In other words, this bug is still measurable in the wild, but the impact is very small.

Looking at ERR_SPDY_SERVER_REFUSED_STREAM, in both Net.ErrorCodesForImages2 and in Net.ErrorCodesForSubresources3, the error rate is less than 2e-5 for the experiment group, and there is no statistically significant difference from the control group in either case.

In Net.ErrorCodesForMainFrame4, error rate is still 1e-4, about 140 times higher than in the control group.