Could you attach a net-internals log per these instructions? Thanks!

https://dev.chromium.org/for-testers/providing-network-details

net-internals log attached. 

Deleted: net-internals-log.json 646 KB

net-internals-log.json 646 KB View Download

"SPDY session closed because of stream with status: 13", which is RST_STREAM_HTTP_1_1_REQUIRED (Which seems a pretty clear indication of what the server really wants).

Removing Auth since the subsequent retry over HTTP/1.1 succeeds as expected.

We should address the issue of ERR_UNEXPECTED flashing in-between.

bnc: This seems to be the same as  issue #713851 . Did you end up making progress on that one?

 Issue 756828  has been merged into this issue.

 Issue 713851  has been merged into this issue.

Copying part of https://crbug.com/717329#c21 here because that issue is private:

... First request gets a 401 response with www-authenticate: NTLM.  Request is retried on same connection with correct authorization header.  Not surprisingly, it gets a HTTP_1_1_REQUIRED error, because HTTP/2 does not handle client certificates.  (It was in fact Microsoft who requested this error code to be included in the
HTTP/2 specification for exactly this use case.)  Then HTTP_STREAM_JOB tries again, with an SSLConfig that has no "h2" element in its |alpn_protos| member.  This would normally open a new connection on which http/1.1 would be negotiated with the server in the TLS handshake.  However, it instead binds to an existing idle SOCKET, which is HTTP/2, so a new HTTP2_SESSION is created on it, which obviously gets another HTTP_1_1_REQUIRED response.  This socket is then closed, so the next retry has to open a new one, which correctly observes the HTTP_1_1_REQUIRED restriction (using HttpServerProperties::MaybeForceHTTP11).

In other words, the bug in Chrome is that receiving an HTTP_1_1_REQUIRED only
forces http/1.1 for new connections, but does not prevent the retry job from
binding to an existing HTTP/2 connection.

The retry logic is HttpNetworkTransaction::HandleHttp11Required().  That sets
|server_ssl_config_|.  This is passed through but not observed in case the
request pools to an existing idle socket, which is done through the following code
path:

HttpStreamFactoryImpl::Job::DoInitConnectionImpl()
InitSocketHandleForHttpRequest()
InitSocketPoolHelper()
ClientSocketHandle::Init()
ClientSocketPoolBase::RequestSocket()
SSLClientSocketPool::RequestSocket()
ClientSocketPoolBaseHelper::RequestSocket()
ClientSocketPoolBaseHelper::RequestSocketInternal()
ClientSocketPoolBaseHelper::AssignIdleSocketToRequest()

One messy solution would be to upcast |request| from
ClientSocketPoolBaseHelper::Request to
ClientSocketPoolBase<SSLSocketParams>::Request, and make sure to only pool to an
SSL socket if that has an ALPN protocol that matches
request.params_.ssl_config_.

Or one could plumb another bool member from HttpNetworkTransaction to
HttpStreamFactoryImpl::JobController to HttpStreamFactoryImpl::Job to disable
pooling to an existing connection.  (This would be similar to the existing
enable_ip_based_pooling_ flag which is plumbed through the same path.)  That
might need to be plumbed further down to the socket pools.

I'm wondering why is there an idle socket along with the active one to the same
server, which is the root of the problem in the first place.  Maybe the socket
pool should close down all sockets but one if that one is HTTP/2?  That would
also solve the whole problem.

Oh, yeah you cannot just randomly mutate the SSLConfig at HttpNetworkTransaction. That breaks pooling assumptions across the stack. :-(

You could bifurcate the socket pools (see InitSocketPoolHelper), but munging the ALPN list into the key would be rather hairy. Probably the ALPN list shouldn't be assembled by HttpNetworkTransaction at all and rather by  the HttpStreamFactoryImpl somewhere, since it's the code that knows what ALPN tokens are accepted. It could fill in that list based on the boolean member you describe and I guess also arrange for the pools to be bifurcated accordingly.

+mmenke if he has thoughts.

Do murderous thoughts count?  (Not about anyone in particular, just in general...Though that may be worse, actually).

I'd rather not bifurcate the pools for this.  In particular, if all requests to a server require NTLM, we're gong to be repeatedly sending H2 requests to the server, getting the notification, and then switching pools.  Which seems like something we don't want, if we want to make both H2 and NTLM first class citizens.  My own preference would be to just flush any sockets/H2 sessions for the server, and to remember never to use H2 with it.  Which also isn't great, of course, but seems better than the alternative to me.

Also, both having an H2 session and HTTP/1.x sockets to a server at the same time just seems scary.

I prefer the approach laid out in comment #10, and I prototyped it at https://crrev.com/c/643150.

I understand both the arguments for per-origin and per-request interpretation of HTTP_1_1_REQUIRED.  Unfortunately the HTTP/2 specification does not provide any guidance for this.  I am aware of at least one external person complaining about the current, per-origin implementation, see  issue 516237 .

 Issue 709827  has been merged into this issue.

Issue 717329 has been merged into this issue.

This issue is the same as  issue 709827 ,  issue 713851 , issue 717329, and  issue 756828 , and only happens on Windows.  The flow for all of these cases is the same: request on HTTP/2 receives a 401 response with www-authenticate: NTLM.  Retry with authorization header on HTTP/2 receives a RST_STREAM with HTTP_1_1_REQUIRED.  (In  issue 713851 , the retry binds to a preconnected idle socket with HTTP/2 negotiated, creates a new HTTP2_SESSION, and gets another HTTP_1_1_REQUIRED response.)  Retry then happens on a new socket, over HTTP/1.1, but a second call to HttpAuthController::MaybeGenerateAuthToken() on Windows cannot generate a new auth token, but instead returns ERR_UNEXPECTED, see https://crbug.com/717329#c18.  This is displayed on the screen for a short amount of time, until the higher level retry mechanism kicks in a successfully loads the request.

Of course, binding to an idle HTTP/2 socket could happen on any platform, but outside Windows that does not result in a user-facing error because of the retries within HttpNetworkTransaction.

There are multiple options to improve the situation:

* Preemptively send the very first retry, the one with the authorization header, over HTTP/1.1 instead of HTTP/2, see https://crbug.com/717329#c16.
* Improve HttpAuthController to deal with having to provide auth token twice in a row, see https://crbug.com/717329#c20.
* Prevent retries from binding to an idle HTTP/2 socket, either by bifurcating the socket pool, see comment #10 above, or flushing all HTTP/2 idle sockets to the origin, see comment #11 above.  Note that there is an initiative to cut down on HTTP/2 sockets, both by only preconnecting one if the server is known to support HTTP/2, or more aggressively shutting down multiple HTTP/2 sockets.  This work could mitigate the binding issue.

Re comment #11:

I agree that both having an H2 session and HTTP/1.x sockets to a server at the same time is scary.

I thought every request first goes out without authentication headers, and such headers only get appended for the retry upon a 401 response.  So using HTTP/2 for every request by default and retrying on the scary concurrent HTTP/1.1 socket should not incur a latency penalty.

NTLM authentication is bound to the socket...Which means what?  Suppose we have a socket that has already gone through NTLM negotiation.  So if we reused the socket for a request, it already has been authenticated, and re-negotiation may not be necessary.  If we send the request over H2, however, that connection has not been authneticated, so we get the error.  Do we see an NTLM challenge with the HTTP_1_1 required message or not?  If not, we still need to see the challenge from the re-issued request before we can start NTLM negotiation.

Either way, we retry on the idle socket previously used for NTLM negotiation.  We either try to send the first round of NTLM authentication headers on a socket that may already be considered successfully authenticated by the server, which would result in extra round trips.  Or we send the request without the data for first round negotiation, in which case we add a round trip in all cases.

> NTLM authentication is bound to the socket...Which means what?  Suppose we
> have a socket that has already gone through NTLM negotiation.  So if we reused
> the socket for a request, it already has been authenticated, and
> re-negotiation may not be necessary.  If we send the request over H2, however,
> that connection has not been authneticated, so we get the error.  Do we see an
> NTLM challenge with the HTTP_1_1 required message or not?  If not, we still
> need to see the challenge from the re-issued request before we can start NTLM
> negotiation.

There is no NTLM challange with the HTTP_1_1_REQUIRED.  It is not a response, it is a stream error.

> Either way, we retry on the idle socket previously used for NTLM negotiation.
> We either try to send the first round of NTLM authentication headers on a
> socket that may already be considered successfully authenticated by the
> server, which would result in extra round trips.  Or we send the request
> without the data for first round negotiation, in which case we add a round
> trip in all cases.

What do we currently do for a site that does not use HTTP/2?

Ok, so if there's no NTLM challenge, then we do indeed always need an extra request (If we need to negotiate NTLM, then we need to see the challenge before we start negotiating.  If we don't, then we'd only need one request if we just used HTTP/1.x directly).

For a site that doesn't use HTTP/2?  We send a request, and if we get an NTLM challenge, we start negotiating NTLM authentication on that socket.  The problem is the HTTP_1_1_REQUIRED isn't a challenge, so we'll have to do the exact same things on the HTTP/1 socket as we'd do without the HTTP_1_1_REQUIRED response.

And of course, even if the HTTP_1_1_REQUIRED could send a challenge, it's possible we already have an HTTP/1 socket with NTLM negotiated.  So if a page requires an NTLM-authenticated socket for every resource, this could effectively double the number of requests over what an HTTP/1.x socket uses.  Or at least that's my understanding.  I'm not the NTLM expert.

Note that since issue 690918 is fixed, there should not be any idle HTTP/2 sockets when HTTP_1_1_REQUIRED arrives on an HTTP/2 connection: creating a connection over a socket closes all idle sockets in the same group.

How does that help?  If another request comes in, we'll then wait to establish another H/2 connection, get another HTTP_1_1_REQUIRED message, and then switch to an existing idle socket...right?  Or am I missing something?

Well, it does not help in case we change the logic to dispatch new connections on HTTP/2.

However, if we stick with the current situation of blocking HTTP/2 for the origin, then it helps the first retry, because it guarantees that the retry will not bind to an idle HTTP/2 socket (since there will not be one).  Binding to an idle HTTP/2 socket and requesting the resource another two times was observed in  issue 713851  and issue 717329.

Right.  I never said it wouldn't work around the bug, I just said it would potentially result in lot of extra requests and additional latency, relative to what would happen if we just decided to always use HTTP/1.x for the host.

(Oops, apparently I wasn't CC'd.)

I would expect non-NTLM resources are faster because they still can use HTTP/2, while the NTLM resources are slower because we'll ask for an HTTP_1_1_REQUIRED every time we load it. We could mitigate the latter with a (bounded-size!) cache of URLs rather than hosts (so even more complexity...), though it'd only help on repeat visits to a given resource.

It's probably reasonable to assume the site has a bunch of HTTP/2-capable resources because if it wanted NTLM on everything, it wouldn't have enabled HTTP/2. Though I could maybe imagine if it being based on anonymous vs. logged in??

This mess only exists because Microsoft has a bunch of legacy socket-based auth use cases (NTLM, client certs, etc.) and they wanted an escape hatch for HTTP/2. Perhaps look at what Edge or IE does? I expect they'd be tuned to the use cases Microsoft intended here. (Though since it's an edge case, if the Edge behavior is too complex, we may wish to pick something simpler regardless.)

Or perhaps the maintainers enabled HTTP/2 and NTLM because they weren't aware of the compatibility issues.

Fair enough. I suppose the intention might be less "Why not both?" and more "try to salvage things if the maintainer does something stupid". :-)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/83eb351c8d1b6e605d35e3e7123b38e7784c5fe3

commit 83eb351c8d1b6e605d35e3e7123b38e7784c5fe3
Author: Bence Béky <bnc@chromium.org>
Date: Tue Sep 05 12:56:09 2017

Document existing HTTP_1_1_REQUEST retry behavior with a unittest.

BUG= 685741 

Change-Id: Ibade6f001a669392c9cf50137e17c97331f49cad
Reviewed-on: https://chromium-review.googlesource.com/646658
Commit-Queue: Bence Béky <bnc@chromium.org>
Reviewed-by: Asanka Herath <asanka@chromium.org>
Cr-Commit-Position: refs/heads/master@{#499613}
[modify] https://crrev.com/83eb351c8d1b6e605d35e3e7123b38e7784c5fe3/net/http/http_network_transaction_unittest.cc

Any chance for an ETA on this? We're having major problems due to this bug. Thanks, Ron

> We're having major problems due to this bug

Can you please elaborate on those major problems? As it stands now, this bug is of low priority as it seems to be of minor impact.

For our applications we are using NTLM authentication for Single Sign-On (in a secure environment I must add). Due to this bug the NTLM authentication is aborted by the client and as a result the SSO doesnt work anymore.

We're using IIS 8.5 as the front-end webserver on Windows 2016 and right now we're looking into disabling HTTP2 alltogether but we can't seem to find out how to do this.

Additionally, we were experiencing the same issue with Firefox but they have fixed this problem in their latest version so we;re out of the woods for Firefox as well as IE/Edge users. 

I disabled HTTP2 on Windows 2016 using this script: https://gist.github.com/nicksterx/8cabfd5c696bd23f8ab4f11ca112cb26

It fixed the issue for me server side. 

I have tried setting the registry settings. Unfortunately that doesn;t work on our end. After setting the registry settings and restarting the entire server, IIS is still serving HTTP2 instead of HTTP 1.1. To clarify, IIS uses the HTTP2 protocol for https connections and HTTP 1.1 for http connections. But obviously I want HTTP2 when using https connections.

> obviously I want HTTP2 when using https connections.

My understanding from comment #16 is that when IIS receives an attempt at NTLM authentication on a HTTP/2 connection, it closes the connection with a HTTP_1_1_REQUIRED error code.

> My understanding from comment #16 is that when IIS receives an attempt at NTLM authentication on a HTTP/2 connection, it closes the connection with a HTTP_1_1_REQUIRED error code.

You're right the connection is closed with HTTP_1_1_REQUIRED. NTLM is not supported when using HTTP/2 https://blogs.iis.net/davidso/http2 .

Yeah, NTLM is incompatible with HTTP/2.

True, NTLM (nor Kerberos or Negotiate) are supported in HTTP/2. But as #16 mentioned the authorization cycle ends with a net::ERR_UNEXPECTED error. This is displayed on the screen but not for a short amount of time as #16 mentioned. For us that error remains for about 30 seconds before the higher level retry mechanism kicks and ends with a successfully (re)loads. Some of our customers are reporting that this higher level retry mechanism doesn't kick in at all. Nor does a manual reload help for them.

Right, the mechanism doesn't work right right now, hence why this bug is still open. You can work around it by disabling HTTP/2 on your server, since if you use NTLM, HTTP/2 is kind of pointless. But we should also implement this error code right.

bnc/mmenke: What's the story here? Was there an agreement as to how to resolve this bug?

Good day, I had created one of the tickets that was merged into this, because I built a SharePoint 2016 farm with Windows Server 2016 servers and using Claims over Kerberos.
I had previously provided raw logs that were marked secure, but as a reminder, when authenticating to our extranet over SSL TLS 1.2, Chrome first shows an apparent 404 "This page could not be reached" and then quickly redirects successfully. IE and FF do not experience this issue. But if it makes anyone feel any better, Safari can't load it at all. ¯\_(ツ)_/¯ 

davidben:  We disagree on approach - I really don't want to see us bifurcate the socket pools yet again, though I'm not going to block taking such an approach.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cb36131241cda29e45b487b2ef35298e175d947e

commit cb36131241cda29e45b487b2ef35298e175d947e
Author: Bence Béky <bnc@chromium.org>
Date: Tue Sep 19 17:06:57 2017

Use RST_STREAM instead of GOAWAY in HTTP_1_1_REQUIRED unittests.

This is to make unittests more relevant to real world server responses.
In particular, the following NetLogs all capture RST_STREAM instead of
GOAWAY sent by the server:
*  https://crbug.com/685741#c2 
*  https://crbug.com/709827#c2 
*  https://crbug.com/713851#c9 
* https://crbug.com/717329#c8
*  https://crbug.com/756828#c4 

Also, while RFC7540 does not specify, HTTP_1_1_REQUIRED is understood to
refer to a single resource, not the entire connection, therefore it
makes sense to be sent in a RST_STREAM frame.  (Though Chrome acts
identically in the two cases: it drains SpdySession.)

BUG= 685741 

Change-Id: Idba6c28d49e8361f37e9d716fc7ef1f6777b0c7e
Reviewed-on: https://chromium-review.googlesource.com/672606
Reviewed-by: Asanka Herath <asanka@google.com>
Commit-Queue: Bence Béky <bnc@chromium.org>
Cr-Commit-Position: refs/heads/master@{#502872}
[modify] https://crrev.com/cb36131241cda29e45b487b2ef35298e175d947e/net/http/http_network_transaction_unittest.cc
[modify] https://crrev.com/cb36131241cda29e45b487b2ef35298e175d947e/net/spdy/chromium/spdy_network_transaction_unittest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3238f2e118490c9e0c6c9e16fa8fd74b907eceb3

commit 3238f2e118490c9e0c6c9e16fa8fd74b907eceb3
Author: Bence Béky <bnc@chromium.org>
Date: Fri Sep 22 22:44:49 2017

Modify Authentication retry logic.

If the server requests NTLM or Negotiate scheme negotiation over an
HTTP/2 connection, then mark the origin as one that requires HTTP/1.1
(as opposed to HTTP/2) and retry the request without using HTTP/2.
Otherwise the server would reset the retry on HTTP/2 with error code
HTTP_1_1_REQUIRED, resulting in an extra roundtrip, and additionally a
flash of error page before actual reload on Windows.

See example recorded network event logs and the corresponding header
value for "www-authenticate" response header:
*  https://crbug.com/685741#c2  "Negotiate\r\nNTLM"
*  https://crbug.com/709827#c2  "Negotiate"
*  https://crbug.com/713851#c9  "Negotiate\r\nNTLM"
* https://crbug.com/717329#c8 "NTLM"
*  https://crbug.com/756828#c4  "NTLM"

Do not worry about proxies, as proxy authentication is not supported,
see HttpNetworkTransactionTest.ConnectStatus407.

Bug:  685741 

Change-Id: I5740310ddbd9c26880f1e13cf8303229153acc80
Reviewed-on: https://chromium-review.googlesource.com/673091
Reviewed-by: Asanka Herath <asanka@chromium.org>
Commit-Queue: Bence Béky <bnc@chromium.org>
Cr-Commit-Position: refs/heads/master@{#503890}
[modify] https://crrev.com/3238f2e118490c9e0c6c9e16fa8fd74b907eceb3/net/http/http_auth_controller.cc
[modify] https://crrev.com/3238f2e118490c9e0c6c9e16fa8fd74b907eceb3/net/http/http_auth_controller.h
[modify] https://crrev.com/3238f2e118490c9e0c6c9e16fa8fd74b907eceb3/net/http/http_network_transaction.cc
[modify] https://crrev.com/3238f2e118490c9e0c6c9e16fa8fd74b907eceb3/net/http/http_network_transaction_unittest.cc
[modify] https://crrev.com/3238f2e118490c9e0c6c9e16fa8fd74b907eceb3/net/http/http_stream_factory_impl_job.cc

Yining,

I think we can easily add test coverage for this scenario in either GCE or Skytap.

Thanks,
Yang

Re comment #10 "Oh, yeah you cannot just randomly mutate the SSLConfig at HttpNetworkTransaction. That breaks pooling assumptions across the stack.": unfortunately this is now done in HttpNetworkTransaction::HandleHttp11Required(), at https://cs.chromium.org/chromium/src/net/http/http_network_transaction.cc?q=HttpServerProperties::ForceHTTP11.  Note that the problems that you foresaw include binding to pre-existing connections (like pre-connect) without consideration to the ALPN tokens in SSLConfig.

Separately, it has been brought to my attention that IIS might set a cookie after successful negotiation, and requests bearing that cookie can be served on any connection, including HTTP/2.