New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 423609 link

Starred by 55 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: 2018-06-14
OS: ----
Pri: 2
Type: Feature

Blocking:
issue 504024


Show other hotlists

Hotlists containing this issue:
Top-Starred-Bugs


Sign in to add a comment

Support Windows authentication for WebSocket

Project Member Reported by ricea@chromium.org, Oct 15 2014

Issue description

The new WebSocket implementation supports Basic and Digest authentication, but authentication using the NTLM and Negotiate schemes is not supported.

The design doc for WebSocket HTTP auth in general contains useful background: https://docs.google.com/a/chromium.org/document/d/129rLtf5x3hvhP5rayLiSxnEjOXS8Z7EnLJgBL4CdwjI/edit

The old implementation never supported HTTP authentication at all, so this is not a regression.
 
Showing comments 5 - 104 of 104 Older
Jest tested this in the dev stream (Version 40.0.2209.0 dev-m (64-bit)) and it seems to be working there.  Any idea when this will hit the stable release?

Comment 6 by ricea@chromium.org, Nov 5 2014

I wouldn't expect it to work reliably yet, even in dev or canary. In cases where multiple requests are required to perform the authentication, I would expect it to fail. If Chrome has already authenticated to the server via HTTP such that the first request contains enough information to authenticate, then it may work, but it would be very dependant on the environment.
I think that's what I'm seeing now, the page that is making the websocket request has already authenticated (and I assume cached credentials), so this is probably why it's working?  Either way, would this be in the next stable release?

Comment 8 by ricea@chromium.org, Jun 25 2015

Blocking: chromium:504024

Comment 9 by sbe...@gmail.com, Sep 25 2015

Any news on this one?

Comment 10 by tkent@chromium.org, Nov 26 2015

Labels: Cr-Blink-Network-WebSockets

Comment 11 by tkent@chromium.org, Nov 27 2015

Labels: -Cr-Blink-WebSockets

Comment 12 by ricea@chromium.org, Feb 12 2016

Status: Available

Comment 13 by ricea@chromium.org, Feb 12 2016

Owner: ----
any updates on this bug ?
This is a big problem for us as we are implementing signalR and we need the ability for one user to have many tabs of the application open, which is not possible when it fallbacks to SSE/longpolling because of this issue.
This is causing issues for our chrome users ONLY.  IE, Edge and Firefox have all implemented this functionality.  Please let us know when/if this issue plans to be addressed.
My team and I are 6 months into a system rewrite and just found the same thing that the commenter on April 8th found.  We use SignalR for notifications and chat features and we have a requirement to have multiple tabs open.  This is a showstopper for us because of the issue with SSE/longpolling.  Is there a timeline for resolution of this bug?

Thanks, 
Mike Devenney
Labels: -Pri-2 Needs-Feedback Pri-3
Owner: ricea@chromium.org
Status: Assigned (was: Available)
I have thought of a way forward. If someone has an environment where they can reproduce this, and is proficient with Fiddler or another tool which can examine the bytes that the browser and server exchanged, then they can help provide the information needed to fix this. I need a comparison of Chrome's behaviour to other browsers where this works. At least Firefox and Edge on Windows.

The main thing I'd like to know is whether other browsers reuse the same TCP/IP connection after receiving the 401 response from the server. We don't do this in Chrome, and I have doubts about whether it's RFC6455-compliant, but if it's necessary and at least two other major browsers are doing it then I could live with it.

On the other hand, if all browsers are closing the connection and starting a new one after the 401 then we will at least know that that has nothing to do with the problem.
I'm currently having an issue with chrome connecting using websocket using SignalR in my local dev environment. I'm getting "HTTP Authentication failed; no valid credentials available". However, other browsers (IE11, Edge, Firefox) in my machine worked fine.

I also, went to visit https://websocketstest.com/ , only my Chrome failed. Note: Antivirus/ firewall (McAfee) has been disabled during this test.

Windows 10 pro - latest windows update

Attached here are my screenshots
websocket-test.png
600 KB View Download
browser-versions.png
666 KB View Download
#19 Windows authentication failing in Chrome is "normal", in that this bug still isn't fixed. If you don't intend to be requiring authentication in your local dev environment, then switching it off should fix the problem.

https://websocketstest.com/ failing in Chrome is not normal. Please file a separate issue about that.
For ricea@chromium.org -- I am attaching two files--one is Chrome failing to connect to IIS with NTLM active, the next is Firefox succeeding...these are just the request and response headers...sorry doing a full Fiddler trace a bit much...IE11 works but does not trace WebSocket on the network...I will try Edge and see if that provides the same.  Note that it looks like Chrome wants to provide the full NTLM credentials on the first go...Firefox needs 3 HTTP requests before it gets it right, using a key sent by the server combined with local credentials to get the final upgrade working (looks like, anyway)...
Chrome WSS NTLM.txt
1.5 KB View Download
FireFox WS NTLM Handshake.txt
3.5 KB View Download
Here are the headers for Edge -- Edge does it in 2 requests...key looks like the Authorization for the first request is just a stub, then IIS provides a larger hash, good response combines the two...sorry I don't know NTLM protocol, just looking at the nature of the headers...
Edge NTLM WSS handshake.txt
2.2 KB View Download
Sorry for all the comments--forgot to mention, I was testing version 59.0.3071.115 but upgraded to  60.0.3112.78 (Official Build) (64-bit) and same behavior...
#21 Very interesting. I didn't realise that Firefox used

Connection: keep-alive, Upgrade

Also interesting that all the browsers do something different.

Unfortunately, just from the headers, there's no way to see if the connection was reused or not.

Comment 25 by aaron...@gmail.com, Aug 11 2017

is there any kind of work around we can do from JavasScript to force different behavior for WebSockets to work with NTLM in Chrome ? 

I've seen this problem for a few years now and am surprised Chrome still doesn't support NTLM for WebSockets. 

ricea@chromium.org, I can create a simple git repo that reproduces the problem for you to experiment with. I also tried looking at what Fiddler shows between IE and Chrome and there doesn't seem like enough information there to tell if the same TCP/IP connection is being reused. One thing is certain though - WebSockets work with NTLM in IE and not Chrome (tested with 60.0.3112.90), and that makes me sad :( 

Please let me know if there's anything I can do to help troubleshoot and if you'd be interested in that git repo. Thanks,
ie_signalrTest.txt
8.2 KB View Download
chrome_signalrTest.txt
16.7 KB View Download
edge_signalrTest.txt
785 bytes View Download

Comment 26 by ricea@chromium.org, Aug 14 2017

Labels: -Needs-Feedback
#25 Thank you for the additional information.

There's no way to change WebSocket connection behaviour from Javascript. That would be a security problem.

I'm sorry that Fiddler didn't provide enough information. I should have tried it rather than assuming that it did.

I will try to find some time to look at this again sometime soon.
I have been researching this as I am currently adding windows authentication to an angularJS intranet solution using websockets. What I discovered during my research is that if I used a static ip for the websocket connection I would receive the HTTP Authentication failed; no valid credentials available error... however, when I use the fully qualified domain name for the websocket request I did receive an Authorization request header and a successfully CORS negotiation. Hope this information is helpful.

Comment 28 by ste.b...@gmail.com, Sep 26 2017

IE: Connection Keep-Alive during negotiation
Chrome: Connection Closes between token exchange.

This causes a bad security context and the negotiation fails.

Is it possible to keep the connection alive during the authentication phase?

Guys, I really needs this bug get fixed as I am using SignalR in an intranet environment.
Here is the error log from the Chrome console log:

Transports.js:28 WebSocket connection to 'ws://localhost:37933/commonHub?id=79939f44-9418-490d-b88b-e6b0f826948e' failed: HTTP Authentication failed; no valid credentials available

Comment 30 by ricea@chromium.org, Nov 28 2017

#29 I have a tentative plan for how to fix this. However, at the very earliest it will be 6 months before all users have the fix enabled. So you will need a backup plan for the meantime.
Have you been able to make any progress on this bug? Any estimates when this could land?
I don't understand why this issue is not being regarded as serious.
It makes Chrome completely unusable in Windows based intranets using websockets.
Unless there is a work around that I have missed?
6 months from November is way too long to wait
Please increase the priority

Comment 33 by ms.d...@gmail.com, Jan 9 2018

I will have to echo the above comment... this bug is extremely serious. We have talked about dropping chrome support. No good workarounds are available. Please increase priority. 

I have an environment where I can reproduce the issue, and am happy to provide any additional information if it will lead to a quicker resolution.
Same here. We are almost to the point where we're going to recommend Edge as our only supported browser to our customers since this is so bad. Please, ricea, make this happen. I can provide source code on gitHub with simple example to prove the bug if that helps get this done any faster. Thanks for help.
While this is painful, a couple of observations... hopefully, helpful for some.

If you have authenticated against a web page on the same domain, I observed that *usually* this would be enough to make Chrome pass credentials on the WebSocket. We observed that some Chrome instances behaved differently - that is, sometimes the same version, on different boxes work and others not. 

When you authenticate, we would observe entering username and password works fine for the regular HTTP connection and most of the time the WebSocket would also authenticate, but on some machines it would never and you would see the dreaded "no valid credentials available" message in the console. Then we accidentally discovered that by entering <DOMAIN>/<USERNAME> and password, it would consistently authenticate across all machines. I can't see an explanation for this other than some implicit passing of the NT Domain name for the vanilla HTTP connection that wasn't occurring on the WebSocket authentication path - and only on some machine configurations.
Microsoft has also confirmed this issue affects Project Honolulu (web-based administration for Windows machines) hence why Microsoft Edge is preferred. That makes everyone sad.
Cc: rsleevi@chromium.org
Components: Internals>Network>Auth
Not sure if this additional information is helpful or not... What I have discovered is this is tied to "Internet Options/Security/Local intranet/Sites/Advanced" if Chrome does not view the connection as a local Intranet it will not send the negotiation header/send the negotiation header properly/wrong token. If I understand it, the following is what defines a local intranet connection. #1 Direct mapping via the sites list... but I believe there is a bug because when I enter servername here, not FQDN, chrome authenticates to my front end application successfully via Negotiation header... but it does not appear to honor ws://servername entries. #2 The PlainHostName rule (aka "The Dot Rule") this applies to my situation as I have a DNS entry for my application that is not FQDN nor contains a dot, so chrome successfully negotiates both my front end application and ws:// via a negotiation header using this rule. #3 Proxy Bypass List, any entries here will default to local intranet zone. #4 (WPAD) Proxy Script which information can be found on this one. 

All of these scenarios should trigger windows authentication but they do not... I now have a scenario where PC's don't have a "Managed Connection" to the same LAN as my intranet solution, these are non-domain PC's that require access to my intranet solution. I can launch the browser using runas /netonly on a non domain machine physically connected to the corporate LAN and Crhome authenticates via Negotiation header to front end web endpoint as well as the websocket endpoint on the same host. Same scenario on a network with VPN tunnel to corporate network and no DNS to our network, the runas /netonly command does not work unless I add server non FQDN to the intranet zone, and even then only my front end web app will authenticate via Negotiation header and the ws:// will not... 

I am starting to think this issue is more of an internet options bug than it is chrome specific.. even tho I can enter ws://servername into the sites list for the local intranet zone, it doesn't seem to have any impact and doesn't treat the url as a "intranet zone"

Comment 39 by ricea@chromium.org, May 21 2018

Status: Started (was: Assigned)
Project Member

Comment 40 by bugdroid1@chromium.org, May 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c786ad8ac69ef2cfa6d953f3664eb676a37d46df

commit c786ad8ac69ef2cfa6d953f3664eb676a37d46df
Author: Adam Rice <ricea@chromium.org>
Date: Tue May 22 09:49:15 2018

Add a flag to enable WebSocket connection reuse

Add a new feature to about:flags named
#enable-websocket-auth-connection-reuse permitting
handshake connections to be reused after an authentication error.

A new feature flag, "WebSocketHandshakeReuseConnection", enables
WebSocketBasicHandshakeStream::CanReuseConnection() to return true and
changes the value of the Connection header from "Upgrade" to
"keep-alive, Upgrade". CanReuseConnection() delegates to
HttpStreamParser so has the same behaviour as HttpBasicStream().

NTLM authentication is per-connection and so cannot be negotiated
without connection reuse. "Negotiate" in the Kerberos mode should in
principle work without connection reuse, but it seems to be unreliable
in practice. It is expected that with the flag enabled Windows
authentication should work as it does in Firefox and Edge, apart from
minor protocol differences.

The flag defaults to disabled and will initially only be used for manual
testing.

All relevant net_unittests, layout tests and browser tests were run
manually with the flag enabled. Also manually tested against some real
websites.

Bug: 423609
Change-Id: I7ecd183d5ea222078bc2fe60a2cff60911a14d1b
Reviewed-on: https://chromium-review.googlesource.com/1066252
Commit-Queue: Adam Rice <ricea@chromium.org>
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560523}
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/chrome/browser/about_flags.cc
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/chrome/browser/flag_descriptions.cc
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/chrome/browser/flag_descriptions.h
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/net/websockets/websocket_basic_handshake_stream.cc
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/net/websockets/websocket_basic_handshake_stream.h
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/net/websockets/websocket_stream.cc
[modify] https://crrev.com/c786ad8ac69ef2cfa6d953f3664eb676a37d46df/tools/metrics/histograms/enums.xml

Cc: asanka@chromium.org mmenke@chromium.org davidben@chromium.org
Just adding some CCs so folks can be aware of the socket lifetime changes such as comment #40 :)

Comment 42 by ricea@chromium.org, May 24 2018

This is now available for testing in the latest Canary release. In the Canary version, there is an "Enable WebSocket connection reuse for authentication" experiment on the about:flags page. You can switch it to "Enabled" to try the feature.

If you use WebSockets with Windows authentication, it would be very helpful if you could try this out. Ideally I'd like to know what server and authentication method you use, what platform your clients are, and whether switching the experiment between "Enabled" and "Disabled" makes any difference. If your application uses a fallback when WebSockets aren't working then you may need to look in the network panel in devtools to see whether WebSockets are being used or not.

Comment 43 Deleted

I downloaded the latest Canary and tested our application. The authentication error is still there after enabling the experiment.

Server: Windows 2012 R2, IIS8.5
Client: Windows 10 (1703)
Authentication method: IIS is configured to use Negotiate (Kerberos) or NTLM, not sure which one is actually used
Error:
jquery.signalR-2.2.2.min.js:9 WebSocket connection to 'wss://xxx/signalr/connect?transport=webSockets&clientProtocol=1.5&connectionToken=...' failed: HTTP Authentication failed; no valid credentials available

Comment 45 by ricea@chromium.org, May 24 2018

#44 Thank you! It's great to get such rapid feedback. Disappointing that the result wasn't more positive.

Out of interest, does the same thing work when you try it in Firefox?

Comment 46 Deleted

Hey Bart - what happens if you force an authentication prompt through the IE settings, then enter domain credentials (<domain>/<user>, not just <user>)?
To add some information to the mix: We belong to a parent company and as such we have a cost code which becomes part of our naming schema so our internal DNS entries go: AppName.costcode.company.com, however, the AD domain is just company.com.

Chrome will not pass single sign-on data to the WebSockets on this website with this binding, however, if I go to machinename.company.com:port it works fine. Seems like Chrome isn't identifying an internal server correctly.

Comment 49 by ricea@chromium.org, May 25 2018

#48 Could you share netlogs (https://www.chromium.org/for-testers/providing-network-details) of a success case and a failure case? It would be extremely helpful. If the netlog might contain private information you should send it to me rather than post it to the bug.

Maybe there is something we can rescue from this after all.

If there's no positive signs after a week or so I will remove the flag again. No point in wasting space in the binary on something that doesn't work.
#45 I don't see any errors in the Firefox console.

#47 Unfortunately, I cannot change these settings at this time (policy). I'll get back to you if/when I can.

#48 Interesting, I'm seeing similar behavior here. The authentication error only appears on our PROD server, not on our TEST server.
Both servers are accessed via DNS aliases: https://xxx.company.com/app for PROD, https://test.xxx.company.com/app for TEST.
TEST is located on site, PROD is in a different location.
Access to PROD via http://machinename.domain.com/app works fine.

#49 I will share netlogs with you via email
Hi, i got the same bug on a IIS 10 Windows Server 2016 (http\ws).

I want to you give a info that i found and even can confirm. Found on https://stackoverflow.com/questions/26863242/signalr-mvc-5-websocket-no-valid-credentials .It seems that if you use https/wws all works fine (tested with Chrome Version 67.0.3396.62).
Project Member

Comment 52 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6f75c0f495ac7358e0bab2aa180ed90f32691917

commit 6f75c0f495ac7358e0bab2aa180ed90f32691917
Author: Adam Rice <ricea@chromium.org>
Date: Mon Jun 04 08:00:05 2018

Change the way SetWebSocketKeyForTesting is called

Previously SetWebSocketKeyForTesting() was called from
TestWebSocketHandshakeStreamCreateHelper::OnBasicStreamCreated(). However,
in future a WebSocketBasicHandshakeStream object will need to recreate
itself inside WebSocketBasicHandshakeStream::RenewStreamForAuth(). That
method doesn't have access to the WebSocketHandshakeStreamCreateHelper
object to call OnBasicStreamCreated().

Remove WebSocketHandshakeStreamCreateHelper::OnBasicStreamCreated() and
instead split WebSocketStreamRequest::OnHandshakeStreamCreated() into two
methods, OnBasicHandshakeStreamCreated() and
OnHttp2HandshakeStreamCreated(). During normal execution they do the same
thing, but in tests OnBasicHandshakeStreamCreated() calls
SetWebSocketKeyForTesting(). This is not needed for the Http2 variant as
the H2/WS protocol does not use this header.

Also, move OnBasicHandshakeStreamCreated(), OnHttp2HandshakeStreamCreated()
and OnFailure() methods from WebSocketStreamRequest to a new subclass,
WebSocketStreamRequestAPI. It doesn't make sense to expose these methods
to the caller of WebSocketStream::CreateAndConnectStream().
WebSocketStreamRequestAPI is used in place of the base class inside the
handshake implementation.

CreateAndConnectStreamForTesting() has to use the real
WebSocketStreamRequestImpl implementation and so cannot call
SetWebSocketKeyForTesting() directly. Instead, add a new parameter to
CreateAndConnectStreamForTesting(), |api_delegate|, providing an object
which WebSocketStreamRequestImpl calls for the methods on
WebSocketStreamRequestAPI. This is used by
WebSocketStreamCreateTestBase::CreateAndConnectStream() to inject a
TestWebSocketStreamRequestAPI object.

DummyWebSocketStreamRequest is renamed TestWebSocketStreamRequestAPI as
now that it calls SetWebSocketKeyForTesting() it is no longer just a
dummy object but actually does something.

BUG=423609

Change-Id: I5ed12ef2097f2bfc500b20f394e0da46234d4fd5
Reviewed-on: https://chromium-review.googlesource.com/1077868
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Commit-Queue: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564013}
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_basic_handshake_stream.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_basic_handshake_stream.h
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_channel_test.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_handshake_stream_create_helper.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_handshake_stream_create_helper.h
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_handshake_stream_create_helper_test.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_http2_handshake_stream.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_http2_handshake_stream.h
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_stream.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_stream.h
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_stream_create_test_base.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_test_util.cc
[modify] https://crrev.com/6f75c0f495ac7358e0bab2aa180ed90f32691917/net/websockets/websocket_test_util.h

Project Member

Comment 53 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8b382c6089f6e7a03067c72b8f0068ab10ff1b12

commit 8b382c6089f6e7a03067c72b8f0068ab10ff1b12
Author: Adam Rice <ricea@chromium.org>
Date: Mon Jun 04 12:36:39 2018

Implement RenewStreamForAuth for WebSockets

An implementation of HttpStream::RenewStreamForAuth() is needed for
connection reuse for authentication. Add an implementation to
WebSocketBasicHandshakeStream. Without the
"WebSocketHandshakeReuseConnection" feature enabled, the method returns
nullptr as before.

Add the WebSocketStreamCreateBasicAuthTest.SuccessfulConnectionReuse
test to verify that connection reuse is really working.

BUG=423609

Change-Id: I92d247c5204646f82894bce6c691f4711c571d0e
Reviewed-on: https://chromium-review.googlesource.com/1077876
Commit-Queue: Adam Rice <ricea@chromium.org>
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564066}
[modify] https://crrev.com/8b382c6089f6e7a03067c72b8f0068ab10ff1b12/net/websockets/websocket_basic_handshake_stream.cc
[modify] https://crrev.com/8b382c6089f6e7a03067c72b8f0068ab10ff1b12/net/websockets/websocket_basic_handshake_stream.h
[modify] https://crrev.com/8b382c6089f6e7a03067c72b8f0068ab10ff1b12/net/websockets/websocket_stream_test.cc

NextAction: 2018-06-05
#44 Could you retest with canary version 69.0.3450.0? There's a better chance it will work now.
#55 This fixed the issue for me.

Awesome job, Adam! Thanks!

Comment 57 Deleted

Comment 58 by boul...@gmail.com, Jun 5 2018

Hello,

I've tried the latest canary version (69.0.3450.0) and it works.

Tested with ws:// on these environments : 
local IIS.
Server on the local network behind a load balancer (haproxy).

Great job, Thanks!

#56, #58 Thanks for checking so quickly!

I will try to get this merged to version 68 (dev channel) so that I can start compatibility & stability experiments.
The NextAction date has arrived: 2018-06-05
Still seeing issues here with 69.0.3450.0, will get you a trace right now. Just wanted to raise a flag sooner rather than later.
net-export from 69.0.3450.0: https://gist.githubusercontent.com/riverar/48e101072b2545e1a67797f2f25753ef/raw/4eaa04938f06cc21488a9944d59d168a392fdd79/chrome-net-export-log.json

Accessing IIS on a locally networked machine (https://ryzen) running Windows Admin Center (previously Project Honolulu) with untrusted Root CA.
#62 The trace shows that authentication is succeeding and two WebSocket connections are successfully established.

Could you be more specific about what issues you're seeing?
Strange. I'm asked four times for credentials (one of which being a custom page, so likely doesn't count). I can set up a dummy server for you to hit directly if you wish. I'm also happy to throw this back over the fence to Microsoft to re-test with your fixes -- perhaps we're hitting problems in their realm now.
Labels: -Pri-3 Merge-Request-68 Pri-2
Requesting merge of #52 and #53 to M68.

The M68 branch has a non-working version of the WebSocketHandshakeReuseConnection feature. This is exposed in about:flags, so to avoid confusion it needs to be either rolled back or rolled forward.

I would prefer to roll forward since it will allow me to start experiments to verify stability and compatibility sooner. It will also facilitate wider-scale testing by users.

Rolling forward means merging
#52 https://chromium-review.googlesource.com/1077868
#53 https://chromium-review.googlesource.com/1077876

#52 is a refactoring to enable testing #53. It is fully covered by existing tests. #53 is the actual fix, together with a test. It only changes behaviour when the WebSocketHandshakeReuseConnection feature is enabled, so I have high confidence it is safe. It includes a test to verify it actually works. Existing tests ensure that nothing is regressed.
#64 Oh, so maybe it's not picking up your ambient (SSO) credentials properly? That may indeed be an issue with the way that Chrome is interfacing with Microsoft's GSSAPI libraries.

If you could set up a dummy server I can test against that would be awesome. But if it is a GSSAPI problem then that's quite far from my area of expertise, so I can't promise too much.
Thanks for the fix. Since this bug has been present for quite a while, and we are ready to promote M68 to Beta, can we target M69 for this?
Project Member

Comment 68 by sheriffbot@chromium.org, Jun 7 2018

Labels: -Merge-Request-68 Hotlist-Merge-Approved Merge-Approved-68
Your change meets the bar and is auto-approved for M68. Please go ahead and merge the CL to branch 3440 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), kariahda@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
#67 In that case what I will probably do is make a CL that just removes the entry from about:flags but does nothing else. Then I can merge that CL to M68 to avoid confusion by having a flag that works in some versions not in others. Once it's safely landed in M68 I can then revert in M69 and resume testing in M69.
Cc: abdulsyed@chromium.org
#67 https://chromium-review.googlesource.com/c/chromium/src/+/1090700 is a WIP CL to remove the flag from about:flags. I will merge it to M68 once it has landed and baked in canary.

This means that for about a week canary releases will be missing the flag. I will put it the flag back in M69 once it is safely removed from M68.
Project Member

Comment 71 by sheriffbot@chromium.org, Jun 11 2018

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 72 by bugdroid1@chromium.org, Jun 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7e3874d48e23ea0ef6f1464763cfb08b48ce86d4

commit 7e3874d48e23ea0ef6f1464763cfb08b48ce86d4
Author: Adam Rice <ricea@chromium.org>
Date: Wed Jun 13 05:34:51 2018

Temporarily remove enable-websocket-auth-connection-reuse from about:flags

The M68 branch has a non-working implementation of the
enable-websocket-auth-connection-reuse flag. In order to avoid the
confusion of having a flag that works in some versions and not in
others, temporarily remove the flag from about:flags.

Once the flag has been disabled in the M68 branch it will be re-enabled
for M69 by reverting this CL.

BUG=423609

Change-Id: Id17ac18e13effcfd5bd946dc1fe0f1c70df96539
Reviewed-on: https://chromium-review.googlesource.com/1090700
Commit-Queue: Adam Rice <ricea@chromium.org>
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566730}
[modify] https://crrev.com/7e3874d48e23ea0ef6f1464763cfb08b48ce86d4/chrome/browser/about_flags.cc
[modify] https://crrev.com/7e3874d48e23ea0ef6f1464763cfb08b48ce86d4/chrome/browser/flag_descriptions.cc
[modify] https://crrev.com/7e3874d48e23ea0ef6f1464763cfb08b48ce86d4/chrome/browser/flag_descriptions.h

Comment 73 by ricea@chromium.org, Jun 13 2018

NextAction: 2018-06-14
I will merge #72 to M68 tomorrow if there are no new crashes.
The NextAction date has arrived: 2018-06-14
Project Member

Comment 75 by bugdroid1@chromium.org, Jun 14 2018

Labels: -merge-approved-68 merge-merged-3440
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9c41da5a513d86b299a72c231cccd42425d1cf23

commit 9c41da5a513d86b299a72c231cccd42425d1cf23
Author: Adam Rice <ricea@chromium.org>
Date: Thu Jun 14 15:11:13 2018

M68 merge: Temporarily remove enable-websocket-auth-connection-reuse from about:flags

The M68 branch has a non-working implementation of the
enable-websocket-auth-connection-reuse flag. In order to avoid the
confusion of having a flag that works in some versions and not in
others, temporarily remove the flag from about:flags.

Once the flag has been disabled in the M68 branch it will be re-enabled
for M69 by reverting this CL.

BUG=423609

TBR=yhirano@chromium.org

Change-Id: Id17ac18e13effcfd5bd946dc1fe0f1c70df96539
Reviewed-on: https://chromium-review.googlesource.com/1090700
Commit-Queue: Adam Rice <ricea@chromium.org>
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#566730}(cherry picked from commit 7e3874d48e23ea0ef6f1464763cfb08b48ce86d4)
Reviewed-on: https://chromium-review.googlesource.com/1101117
Reviewed-by: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/branch-heads/3440@{#360}
Cr-Branched-From: 010ddcfda246975d194964ccf20038ebbdec6084-refs/heads/master@{#561733}
[modify] https://crrev.com/9c41da5a513d86b299a72c231cccd42425d1cf23/chrome/browser/about_flags.cc
[modify] https://crrev.com/9c41da5a513d86b299a72c231cccd42425d1cf23/chrome/browser/flag_descriptions.cc
[modify] https://crrev.com/9c41da5a513d86b299a72c231cccd42425d1cf23/chrome/browser/flag_descriptions.h

Comment 76 by aaron...@gmail.com, Jun 19 2018

Thanks so much for getting this working, I really appreciate the effort. Just wondering if this is still on track to be included in M69 ? I downloaded latest yesterday & noticed the flag not there (as expected from above comments). Will this be turned on by default when it is merged in fully? 

Comment 77 by ricea@chromium.org, Jun 20 2018

#76 Thanks for the reminder. I will put the flag back today, then start experiments to confirm that it doesn't break anything else.
Project Member

Comment 78 by bugdroid1@chromium.org, Jun 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed63fc04e03c017198e9956b74883d3c3cb501d5

commit ed63fc04e03c017198e9956b74883d3c3cb501d5
Author: Adam Rice <ricea@chromium.org>
Date: Wed Jun 20 05:57:46 2018

Revert "Temporarily remove enable-websocket-auth-connection-reuse from about:flags"

This reverts commit 7e3874d48e23ea0ef6f1464763cfb08b48ce86d4.

Reason for revert: The flag has now been removed from the about:flags page on M68 where it doesn't work. Re-introduce it in M69 where it works.

Original change's description:
> Temporarily remove enable-websocket-auth-connection-reuse from about:flags
> 
> The M68 branch has a non-working implementation of the
> enable-websocket-auth-connection-reuse flag. In order to avoid the
> confusion of having a flag that works in some versions and not in
> others, temporarily remove the flag from about:flags.
> 
> Once the flag has been disabled in the M68 branch it will be re-enabled
> for M69 by reverting this CL.
> 
> BUG=423609
> 
> Change-Id: Id17ac18e13effcfd5bd946dc1fe0f1c70df96539
> Reviewed-on: https://chromium-review.googlesource.com/1090700
> Commit-Queue: Adam Rice <ricea@chromium.org>
> Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#566730}

TBR=ricea@chromium.org,yhirano@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 423609
Change-Id: Ifb83673ab8e6efaefbbc50e85b560b889fc160ad
Reviewed-on: https://chromium-review.googlesource.com/1107437
Reviewed-by: Adam Rice <ricea@chromium.org>
Reviewed-by: Matt Falkenhagen <falken@chromium.org>
Commit-Queue: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#568744}
[modify] https://crrev.com/ed63fc04e03c017198e9956b74883d3c3cb501d5/chrome/browser/about_flags.cc
[modify] https://crrev.com/ed63fc04e03c017198e9956b74883d3c3cb501d5/chrome/browser/flag_descriptions.cc
[modify] https://crrev.com/ed63fc04e03c017198e9956b74883d3c3cb501d5/chrome/browser/flag_descriptions.h

Comment 79 by ricea@chromium.org, Jun 20 2018

Cc: nikunjb@chromium.org

Comment 80 by ricea@chromium.org, Jun 21 2018

I have added a Chrome status entry for the flag: https://www.chromestatus.com/feature/4885239100866560

The flag will be enabled for 5% of the dev channel once it updates to a newer version (probably in a day or two).

If you have users using the dev channel, they may report problems due to suddenly using the WebSocket code path when they were always using the fallback before. I'm interested in how prevalent such issues are, so please report here if you see anything that looks related. The easiest way to confirm an issue is related is to manually change the flag in about:flags and see if the issue goes away or comes back.

This only applies to the dev channel right now. Any problems you see in beta or stable channels are unrelated.

Comment 81 by ricea@chromium.org, Jun 28 2018

Update: due to various difficulties, the experiment hasn't actually started yet. I will update this issue once the experiment has actually started.
The experiment has started and is ramping up. Since it's only at 5% of the dev channel, not a lot of people are affected.
Results of the first experiment:

No effect on stability.

EMPTY_RESPONSE results increased by 18,000%. I think this was due to unintended connection reuse after failures. I have a CL in progress to fix this.

There was a 70% reduction in 403: Forbidden status codes, but this may be a spurious side-effect of the EMPTY_RESPONSE errors.

I plan to run a new experiment at 10% of dev/canary once my fix has landed.
Project Member

Comment 84 by bugdroid1@chromium.org, Jul 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d4596a8e894e184950edb401b5f9462cb69f590c

commit d4596a8e894e184950edb401b5f9462cb69f590c
Author: Adam Rice <ricea@chromium.org>
Date: Fri Jul 13 08:06:17 2018

Close the WebSocket connection on handshake fail

With the WebSocketHandshakeReuseConnection feature enabled, WebSocket
connections can be reused for auth. However, they can also be returned
to the pool after other kinds of response. A WebSocket connection should
not be reused if the handshake has failed, so explicitly close the
connection on handshake failures.

BUG=423609

Change-Id: If02924395cff1c36b1a172ded26df31840f1358c
Reviewed-on: https://chromium-review.googlesource.com/1127079
Commit-Queue: Adam Rice <ricea@chromium.org>
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Cr-Commit-Position: refs/heads/master@{#574856}
[modify] https://crrev.com/d4596a8e894e184950edb401b5f9462cb69f590c/net/BUILD.gn
[modify] https://crrev.com/d4596a8e894e184950edb401b5f9462cb69f590c/net/websockets/websocket_basic_handshake_stream.cc
[add] https://crrev.com/d4596a8e894e184950edb401b5f9462cb69f590c/net/websockets/websocket_basic_handshake_stream_test.cc
[modify] https://crrev.com/d4596a8e894e184950edb401b5f9462cb69f590c/net/websockets/websocket_handshake_stream_create_helper_test.cc
[modify] https://crrev.com/d4596a8e894e184950edb401b5f9462cb69f590c/net/websockets/websocket_test_util.cc
[modify] https://crrev.com/d4596a8e894e184950edb401b5f9462cb69f590c/net/websockets/websocket_test_util.h

Project Member

Comment 85 by bugdroid1@chromium.org, Jul 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dee384e12aa722fb727b01d45e22329906ca0f24

commit dee384e12aa722fb727b01d45e22329906ca0f24
Author: Adam Rice <ricea@chromium.org>
Date: Fri Jul 13 11:30:29 2018

WS: Stop adding Keep-Alive to Connection header

When the WebSocketHandshakeReuseConnection feature was enabled,
"Keep-Alive" would be included in the Connection header. This behaviour
was copied from Firefox, however it is non-standard.

Microsoft Edge does not include "Keep-Alive" in the Connection header,
and Windows authentication works with WebSockets in Microsoft Edge. So
it is unlikely any server depeneds on the presence of "Keep-Alive".

Stop including "Keep-Alive" in the Connection header when
WebSocketHandshakeReuseConnection is enabled.

BUG=423609

Change-Id: Icaf232872c25b13951c7b22ac8d7f375eeadc9ff
Reviewed-on: https://chromium-review.googlesource.com/1131354
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Commit-Queue: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#574882}
[modify] https://crrev.com/dee384e12aa722fb727b01d45e22329906ca0f24/net/websockets/websocket_stream.cc
[modify] https://crrev.com/dee384e12aa722fb727b01d45e22329906ca0f24/net/websockets/websocket_stream_test.cc

Labels: -OS-All
I removed "Keep-Alive" from the "Connection" header. This should make no difference, but if people who have had success before could retest with the latest canary or dev channel release I would appreciate it.
I updated to canary version 69.0.3495.0 and everything is still working as expected.
#87 Thanks!
I've started the second experiment and it's currently ramping up. It should reach 10% of the dev channel early next week. As before, if you notice any weird WebSocket behaviour on the dev channel please report it here.
The 10% experiment of dev finished with no statistically-significant difference from control. This is good.

I've started a new experiment with 50% of the dev channel enabled, but I don't think it will manage to ramp up as far as 50% before M69 becomes beta. I am going to prepare a new 50% experiment for the beta channel.
Project Member

Comment 91 by bugdroid1@chromium.org, Aug 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d934bde765d4649e8142576837caf07289e4638f

commit d934bde765d4649e8142576837caf07289e4638f
Author: Adam Rice <ricea@chromium.org>
Date: Fri Aug 03 09:20:16 2018

Add WebSocketHandshakeReuseConnection to fieldtrial testing config

Enable WebSocketHandshakeReuseConnection in
fieldtrial_testing_config.json so that it gets testing coverage on the
bots before it becomes the default.

BUG=423609

Change-Id: I692aa12c5685d61a47b26aa014d10db7bb91d98a
Reviewed-on: https://chromium-review.googlesource.com/1151182
Reviewed-by: Steven Holte <holte@chromium.org>
Commit-Queue: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#580484}
[modify] https://crrev.com/d934bde765d4649e8142576837caf07289e4638f/testing/variations/fieldtrial_testing_config.json

I've started the experiment on the beta channel. It's currently at 12% of beta users, ramping up to 50%. If you have any users on the beta channel, they may start seeing different behaviour.
The experiment on the stable channel has started. It is ramping up along with the release of M69 and should eventually reach 1% of the population. If you see any weirdness that you think may be related, please report it here.
Project Member

Comment 95 by bugdroid1@chromium.org, Oct 16

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6612adc3ddebc52aff5ece790b7486b5977624aa

commit 6612adc3ddebc52aff5ece790b7486b5977624aa
Author: Adam Rice <ricea@chromium.org>
Date: Tue Oct 16 18:08:47 2018

Ship WebSocket auth connection reuse

This will permit Windows authentication to work with WebSockets. See
Intent to Ship thread:
https://groups.google.com/a/chromium.org/d/msg/blink-dev/kEtUcy7W0fU/dTyvzUy0BgAJ

Turn on the WebSocketHandshakeReuseConnection feature by default.

Test coverage for the old behaviour has been removed; since the flag
itself will be removed in the next version, this seems acceptable.

BUG=423609

Change-Id: I1373b7b4adcfe695f56a174a7fed2a3bd67ad56b
Reviewed-on: https://chromium-review.googlesource.com/c/1283349
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Commit-Queue: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600045}
[modify] https://crrev.com/6612adc3ddebc52aff5ece790b7486b5977624aa/net/websockets/websocket_basic_handshake_stream.cc
[modify] https://crrev.com/6612adc3ddebc52aff5ece790b7486b5977624aa/net/websockets/websocket_stream_test.cc

Project Member

Comment 96 by bugdroid1@chromium.org, Oct 17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/732d7611de387f10dce741e460e947ba5a762dbb

commit 732d7611de387f10dce741e460e947ba5a762dbb
Author: Adam Rice <ricea@chromium.org>
Date: Wed Oct 17 05:17:42 2018

WebSocketHandshakeReuseConnection experiment group

Change the experiment group for WebSocketHandshakeReuseConnection from
"Enabled" to "Shipped" to keep Finch happy.

Intent to ship thread:

https://groups.google.com/a/chromium.org/d/msg/blink-dev/kEtUcy7W0fU/dTyvzUy0BgAJ

BUG=423609

Change-Id: Ic55912d6b25d43c33b438da799158c792e878790
Reviewed-on: https://chromium-review.googlesource.com/c/1283529
Reviewed-by: Steven Holte <holte@chromium.org>
Commit-Queue: Adam Rice <ricea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600284}
[modify] https://crrev.com/732d7611de387f10dce741e460e947ba5a762dbb/testing/variations/fieldtrial_testing_config.json

This is now shipping in M70. It's basically fixed, but I'd still like more feedback from people who are actually using WebSockets with Windows auth. In particular, there are signs from the metrics that fallback to NTLM may not be working correctly. If someone could test that configuration that would be great.
We experienced problems with Chrome 70 against an internal application that uses websockets with kerberos authentication. We instructed users to set the flag as a workaround.

The server app is a python/aiohttp web application with custom middleware to handle the authentication. The middleware uses the kerberos_sspi library (on Windows) to handle the challenge and response.

If anyone has tips on how we should rewrite the server auth parts to make it work with the default Chrome implementation I would be grateful.
#98 Sorry for the trouble. Including a "Connection: close" header in the "401 Unauthorized" HTTP response from the server will give you the old behaviour without having to set a flag.

I'd like to get more information about why it's failing to try to improve the compatibility of our implementation. Does it work in Firefox or other browsers?

Would it be possible to send me a netlog with from chrome://net-export/ with "Include raw bytes (will include cookies and credentials)" enabled? Unfortunately this will probably contain internal information and there's no guarantee I will be able to identify your problem. So you might not want to do that. You could try looking at chrome://net-internals/#events yourself, but it can be very hard to understand if you are not familiar with it.
Thanks a lot for your help, adding a "Connection: close" header indeed fixes the issue.

Attached are network logs. 1 and 2 are before the server-side code change, 3 is after the server-side code change.

1. chrome-net-export-log-before_changing_server_code.json:
net export, flag Enable WebSocket connection reuse for authentication == DEFAULT.

2. chrome-net-export-log-before_changing_server_code-flag_disabled.json
net export, flag Enable WebSocket connection reuse for authentication == DISABLED.

3. chrome-net-export-log-after_changing_server_code.json
net export, after changing server-side code, flag Enable WebSocket connection reuse for authentication == DEFAULT.

Chrome version: 70.0.3538.77 (Official Build) (64-bit)

Other browsers:

IE11:
* Does NOT work w/ original server code
* Works after server-side code fixes

Firefox 61.0:
* Works w/ original server code
* Works after server-side code fixes

chrome-net-export-log-after_changing_server_code.json
91.1 KB View Download
chrome-net-export-log-before_changing_server_code.json
92.8 KB View Download
chrome-net-export-log-before_changing_server_code-flag_disabled.json
65.1 KB View Download
#100 Thanks! I think you've identified a bug in Chrome: we are trying to retry the request after your server closes the connection, but we're failing to do so.

It will be a while before I can fix it, so I suggest you stick with the "Connection: close" header, which should always work.
I can confirm it works with HTTPS, but still does not work via HTTP. We are using NTLM as Windows authentication.

What I see in Wireshark is this:

To Proxy:
CONNECT echo.websocket.org:80 HTTP/1.1
Host: echo.websocket.org:80
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36


From Proxy:
HTTP/1.1 407 Proxy Access Denied
Expires: 0
Server: WebMarshal Proxy
Cache-Control: no-cache
Proxy-Connection: keep-alive
Via: 1.1 2012R2TEMPLATE
Content-Length: 2233
Content-Type: text/html; charset=utf-8
Proxy-Authenticate: Negotiate
Proxy-Authenticate: NTLM
X-WebMarshal-RequestID: 681C197A-6233-4EAF-B252-5FAE00CE743A


After that, Chrome (Version 70.0.3538.110 (Official Build) (64-bit) with and without #enable-websocket-auth-connection-reuse flag, sends FIN.
#102 Thanks for the report. I will attempt to reproduce this.
Project Member

Comment 104 by bugdroid1@chromium.org, Jan 10 (6 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3745129ebc2a174f7723977e5c6aa4e8f1372ac4

commit 3745129ebc2a174f7723977e5c6aa4e8f1372ac4
Author: Adam Rice <ricea@chromium.org>
Date: Thu Jan 10 15:22:23 2019

Remove WebSocketHandshakeReuseConnection flag and feature

Remove the feature and flag for WebSocketHandshakeReuseConnection. It is
now unconditionally enabled.

After this, the flag no longer appears in about:flags.

BUG=423609

Change-Id: Iefd66092f99654f86a9e61b1fbe7fde8a8665164
Reviewed-on: https://chromium-review.googlesource.com/c/1358312
Commit-Queue: Adam Rice <ricea@chromium.org>
Reviewed-by: Yutaka Hirano <yhirano@chromium.org>
Reviewed-by: Steven Holte <holte@chromium.org>
Cr-Commit-Position: refs/heads/master@{#621576}
[modify] https://crrev.com/3745129ebc2a174f7723977e5c6aa4e8f1372ac4/chrome/browser/about_flags.cc
[modify] https://crrev.com/3745129ebc2a174f7723977e5c6aa4e8f1372ac4/chrome/browser/flag_descriptions.cc
[modify] https://crrev.com/3745129ebc2a174f7723977e5c6aa4e8f1372ac4/chrome/browser/flag_descriptions.h
[modify] https://crrev.com/3745129ebc2a174f7723977e5c6aa4e8f1372ac4/net/websockets/websocket_basic_handshake_stream.cc
[modify] https://crrev.com/3745129ebc2a174f7723977e5c6aa4e8f1372ac4/net/websockets/websocket_basic_handshake_stream.h
[modify] https://crrev.com/3745129ebc2a174f7723977e5c6aa4e8f1372ac4/testing/variations/fieldtrial_testing_config.json

Showing comments 5 - 104 of 104 Older

Sign in to add a comment