New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 439560 link

Starred by 116 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug



Sign in to add a comment

WebRTC cannot connect to TURN servers through HTTP proxies that require auth

Reported by davidbak...@gmail.com, Dec 5 2014

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36

Steps to reproduce the problem:
Chrome cannot connect to TURN servers via TCP or SSL through an HTTP proxy if that HTTP proxy requires authentication. I'm able to reproduce this by:

1. Setting up a squid HTTP proxy that requires authentication and allows connections to port 3478
2. Setting up a Mac with network access only to the machine hosting the Squid proxy
3. Setting the HTTP and HTTPS proxies in OS X's Network configuration, with username and password
4. Opening chrome, entering credentials for the proxy when prompted
5. Placing a webrtc call

What is the expected behavior?
Chrome should auth successfully with the proxy and jingle should be able to communicate with the TURN server through the proxy.

What went wrong?
The ICE connection state changes to failed after about 10 seconds, so the call fails. No connection is established to the TURN server.

If I edit the Squid config not to require auth, restart it and place another call, the call sets up successfully and media flows through the TURN server.

I am using a coturn TURN server (without SSL) and transport=tcp, specified by IP address (although I observe the same behaviour using an SSL transport).

Did this work before? No 

Chrome version: 41.0.2240.0  Channel: canary
OS Version: OS X 10.10.1
Flash Version: Shockwave Flash 15.0 r0

 Bug 395614  also covers this behaviour but it appears that the change made to fix that bug does not fix the problem.

I am moving webrtc  issue 4076  to here as I believe the problem is in the Chromium code base and I have a patch that fixes it.

 
Labels: Cr-Blink-WebRTC
Owner: jiayl@chromium.org
Status: Assigned
Assigning to jiayl@ since she's reviewing the patch.  
 Issue webrtc:4076  has been merged into this issue.

Comment 5 by jiayl@chromium.org, Dec 8 2014

Owner: rsleevi@chromium.org
The network team is planning to fix this from the net/ layer the last time we talked about it.
Okay, thanks for looking. Assuming the fix in the net layer is slightly longer term, would this be valid way of fixing this in the shorter term? It would make a big difference to a lot of our users who are running in to this problem.
Cc: rsleevi@chromium.org
Labels: Cr-Internals-Network-Proxy
Owner: cbentzel@chromium.org
Status: Untriaged
I'm not actively working on this, nor am I aware of plans to actively work on this. The last time we discussed, the discussions sort of petered out.

I'm pushing to cbentzel@ w/ Untriaged to review for priority & assignment. Chris, the issue here is a disconnect in one of the "uncomfortable" APIs //net exposes regarding exposing "Streams-ish" like sockets. See also https://tools.ietf.org/html/draft-hutton-httpbis-connect-protocol-00 , http://tools.ietf.org/html/draft-hutton-rtcweb-nat-firewall-considerations-03 , and https://tools.ietf.org/html/rfc7065 
Labels: -OS-Mac OS-All
Labels: Cr-Internals-Network-Auth
The impact of this bug is quite significant: many corporate networks firewall all internet traffic other than by passing through an HTTP proxy, which then often auths client connections using NTLM or even Basic HTTP proxy-auth.  Therefore the *only* way to ever get WebRTC media to flow is by tunnelling it through the proxy via an HTTP CONNECT, connecting to a TURN/TLS listener on port 443 on a TURN relay in a manner that is indistinguishable from a proxied HTTPS request.

Dave's patch here seems to address a genuine bug in the current implementation, even if the eventual aim is a rearchitecture.  If it's accepted, it should immediately improve WebRTC connection reliability for a very significant number of corporate users, and should directly help WebRTC confidence and uptake.

Matthew,

Though I agree with you in many of these points, I'm afraid Dave's patch introduces a number of bugs that would be equally undesirable. In the system of tradeoffs, it unfortunately means chosing to not have WebRTC work in your network, rather than introduce bugs for all users on other networks.

This is something that the WebRTC and Networking teams will need to jointly work on, but both are implemented in such a way that this is not trivial.
Ryan: fair enough - i'd missed the comments on codereview.  Thanks for the update.
Status: Assigned
Labels: -Cr-Blink-WebRTC Cr-Blink-Webrtc-Network WebrtcTriaged
Re #7, I thought there was an understanding regarding (at a high level) what needed to be done in the Chrome net stack; I think the thing that was an open question was the timeframe.

Chris, any thoughts on when this could be addressed? This is one of the few remaining environments where WebRTC doesn't work.
I'll try to come up with something early next week. Ping me Monday if haven't responded.
rsleevi: Is there something simple which could be done in isolation for the WebRTC case, or does this need to be part of a major API/impl redo? You've thought a lot about the general issues here.
I don't have any simple ideas, sadly. There's a lot of 'hidden' state between these two, where we want them to have some elements of distinctness (e.g. WebRTC sockets shouldn't count against socket pooling limits), but we also have shared state. We also have a number of caches for things like socket authentication (client certs) and user authentication (HTTP), and we don't have a solid model for 'read only' sharing of that data, nor is read-only sharing an entirely desirable model.

That is, if we share the caches, then if WebRTC declines to provide data for something, the normal pools would interpret it as the _user_ declining. So what we really need is a way to get the WebRTC request for things into the same layers that we "ask" the embedder for things (where the embedder either prompts or defers to the caches), as then we know whatever we receive has been "blessed" by the user. That's the tricky part and why nothing stands out, and the ideas such as read-only caches and copying the data feel like they quickly get full of edge cases that become impossible to create a reasonable prediction model for.
I have encountered this issue could you update us on the status?

Comment 19 by Deleted ...@, Mar 27 2015

Hello, please provide some news on this. I tthought this should have been fixed in 41? We are having the same issues with our User for Circuit where the Proxy enforces Authentication so that no webRTC / no speech audio  video is possible for our Customer. 

Comment 20 by g...@tokbox.com, Oct 21 2015

I faced this issue today in a big corporate network.   I think it is critical to have it fixed.

Find the logs from chrome://network-internals attached.  You can see how a WS connection sends Proxy-Authorization header and it succeeds but a TURN connection receives 407 and don't try to send any authorization header.

It was Chrome 45 in Windows 7.
corporate_proxy_auth_network_internals.txt
4.5 KB View Download

Comment 21 by blum@chromium.org, Oct 26 2015

Cc: pthatcher@chromium.org

Comment 22 by g...@tokbox.com, Oct 29 2015

@pthatcher Any opinion/plan regarding this issue?  

Comment 23 by juberti@webrtc.org, Oct 30 2015

@rsleevi needs to weigh in on whether any solution to the issues mentioned in #17 has emerged over the past year.
No.
Is there any update on this?

We are also facing this issue and our customers are not able to update to webrtc based application as they are behind corporate network.
cbentzel@ this is assigned to you, do you have any plans/eta for this?
Components: Enterprise
Labels: -Pri-2 Pri-3
No plans/eta - sorry for not updating earlier. Moving to Pri-3 until we get this associated with a milestone.
could you add something that allows detecting this state from the JS API at least? Or add it to your UMA tracking.

That would allow quantifying the problem.
I can check and see what existing UMA we have for this.

As an example of the seriousness of this problem, it affects the European Commission and all their agencies which includes, but not limited to:
 
The European Commission: DG HR
The European Commission: DG Employment and Social Affairs 
Joint Research Center (JRC Geel) 
The Court of Auditors
EU Parliament


They are unable to make WebRTC calls for this reason.
Its extremely important to fix.

I looked but couldn't find any counters indicating auth usage within proxies.

Note that NTLM auth is currently supported on Chrome for Windows.
Cc: cbentzel@chromium.org
cbentzel would know for comment #31, and would also be able to address any scheduling/prioritization needs (but unlikely this quarter, and we're very keen to explore removing NTLM support)

Comment 33 by arath...@gmail.com, Apr 13 2016

N.B. that this isn't just a problem for NTLM auth, but any HTTP auth mechanism whatsoever.  Over at matrix.org it continues to be our number one reason for WebRTC failures: we've had many instances of folks behind a corporate squid that prompts for LDAP creds to track identity, and whose only option is to tunnel TURNS through the proxy... were it not for this bug.

Comment 34 by blum@chromium.org, Apr 14 2016

Cc: blum@chromium.org
Any updates on this item? planned fix date?
Thanks


cbentzel@ do you have any news in this area?
There are no updates to share, because no such work has been scheduled for this or next quarter.

Comment 38 by fi...@appear.in, Jul 8 2016

I sampled data from 100k calls using chrome on appear.in (which is running turn with the full spread of udp, tcp and tls on port 443). 956 of those calls fail. This rate might be a service-dependent rate, compare with uma stats.


From the 965 failures, 354 fail because the peer could not gather a relay candidate and another 189 because the local client could not gather a relay. In an environment where websockets work which means either that WebRTC should be tunneled over WS or this is caused by this bug.
The split is unknown (which is why I asked for a way to quantify this in #28 -- if you have ideas for getting better estimates e.g. by looking whether this happens in 10.x.x.x. type networks I am happy to do the legwork).

That is 55% of the failures. The weekly number of webrtc minutes is estimated to be a billion. Multiply that by 0.00956 and 0.55. Also lets assume that mere 20% of the failures would be fixed, so multiply by 0.2. Which means roughly a million minutes per week doesn't happen.


Doesn't that justify giving this a higher priority?

Comment 39 by blum@chromium.org, Aug 23 2016

Cc: gustavh@google.com
This is impacting use in a range of health scenarios where specific requests to change authentication rules the media traffic is the only solution. 

A huge issue is that we have to even propose this type of change, which is considered a hack by many. 

Her is typical exchange:

I have an application which runs in a browser that you don't normally support. Oh, and by the way, it uses a part of the browser that does not respect the users existing authentication to your proxy, so you will have to request a network change to not require proxy auth for these proxy requests (hands over a list of possible request patterns)

If you have concerns for adoption of this technology across corporate and institutional customers, this must be fixed.
 
Also this impacts our business a lot since we are trying to integrate our product into fintech and most of German banks are affected by this bug, so would be nice to have this issue fixed since it is opened for almost 2 years now.
Cc: guidou@chromium.org

Comment 43 by blum@chromium.org, Aug 30 2016

Cc: blumberg@chromium.org
Labels: -Pri-3 M-57 Pri-2
Owner: cfredric@chromium.org
The current technical plan is to copy the credentials for the main request context HttpAuthCache to the WebRTC network context HttpAuthCache, and not support login prompts for WebRTC triggered network connections. This means that there could still be edge cases where this does not work, such as if an extension provides the auth credentials, or if the TURN connection is routed through a different authenticating proxy than network requests. However, it seems like the simplest solution, should address a lot of the cases that the current behavior does not, and there are also product-level concerns over whether we want to show an auth prompt for WebRTC connections particularly if to something different than the top-level origin.

Comment 46 by blum@chromium.org, Jan 26 2017

Cc: huib@chromium.org
Project Member

Comment 47 by bugdroid1@chromium.org, Jan 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/77ec64724b4718f600f0e36d1ab31a6f6b00656d

commit 77ec64724b4718f600f0e36d1ab31a6f6b00656d
Author: asanka <asanka@chromium.org>
Date: Mon Jan 30 22:30:44 2017

[WebRTC] Copy HttpAuthCache from main request context into webrtc context

WebRTC uses ProxyResolvingClientSocket to tunnel through proxies. The
socket creates a new request context using a provided request context as
a base. This CL changes this logic to copy the HttpAuthCache from the
original request context to the next context so that proxies that
require explict credentials can be used.

Based on https://codereview.chromium.org/2552403002 by cfredric.

R=sergeyu@chromium.org
BUG= 439560 

Review-Url: https://codereview.chromium.org/2666713002
Cr-Commit-Position: refs/heads/master@{#447091}

[modify] https://crrev.com/77ec64724b4718f600f0e36d1ab31a6f6b00656d/jingle/glue/proxy_resolving_client_socket.cc
[modify] https://crrev.com/77ec64724b4718f600f0e36d1ab31a6f6b00656d/jingle/glue/proxy_resolving_client_socket_unittest.cc

Cc: cfredric@chromium.org
Owner: asanka@chromium.org
Status: Fixed (was: Assigned)
Labels: -M-57 M-58
Owner: jkallar@chromium.org
jkallar@ could you help verify this issue? Come by when you have time and I'll explain in more detail.

Comment 51 Deleted

The fix is working great.

Configuration: Chrome Stable 56.0.2924.87 and Chrome Canary 58.0.3004.3 on Windows 10 behind a Squid proxy that requires authentication.

Results: the connection to the TURN server is now successful on Canary but still fails on Stable.
jansson@:  I will help to verify this

Comment 54 by g...@tokbox.com, Feb 8 2017

We just tested it in our network where Chrome 56 failed to connect to TURN and Canary 58 is able to connect properly.  Thank you very much!
\o/
a.lebec@ : I'm on Windows 10 (ip address = aaa.xxx.yyx.zzz) using Chrome Dev 58.0.3018.3 and I've set up a Squid proxy with Authentication and I've configured Chrome to use the Squid proxy 

I then make an appRTC (https://appr.tc) call and I'm asked for the Squid proxy credentials. 

I join the appRTC call (https://appr.tc) from a Mac machine running Chrome 56.0.2924.87 Stable

On the Mac machine I see, using web RTC internals (chrome://webrtc-internals), when clicking on "Conn-audio-1-0 (googCandidatePair)":

googRemoteAddress	aaa.xxx.yyx.zzz:62995
googRemoteCandidateType	stun

i.e that the remote address is not the Squid proxy address and STUN and not TURN is being used

Ok, if I look at original bug Reported by davidbaker992@gmail.com, Dec 5 2014 (first post in this bug) and step 2:

"2. Setting up a Mac with network access only to the machine hosting the Squid proxy"

I think I need to do this step but on Windows 10.  I wondered thus a.lebec@ how did this?

Thanks,  Jaspal

jkallar: try adding ?it=relay to the apprtc url and blocking udp. That should should TURN/TCP or TURN/TLS being used.
philipp.hancke@:

Thanks for adding the '?it=relay to the apprtc url' tip.

I tried it in conjunction with creating new rules in windows 10 firewall to block all udp inbound and outbound traffic 

Again I do next:

- make an appRTC (https://appr.tc) call and I'm asked for the Squid proxy credentials. 

- I join the appRTC call (https://appr.tc) from a Mac machine running Chrome 56.0.2924.87 Stable 

On Mac machine I get: 

when clicking on "Conn-audio-1-0 (googCandidatePair)"  [chrome://webrtc-internals]:

googRemoteAddress	aaa.bbb.ccc.ddd:26937
googRemoteCandidateType	relay
.
.
.
googTransportType	udp


So still UDP (even after creating Windows 10 firewall rules) but this time a TURN server ip address.

Although better I still don't get the ip address of my Squid server

This is my first time creating firewall rules in Windows 10, say maybe I did something wrong there

Any pointers?


jkallar@: googTransportType is a very confusing thing --- bear with me (-:

googTransportType describes the type of the port on the TURN server -- and the server opens a UDP port so 'udp' is correct here.
On the win machine Search the stats for the localCandidateId field in the Conn-audio-1-0.
Then search for the Cand-somethingsomethingsomething block indicated in that field. Search for the priority there, do a 24 bit right-shift on that number. The result should be 2, 1 or 0. If its 2 the transport between client and TURN server is UDP and your port block doesn't work. If its 0 (turn/tls) something is wrong because apprtc is not running TURN/TLS now. If its 1 (tcp) you win.

Now if you look at the mac this might still result 2 being shown. Basically you have a connection like this:
win -> turn-over-tcp -> squid -> turn-over-tcp -> turn server -> rtp-over-udp -> another turn server -> turn-over-udp -> mac

Hope that helps!
Tested scenarios:
-----------------

(1) AppRTC call between a Windows 10 VM (caller) with an authenticated proxy to a Mac machine (callee) with no authenticated proxy (windows VM initiated the appRTC call, Mac joins the call)

(2) AppRTC call between Mac machine (caller) with no authenticated proxy to a Windows 10 VM  (callee) with an authenticated proxy (Mac initiated the appRTC call, Windows VM joins the call)

(3) AppRTC call between two Windows 10 VM’s both with an authenticated proxy  


In all of the above scenarios:

— A Linux Squid proxy server was set up with ‘basic’ authentication 
— The Windows 10 VM had all outbound UDP traffic blocked.  
— The Windows 10 VM’s external ip address was that of the Squid proxy ip address (I used www.ip4.me to verify this)
 

Test steps:
—————------

(a) Caller browses to https://appr.tc/?it=relay and enters a room name
(b) Callee browses to https://appr.tc/?it=relay and enters the same room name from step (a) and joins the room
(c) Verify video and audio is flowing between caller and callee
(c) Verify: on windows 10 VM use Wireshark for a connected appRTC call to check we see TCP packets (of variables size) to and from the Squid proxy server
(d) Caller or Callee disconnects the appRTC call (click ‘hang-up’ icon)  


Issues:
———----

In scenarios (1) and (3):

— the callee (party joining the appRTC call) failed to connect to an appRTC call the first time using the same room as the caller  (I tried this many times)
— only the second time did the callee connect to an appRTC call i.e ith aad to refresh the https://appr.tc/?it=relay and enter the same room name again
— ‘chrome://webrtc-internals’ shows ‘iceconnectionstatechange = failed” for the first round appRTC call connect failure.


In scenarios (2):

— The call is connected as expected i.e first time round 


Can anybody say why I see issues in scenarios (1) and (3) and not in scenario (2)
Owner: asanka@chromium.org
ping regarding the last comment
Cc: cvintila@chromium.org
Cc: rantonysamy@chromium.org

Sign in to add a comment