New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 736725 link

Starred by 25 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Mac
Pri: 1
Type: Bug
M



Sign in to add a comment

Renderer hang crash in WebRtcMediaStreamTrackAdapter[Map]

Project Member Reported by hbos@chromium.org, Jun 26 2017

Issue description

WebRtcMediaStreamTrackAdapterMap::AdapterRef::~AdapterRef acquires a lock on the UI thread and proceeds to do a synchronous invoke on the webrtc signaling thread which is blocked.

It is blocked because the webrtc::PeerConnection::SetRemoteDescription leads to a WebRtcMediaStreamTrackADapterMap::GetOrCreateRemoteTrackAdapter call on the signaling thread which attempts to acquire the same lock as above.

We get a deadlock. Here's a filtered view of the stack traces:

UI RTCPeerConnectionHandler::SetRemoteDescription
   async post to ST

ST webrtc::PeerConnection::SetRemoteDescription
   ...
   RTCPeerConnectionHandler::Observer::OnAddStream
   WebRtcMediaStreamTrackAdapterMap::GetOrCreateRemoteTrackAdapter
   acquire lock -- BLOCKED

--

UI RemoteMediaStreamImpl::Observer::OnChangedOnMainThread
   RemoteMediaStreamImpl::OnChanged
   WebRtcMediaStreamTrackAdapterMap::AdapterRef::~AdapterRef <acquire lock>
   WebRtcMediaStreamTrackAdapter::Dispose
   ...
   VideoTrackProxyWithInternal<...>::UnregisterObserver
   sync invoke on ST -- BLOCKED

For full details, view reports 3 and 4 (not 1 and 2!) at:
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20product.version%3D%2761.0.3136.0%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=&index=0

 

Comment 1 by hbos@chromium.org, Jun 26 2017

A solution to this problem is to have WebRtcMediaStreamTrackAdapterMap::AdapterRef::~AdapterRef call WebRtcMediaStreamTrackAdapter::Dispose *AFTER* it releases the lock.
Project Member

Comment 2 by bugdroid1@chromium.org, Jun 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2d67a5f9f90cb31b4358c74e1450f4bc83c7293e

commit 2d67a5f9f90cb31b4358c74e1450f4bc83c7293e
Author: hbos <hbos@chromium.org>
Date: Mon Jun 26 11:30:23 2017

Fix deadlock in WebRtcMediaStreamTrackAdapterMap.

BUG= 736725 

Review-Url: https://codereview.chromium.org/2958723003
Cr-Commit-Position: refs/heads/master@{#482241}

[modify] https://crrev.com/2d67a5f9f90cb31b4358c74e1450f4bc83c7293e/content/renderer/media/webrtc/webrtc_media_stream_track_adapter_map.cc

Comment 3 by hbos@chromium.org, Jun 26 2017

Status: Fixed (was: Started)

Comment 4 by hbos@chromium.org, Jun 26 2017

Labels: -Restrict-View-Google
Issue 737008 has been merged into this issue.
Status: Available (was: Fixed)
Reopening, since it looks like a deadlock is still possible if the UI thread is calling AddStream; call stack looks like:

webrtc::internal::SynchronousMethodCall::Invoke(rtc::Location const &,rtc::Thread *)
webrtc::PeerConnectionFactoryProxyWithInternal<webrtc::PeerConnectionFactoryInterface>::CreateLocalMediaStream(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)
content::PeerConnectionDependencyFactory::CreateLocalMediaStream(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)
content::LocalWebRtcMediaStreamAdapter::LocalWebRtcMediaStreamAdapter(content::PeerConnectionDependencyFactory *,scoped_refptr<content::WebRtcMediaStreamTrackAdapterMap>,blink::WebMediaStream const &)
content::WebRtcMediaStreamAdapter::CreateLocalStreamAdapter(content::PeerConnectionDependencyFactory *,scoped_refptr<content::WebRtcMediaStreamTrackAdapterMap>,blink::WebMediaStream const &)
content::WebRtcMediaStreamAdapterMap::GetOrCreateLocalStreamAdapter(blink::WebMediaStream const &)
content::RTCPeerConnectionHandler::AddStream(blink::WebMediaStream const &,blink::WebMediaConstraints const &)

Here's a crash link: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.url.simplified%20IN%20(%27https%3A%2F%2Fhangouts.google.com%2F*%27%2C%20%27https%3A%2F%2Fmeet.google.com%2F*%27)%20AND%20custom_data.ChromeCrashProto.magic_signature_1.component%20LIKE%20%27src%2Fthird_party%2Fwebrtc%25%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=447ad2e797232f16&index=0#3

In short, the UI thread acquires the lock, and is trying to invoke on the signaling thread, but then the signaling thread is also trying to acquire the lock.

Comment 7 by saeedj@google.com, Jan 12 2018

Cc: saeedj@chromium.org

Comment 8 by tommi@chromium.org, Jan 16 2018

Cc: -tommi@chromium.org hbos@chromium.org
Owner: tommi@chromium.org
taking a look while hbos is away
Project Member

Comment 9 by bugdroid1@chromium.org, Jan 16 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ea0efafb2425865de79501bc3cf76dc539c995d8

commit ea0efafb2425865de79501bc3cf76dc539c995d8
Author: Tommi <tommi@chromium.org>
Date: Tue Jan 16 23:29:28 2018

Reduce the lock scope in WebRtcMediaStreamAdapterMap.

Specifically this avoids holding the lock inside of GetOrCreateRemoteStreamAdapter
and GetOrCreateLocalStreamAdapter while we call out to external implementations
(CreateRemoteStreamAdapter and CreateLocalStreamAdapter).

Holding the lock could cause us to deadlock with the main thread or webrtc's signaling thread
if the lock was needed on the other thread during the nested call.

Bug:  736725 
Change-Id: I121db4482bd9492de268b062f872b2af1d8484ca
Reviewed-on: https://chromium-review.googlesource.com/868656
Reviewed-by: Guido Urdaneta <guidou@chromium.org>
Commit-Queue: Tommi <tommi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529518}
[modify] https://crrev.com/ea0efafb2425865de79501bc3cf76dc539c995d8/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.cc
[modify] https://crrev.com/ea0efafb2425865de79501bc3cf76dc539c995d8/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.h

Comment 10 by hbos@chromium.org, Jan 17 2018

@tommi Is this fixed now or is there more work to be done?

Comment 11 by tommi@chromium.org, Jan 17 2018

Status: Fixed (was: Available)
No more worked planned from my side at least. I think we can call it fixed.

Comment 12 by tommi@chromium.org, Jan 29 2018

hbos, guidou - should the fix be merged to M64?

Comment 13 by hbos@chromium.org, Jan 29 2018

Labels: Merge-Request-64 OS-Linux OS-Mac OS-Windows
This is likely a duplicate of  issue 805869  which can be triggered by calling various RTCPeerConnection APIs inside of a callback. It seems worth merging if there will be more updates.
Project Member

Comment 14 by sheriffbot@chromium.org, Jan 29 2018

Labels: -Merge-Request-64 Hotlist-Merge-Review Merge-Review-64
This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 15 by hbos@chromium.org, Jan 29 2018

 Issue 805869  has been merged into this issue.

Comment 16 by hbos@chromium.org, Jan 29 2018

I have verified that this is indeed a duplicate issue and applying patch https://chromium-review.googlesource.com/c/chromium/src/+/868656 on top of 64.0.3282.99 makes the issue not reproducible.
Cc: gov...@chromium.org
Labels: M-64 M
Why is this an urgent fix? We've already started rolling out M64 Stable. Can you please confirm why it's needed for 64 vs waiting until 65?
Cl listed at #9 is already in M65 branch (3325), correct?

Comment 19 by tommi@chromium.org, Jan 30 2018

Since this is a deadlock, we don't know for sure how frequently users hit this. Even so, there are tens of thousands of crash dumps just from a cursory look:

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20expanded_custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=&index=0

We know for sure that users of high profile websites such as facebook, hangouts and others, are affected by this.
The fix is relatively simple, safe and has been verified to address the problem. I.e. low risk.

So, while I wouldn't consider this to be as urgent as e.g. a security vulnerability would be, it's relatively likely to happen and when it does, it completely breaks the user experience.

Based on that, I think it's important that we get this merged to 64 and bundle it with similarly important fixes in the next update. In my estimate, this issue in itself doesn't justify a new update, but it deserves to be included with a push if we're doing one (which I'm assuming we will be doing).
Cc: abdulsyed@chromium.org
+abdulsyed@ for M64 merge review. PTAL comment #19.

Comment 21 by tommi@chromium.org, Jan 30 2018

Based on the commit position (529518), I'm pretty sure the fix is already in M65. Looks like the fix landed a couple of days before the branch cut.

hbos - can you confirm?

Comment 22 by hbos@chromium.org, Jan 30 2018

The date was before the cut and git find-releases confirms that the fix is in M65.
commit ea0efafb2425865de79501bc3cf76dc539c995d8 was:
  initially in 65.0.3323.0

Still requesting merge into M64 as per #19.

Comment 23 by fi...@appear.in, Jan 30 2018

    https://bugs.chromium.org/p/chromium/issues/detail?id=805869#c18
has a reproduction jsfiddle:
    https://jsfiddle.net/mbhjkevx/1/

This simply does not work in chrome 63.0.3239.132, hangs in 64.0.3282.119 (beta) and works (shows 'done' in the console) in 65.0.3325.18 (all on linux)

Comment 24 by tommi@chromium.org, Jan 30 2018

continuing comment #19:

As far as safety of the patch, I'd say that it's very safe. That has further been verified with baking for the last couple of weeks and further vetted by hbos.
Labels: -Merge-Review-64 Merge-Approved-64
Approving merge to M64. Branch:3282
Project Member

Comment 26 by bugdroid1@chromium.org, Jan 30 2018

Labels: -merge-approved-64 merge-merged-3282
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d9770d818dfa779454bce8210301f9eadbec515

commit 1d9770d818dfa779454bce8210301f9eadbec515
Author: Niklas Enbom <niklase@chromium.org>
Date: Tue Jan 30 21:25:04 2018

Reduce the lock scope in WebRtcMediaStreamAdapterMap.

Specifically this avoids holding the lock inside of GetOrCreateRemoteStreamAdapter
and GetOrCreateLocalStreamAdapter while we call out to external implementations
(CreateRemoteStreamAdapter and CreateLocalStreamAdapter).

Holding the lock could cause us to deadlock with the main thread or webrtc's signaling thread
if the lock was needed on the other thread during the nested call.

TBR=tommi@chromium.org

(cherry picked from commit ea0efafb2425865de79501bc3cf76dc539c995d8)

Bug:  736725 
Change-Id: I121db4482bd9492de268b062f872b2af1d8484ca
Reviewed-on: https://chromium-review.googlesource.com/868656
Reviewed-by: Guido Urdaneta <guidou@chromium.org>
Commit-Queue: Tommi <tommi@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#529518}
Reviewed-on: https://chromium-review.googlesource.com/894201
Reviewed-by: Tommi <tommi@chromium.org>
Cr-Commit-Position: refs/branch-heads/3282@{#622}
Cr-Branched-From: 5fdc0fab22ce7efd32532ee989b223fa12f8171e-refs/heads/master@{#520840}
[modify] https://crrev.com/1d9770d818dfa779454bce8210301f9eadbec515/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.cc
[modify] https://crrev.com/1d9770d818dfa779454bce8210301f9eadbec515/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.h

Comment 27 by hbos@chromium.org, Jan 31 2018

Thanks Niklas.

Comment 28 by hbos@chromium.org, Jan 31 2018

Cc: jansson@chromium.org vasanthakumar@chromium.org
 Issue 805852  has been merged into this issue.
Labels: TE-Verified-M64 TE-Verified-64.0.3282.140
Verified the fix on Mac 10.12.6, Win-10 and Ubuntu 14.04 using Chrome version #64.0.3282.140 as per the comment #23.
Attaching screen cast for reference.
Did not observe any chrome freeze after navigating to https://jsfiddle.net/mbhjkevx/1/ and allowing access to camera and mic by clicking the elements on the webpage. 
Hence, the fix is working as expected. 
Adding the verified labels.

Thanks...!!
736725.mp4
1.7 MB View Download

Comment 30 by hbos@chromium.org, Feb 1 2018

Thanks krajshree.

Comment 31 by hbos@chromium.org, Feb 1 2018

My stable version is M63 and I noticed it does freeze on this version as well. But M64 is already being rolled out so no need to do anything.

Comment 32 by q...@chromium.org, Feb 24 2018

Blocking: 813218

Comment 33 by tommi@chromium.org, Feb 24 2018

qaz - fyi this issue (736725) has been fixed and rolled into 64. Isn't issue 813218 for 65?

Comment 34 by q...@chromium.org, Feb 26 2018

Blocking: -813218
This is not related to issue 813218, then.

Comment 35 by hbos@chromium.org, Apr 12 2018

Cc: marchuk@google.com rbasuvula@chromium.org
 Issue 796469  has been merged into this issue.

Sign in to add a comment