Renderer hang crash in WebRtcMediaStreamTrackAdapter[Map] |
|||||||||||||||
Issue descriptionWebRtcMediaStreamTrackAdapterMap::AdapterRef::~AdapterRef acquires a lock on the UI thread and proceeds to do a synchronous invoke on the webrtc signaling thread which is blocked. It is blocked because the webrtc::PeerConnection::SetRemoteDescription leads to a WebRtcMediaStreamTrackADapterMap::GetOrCreateRemoteTrackAdapter call on the signaling thread which attempts to acquire the same lock as above. We get a deadlock. Here's a filtered view of the stack traces: UI RTCPeerConnectionHandler::SetRemoteDescription async post to ST ST webrtc::PeerConnection::SetRemoteDescription ... RTCPeerConnectionHandler::Observer::OnAddStream WebRtcMediaStreamTrackAdapterMap::GetOrCreateRemoteTrackAdapter acquire lock -- BLOCKED -- UI RemoteMediaStreamImpl::Observer::OnChangedOnMainThread RemoteMediaStreamImpl::OnChanged WebRtcMediaStreamTrackAdapterMap::AdapterRef::~AdapterRef <acquire lock> WebRtcMediaStreamTrackAdapter::Dispose ... VideoTrackProxyWithInternal<...>::UnregisterObserver sync invoke on ST -- BLOCKED For full details, view reports 3 and 4 (not 1 and 2!) at: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20product.version%3D%2761.0.3136.0%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=&index=0
,
Jun 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2d67a5f9f90cb31b4358c74e1450f4bc83c7293e commit 2d67a5f9f90cb31b4358c74e1450f4bc83c7293e Author: hbos <hbos@chromium.org> Date: Mon Jun 26 11:30:23 2017 Fix deadlock in WebRtcMediaStreamTrackAdapterMap. BUG= 736725 Review-Url: https://codereview.chromium.org/2958723003 Cr-Commit-Position: refs/heads/master@{#482241} [modify] https://crrev.com/2d67a5f9f90cb31b4358c74e1450f4bc83c7293e/content/renderer/media/webrtc/webrtc_media_stream_track_adapter_map.cc
,
Jun 26 2017
,
Jun 26 2017
,
Jun 27 2017
Issue 737008 has been merged into this issue.
,
Jan 11 2018
Reopening, since it looks like a deadlock is still possible if the UI thread is calling AddStream; call stack looks like: webrtc::internal::SynchronousMethodCall::Invoke(rtc::Location const &,rtc::Thread *) webrtc::PeerConnectionFactoryProxyWithInternal<webrtc::PeerConnectionFactoryInterface>::CreateLocalMediaStream(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &) content::PeerConnectionDependencyFactory::CreateLocalMediaStream(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &) content::LocalWebRtcMediaStreamAdapter::LocalWebRtcMediaStreamAdapter(content::PeerConnectionDependencyFactory *,scoped_refptr<content::WebRtcMediaStreamTrackAdapterMap>,blink::WebMediaStream const &) content::WebRtcMediaStreamAdapter::CreateLocalStreamAdapter(content::PeerConnectionDependencyFactory *,scoped_refptr<content::WebRtcMediaStreamTrackAdapterMap>,blink::WebMediaStream const &) content::WebRtcMediaStreamAdapterMap::GetOrCreateLocalStreamAdapter(blink::WebMediaStream const &) content::RTCPeerConnectionHandler::AddStream(blink::WebMediaStream const &,blink::WebMediaConstraints const &) Here's a crash link: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.url.simplified%20IN%20(%27https%3A%2F%2Fhangouts.google.com%2F*%27%2C%20%27https%3A%2F%2Fmeet.google.com%2F*%27)%20AND%20custom_data.ChromeCrashProto.magic_signature_1.component%20LIKE%20%27src%2Fthird_party%2Fwebrtc%25%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=447ad2e797232f16&index=0#3 In short, the UI thread acquires the lock, and is trying to invoke on the signaling thread, but then the signaling thread is also trying to acquire the lock.
,
Jan 12 2018
,
Jan 16 2018
taking a look while hbos is away
,
Jan 16 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ea0efafb2425865de79501bc3cf76dc539c995d8 commit ea0efafb2425865de79501bc3cf76dc539c995d8 Author: Tommi <tommi@chromium.org> Date: Tue Jan 16 23:29:28 2018 Reduce the lock scope in WebRtcMediaStreamAdapterMap. Specifically this avoids holding the lock inside of GetOrCreateRemoteStreamAdapter and GetOrCreateLocalStreamAdapter while we call out to external implementations (CreateRemoteStreamAdapter and CreateLocalStreamAdapter). Holding the lock could cause us to deadlock with the main thread or webrtc's signaling thread if the lock was needed on the other thread during the nested call. Bug: 736725 Change-Id: I121db4482bd9492de268b062f872b2af1d8484ca Reviewed-on: https://chromium-review.googlesource.com/868656 Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Tommi <tommi@chromium.org> Cr-Commit-Position: refs/heads/master@{#529518} [modify] https://crrev.com/ea0efafb2425865de79501bc3cf76dc539c995d8/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.cc [modify] https://crrev.com/ea0efafb2425865de79501bc3cf76dc539c995d8/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.h
,
Jan 17 2018
@tommi Is this fixed now or is there more work to be done?
,
Jan 17 2018
No more worked planned from my side at least. I think we can call it fixed.
,
Jan 29 2018
hbos, guidou - should the fix be merged to M64?
,
Jan 29 2018
This is likely a duplicate of issue 805869 which can be triggered by calling various RTCPeerConnection APIs inside of a callback. It seems worth merging if there will be more updates.
,
Jan 29 2018
This bug requires manual review: Request affecting a post-stable build Please contact the milestone owner if you have questions. Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), abdulsyed@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Jan 29 2018
Issue 805869 has been merged into this issue.
,
Jan 29 2018
I have verified that this is indeed a duplicate issue and applying patch https://chromium-review.googlesource.com/c/chromium/src/+/868656 on top of 64.0.3282.99 makes the issue not reproducible.
,
Jan 29 2018
Why is this an urgent fix? We've already started rolling out M64 Stable. Can you please confirm why it's needed for 64 vs waiting until 65?
,
Jan 29 2018
Cl listed at #9 is already in M65 branch (3325), correct?
,
Jan 30 2018
Since this is a deadlock, we don't know for sure how frequently users hit this. Even so, there are tens of thousands of crash dumps just from a cursory look: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20expanded_custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27%20AND%20expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20webrtc%3A%3Ainternal%3A%3ASynchronousMethodCall%3A%3AInvoke%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=&index=0 We know for sure that users of high profile websites such as facebook, hangouts and others, are affected by this. The fix is relatively simple, safe and has been verified to address the problem. I.e. low risk. So, while I wouldn't consider this to be as urgent as e.g. a security vulnerability would be, it's relatively likely to happen and when it does, it completely breaks the user experience. Based on that, I think it's important that we get this merged to 64 and bundle it with similarly important fixes in the next update. In my estimate, this issue in itself doesn't justify a new update, but it deserves to be included with a push if we're doing one (which I'm assuming we will be doing).
,
Jan 30 2018
+abdulsyed@ for M64 merge review. PTAL comment #19.
,
Jan 30 2018
Based on the commit position (529518), I'm pretty sure the fix is already in M65. Looks like the fix landed a couple of days before the branch cut. hbos - can you confirm?
,
Jan 30 2018
The date was before the cut and git find-releases confirms that the fix is in M65. commit ea0efafb2425865de79501bc3cf76dc539c995d8 was: initially in 65.0.3323.0 Still requesting merge into M64 as per #19.
,
Jan 30 2018
https://bugs.chromium.org/p/chromium/issues/detail?id=805869#c18
has a reproduction jsfiddle:
https://jsfiddle.net/mbhjkevx/1/
This simply does not work in chrome 63.0.3239.132, hangs in 64.0.3282.119 (beta) and works (shows 'done' in the console) in 65.0.3325.18 (all on linux)
,
Jan 30 2018
continuing comment #19: As far as safety of the patch, I'd say that it's very safe. That has further been verified with baking for the last couple of weeks and further vetted by hbos.
,
Jan 30 2018
Approving merge to M64. Branch:3282
,
Jan 30 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1d9770d818dfa779454bce8210301f9eadbec515 commit 1d9770d818dfa779454bce8210301f9eadbec515 Author: Niklas Enbom <niklase@chromium.org> Date: Tue Jan 30 21:25:04 2018 Reduce the lock scope in WebRtcMediaStreamAdapterMap. Specifically this avoids holding the lock inside of GetOrCreateRemoteStreamAdapter and GetOrCreateLocalStreamAdapter while we call out to external implementations (CreateRemoteStreamAdapter and CreateLocalStreamAdapter). Holding the lock could cause us to deadlock with the main thread or webrtc's signaling thread if the lock was needed on the other thread during the nested call. TBR=tommi@chromium.org (cherry picked from commit ea0efafb2425865de79501bc3cf76dc539c995d8) Bug: 736725 Change-Id: I121db4482bd9492de268b062f872b2af1d8484ca Reviewed-on: https://chromium-review.googlesource.com/868656 Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Tommi <tommi@chromium.org> Cr-Original-Commit-Position: refs/heads/master@{#529518} Reviewed-on: https://chromium-review.googlesource.com/894201 Reviewed-by: Tommi <tommi@chromium.org> Cr-Commit-Position: refs/branch-heads/3282@{#622} Cr-Branched-From: 5fdc0fab22ce7efd32532ee989b223fa12f8171e-refs/heads/master@{#520840} [modify] https://crrev.com/1d9770d818dfa779454bce8210301f9eadbec515/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.cc [modify] https://crrev.com/1d9770d818dfa779454bce8210301f9eadbec515/content/renderer/media/webrtc/webrtc_media_stream_adapter_map.h
,
Jan 31 2018
Thanks Niklas.
,
Jan 31 2018
,
Feb 1 2018
Verified the fix on Mac 10.12.6, Win-10 and Ubuntu 14.04 using Chrome version #64.0.3282.140 as per the comment #23. Attaching screen cast for reference. Did not observe any chrome freeze after navigating to https://jsfiddle.net/mbhjkevx/1/ and allowing access to camera and mic by clicking the elements on the webpage. Hence, the fix is working as expected. Adding the verified labels. Thanks...!!
,
Feb 1 2018
Thanks krajshree.
,
Feb 1 2018
My stable version is M63 and I noticed it does freeze on this version as well. But M64 is already being rolled out so no need to do anything.
,
Feb 24 2018
,
Feb 24 2018
qaz - fyi this issue (736725) has been fixed and rolled into 64. Isn't issue 813218 for 65?
,
Feb 26 2018
,
Apr 12 2018
|
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by hbos@chromium.org
, Jun 26 2017