CFI failure in WebRtcBrowserTest.CanForwardRemoteStream |
|||||
Issue descriptionI've run CFI trybot to test the new Clang toolchain with LTO-ed Gold plugin, and it failed 3 times on a single test case: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_cfi_rel_ng/builds/83 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_cfi_rel_ng/builds/82 https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_cfi_rel_ng/builds/80 It seems to be persistent and not a flake. So far, I failed to reproduce this locally, but I have not given up yet. I don't think that this issue is a blocker for rolling out new Clang toolchain, as all but one tests pass, and this only affects CFI, which is not in production yet. And we can always add something to cfi blacklist to keep the bot green.
,
Apr 9 2016
Removed 3 comments above, as I was able to reproduce the issue locally with the binary built on the try bot.
,
Apr 12 2016
There're hints that something happens when you changes the Gold plugin (LTO-ed/not LTO-ed) without a clobber. For example, a CFI try job fails on a whitespace change, if there was LTO-ed plugin on that slave previously: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_cfi_rel_ng/builds/95 After clobber: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_cfi_rel_ng/builds/96 I plan to make a local build like that and try to get the compile error like above.
,
Apr 12 2016
Reproduced the test failure locally: CFI: Most likely a control flow integrity violation; for more information see: https://www.chromium.org/developers/testing/control-flow-integrity #0 0x000004340ca8 base::debug::(anonymous namespace)::StackDumpSignalHandler() #1 0x7ffff340f340 <unknown> #2 0x00000409608f content::MediaStreamAudioSource::From() #3 0x0000040e3a14 content::PeerConnectionDependencyFactory::CreateRemoteAudioTrack() #4 0x000004096981 content::MediaStreamCenter::didCreateMediaStreamTrack() #5 0x000001f446c2 blink::MediaStreamCenter::didCreateMediaStreamTrack() #6 0x00000370e4ab blink::MediaStreamTrack::clone() #7 0x00000370bf13 blink::MediaStream::clone() #8 0x00000397551c blink::MediaStreamV8Internal::cloneMethodCallback() #9 0x00000210703d v8::internal::FunctionCallbackArguments::Call() #10 0x000002156633 v8::internal::(anonymous namespace)::HandleApiCallHelper<>() #11 0x00000217687e v8::internal::Builtin_Impl_HandleApiCall() #12 0x092a48709327 <unknown> r8: 00007fffffff9678 r9: 00007fffffff8cf8 r10: 00002b18e082fbc8 r11: 0000000000000246 r12: 0000000000000000 r13: 0000104c39436710 r14: 00007fffffff9cb8 r15: 00007fffffff9cb8 di: 00007fffffff8cf8 si: 00002b18e082f928 bp: 00007fffffff8cd0 bx: 00007fffffff8cf8 dx: 0000000001e62d70 ax: 00000cc437b6cd80 cx: 0000000005a0c6d0 sp: 00007fffffff8cc0 ip: 000000000409608f efl: 0000000000010206 cgf: 0000000000000033 erf: 0000000000000000 trp: 0000000000000006 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] ../../content/test/webrtc_content_browsertest_base.cc:71: Failure Value of: ExecuteScriptAndExtractString( shell()->web_contents(), javascript, &result) Actual: false Expected: true Failed to execute javascript callAndForwardRemoteStream({video: true, audio: true});. From javascript: (nothing) When executing 'callAndForwardRemoteStream({video: true, audio: true});' ../../content/test/webrtc_content_browsertest_base.cc:94: Failure They key was to use exactly the same flags to GN (minus goma) to build the tests. This still does not tell us anything about the compile error above, though.
,
Apr 12 2016
I mean, the same flags as the trybot: gn gen out/gn-cfi '--args=is_cfi=true is_debug=false is_component_build=false symbol_level=1 dcheck_always_on=true' --check
,
Apr 13 2016
The stacktrace from the renderer: Breakpoint 1, 0x000000000048ea14 in __ubsan_handle_cfi_check_fail () (gdb) bt #0 0x000000000048ea14 in __ubsan_handle_cfi_check_fail () #1 0x0000000004304aea in content::MediaStreamAudioSource::From(blink::WebMediaStreamSource const&) () at ../../content/renderer/media/media_stream_audio_source.cc:37 #2 0x0000000004367799 in content::PeerConnectionDependencyFactory::CreateRemoteAudioTrack(blink::WebMediaStreamTrack const&) () at ../../content/renderer/media/webrtc/peer_connection_dependency_factory.cc:596 #3 0x000000000430525a in content::(anonymous namespace)::CreateNativeAudioMediaStreamTrack(blink::WebMediaStreamTrack const&, content::PeerConnectionDependencyFactory*) () at ../../content/renderer/media/media_stream_center.cc:48 #4 0x0000000004305103 in content::(anonymous namespace)::CreateNativeMediaStreamTrack(blink::WebMediaStreamTrack const&, content::PeerConnectionDependencyFactory*) () at ../../content/renderer/media/media_stream_center.cc:84 #5 0x0000000002214145 in blink::MediaStreamCenter::didCreateMediaStreamTrack(blink::MediaStreamComponent*) () at ../../third_party/WebKit/Source/platform/mediastream/MediaStreamCenter.cpp:121 #6 0x00000000038c24e0 in blink::MediaStreamTrack::clone(blink::ExecutionContext*) () at ../../third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.cpp:169 #7 0x00000000038bfe31 in clone () at ../../third_party/WebKit/Source/modules/mediastream/MediaStream.cpp:266 #8 0x0000000003afe66c in cloneMethod () at gen/blink/bindings/modules/v8/V8MediaStream.cpp:321 #9 0x00000000023adefc in Call () at ../../v8/src/api-arguments.cc:16 #10 0x0000000002400164 in HandleApiCallHelper<false> () at ../../v8/src/builtins.cc:4562 #11 0x0000000002421b34 in Builtin_Impl_HandleApiCall () at ../../v8/src/builtins.cc:4580 #12 0x000000000242190b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) () at ../../v8/src/builtins.cc:4577
,
Apr 13 2016
Error message: ../../content/renderer/media/media_stream_audio_source.cc:37:10: runtime error: control flow integrity check for type 'content::MediaStreamAudioSource' failed during base-to-derived cast (vtable address 0x000005dbb080) [120277:120293:0412/172744:1647129309752:WARNING:rtp_rtcp_impl.cc(168)] Process: Timeout: No RTCP RR received. 0x000005dbb080: note: vtable is of type 'content::MediaStreamRemoteAudioSource'
,
Apr 13 2016
Note to myself: cmd to run: gdb --args ./out/gn-cfi-diag/content_browsertests --gtest_filter=WebRtcBrowserTest.CanForwardRemoteStream --no-sandbox --renderer-cmd-prefix="xterm -e gdb --args" Then set a breakpoint to __ubsan_handle_cfi_check_fail
,
Apr 13 2016
The code: https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/media/media_stream_audio_source.cc&q=content/renderer/media/media_stream_audio_source.cc&sq=package:chromium&l=37 // static MediaStreamAudioSource* MediaStreamAudioSource::From( const blink::WebMediaStreamSource& source) { if (source.isNull() || source.getType() != blink::WebMediaStreamSource::TypeAudio) { return nullptr; } return static_cast<MediaStreamAudioSource*>(source.getExtraData()); } This issue is probably related to the class comment for content::MediaStreamRemoteAudioSource https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/media/webrtc/media_stream_remote_audio_track.h&q=MediaStreamRemoteAudioSource&sq=package:chromium&type=cs&l=52 // Inheriting from ExtraData directly since MediaStreamAudioSource has // too much unrelated bloat. // TODO(tommi): MediaStreamAudioSource needs refactoring. // TODO(miu): On it! ;-)
,
Apr 13 2016
Yuri, Tomas, Control Flow Integrity (https://www.chromium.org/developers/testing/control-flow-integrity) found an invalid cast from content::MediaStreamRemoteAudioSource to content::MediaStreamAudioSource, which used to be related classes, but later some refactoring happened. Can you please take a look at the stacktrace (#9) and the error report (#10)? They look scary and actionable. Tentatively assigning to mui@, but please find a better owner or reassign back to me, if that does not work for you. Thanks!
,
Apr 13 2016
To reproduce (sorry, the instructions will be awkward, because this bug was only caught in a very specific configuration; it took me more than a day to reproduce locally): 1. Update the checkout to the latest master. 2. Patch this CL: https://codereview.chromium.org/1881693003/ 3. Run 'gclient sync' to download the new toolchain 4. Download gold plugin: build/download_gold_plugin.py 5. Configure the build: gn gen out/gn-cfi-diag '--args=is_cfi=true use_cfi_diag=true is_debug=false is_component_build=false symbol_level=1 dcheck_always_on=true' --check 6. Build content_browsertests: ninja -C out/gn-cfi-diag/ content_browsertests 7. Run the test: gdb --args ./out/gn-cfi-diag/content_browsertests --gtest_filter=WebRtcBrowserTest.CanForwardRemoteStream --no-sandbox --renderer-cmd-prefix="xterm -e gdb --args" 8. After the xterm window pops up: b __ubsan_handle_cfi_check_fail r 9. It will stop when the error occurs. Then you could see the backtrace: bt
,
Apr 14 2016
Yuri, can you please take a look at the bug, please? It's quite webrtc-specific to be able to quickly fix it myself. And there's a good chance that all the needed knowledge is already in your head, given the TODO mentioned above. Thanks!
,
Apr 14 2016
Friendly ping!
,
Apr 14 2016
An amendment to #14: step 2 may be omitted. I was able to reproduce the same problem with the current toolchain as well.
,
Apr 15 2016
I've found the reason why try bot catches the bug, and buildbot does not: dcheck_always_on. On try bot it's dcheck_always_on=true, and the test fails. Buildbot has dcheck_always_on=false and does not see the issue. The invalid cast happens in this DCHECK (it's a part of the stack trace reported in #9): https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/media/webrtc/peer_connection_dependency_factory.cc&q=peer_connection_dependency_factory.cc:596&sq=package:chromium&l=596 DCHECK(MediaStreamAudioSource::From(source));
,
Apr 15 2016
,
Apr 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f92c2a9c1e07302159ebe7aefcb2d5d51b5b2e2c commit f92c2a9c1e07302159ebe7aefcb2d5d51b5b2e2c Author: krasin <krasin@google.com> Date: Fri Apr 15 23:44:42 2016 Disable WebRtcBrowserTest.CanForwardRemoteStream on 'CFI Linux' buildbot. The bug is acknowledged by the code owners, they have a large refactoring in place and would like to come back to the issue when the churn is down. BUG= 601957 Review URL: https://codereview.chromium.org/1895513003 Cr-Commit-Position: refs/heads/master@{#387751} [modify] https://crrev.com/f92c2a9c1e07302159ebe7aefcb2d5d51b5b2e2c/testing/buildbot/chromium.fyi.json
,
Apr 15 2016
Per off-thread discussion, there's an active refactoring happening in media / webrtc land, so this bug will lie low for a month, and then we'll reconsider.
,
May 3 2016
Update: WIP. I hope to have the refactoring done by end of this week, plus or minus a few days. :)
,
May 3 2016
Thank you for the update! Please, don't hesitate to ping me via mail / chat, if you have any troubles with reproducing the issue.
,
May 17 2016
Any updates on this?
,
May 19 2016
Yuri's refactoring CL has landed and is available in the latest canary builds.
,
May 19 2016
Thanks! I tried to reproduce the bug, and it seems that the test now passes. I've created a CL to reenable the test on three bots: https://codereview.chromium.org/1995053002# Once it's submitted and if the bots are fine, I will close the issue.
,
May 20 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/545143c2d3f0b44e3f2d8c118b7402a13d6db631 commit 545143c2d3f0b44e3f2d8c118b7402a13d6db631 Author: krasin <krasin@google.com> Date: Fri May 20 01:36:57 2016 Reenable WebRtcBrowserTest.CanForwardRemoteStream on vptr bots. BUG= 601957 Review-Url: https://codereview.chromium.org/1995053002 Cr-Commit-Position: refs/heads/master@{#394952} [modify] https://crrev.com/545143c2d3f0b44e3f2d8c118b7402a13d6db631/testing/buildbot/chromium.fyi.json
,
May 20 2016
bots are fine: https://build.chromium.org/p/chromium.fyi/builders/CFI%20Linux/builds/5442 |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by krasin@chromium.org
, Apr 9 2016