Crash during run of content_unittests --gtest_filter="*Video*" on Linux Debug |
||||||||
Issue description
Repro steps:
1. Create a Linux Debug build of content_unittests
2. Run content_unittests --gtest_filter="*Video*"
Expected result:
All tests should pass
Actual result:
1 test crashed:
WebContentsVideoCaptureDeviceTest.SafelyStartsUpAfterWebContentsHasGone (../../content/browser/media/capture/web_contents_video_capture_device_unittest.cc:637)
I run these tests all the time, and they were never flaky before. Will probably do a bisect to find out when it started happening.
Interestingly, the crash does not happen when running the test individually. Also does not happen on Release build.
Crash stack trace:
[3479:3479:0118/121024.645573:2598356820157:FATAL:scoped_task_scheduler.cc(246)] Check failed: !TaskScheduler::GetInstance().
#0 0x7ff1a2340afe base::debug::StackTrace::StackTrace()
#1 0x7ff1a23ae9df logging::LogMessage::~LogMessage()
#2 0x0000020c4c14 base::test::ScopedTaskScheduler::ScopedTaskScheduler()
#3 0x00000206443f _ZN4base10MakeUniqueINS_4test19ScopedTaskSchedulerEJPNS_11MessageLoopEEEENS_8internal16MakeUniqueResultIT_E6ScalarEDpOT0_
#4 0x000002063daf content::TestBrowserThreadBundle::CreateThreads()
#5 0x00000206374d content::TestBrowserThreadBundle::Init()
#6 0x0000020634f4 content::TestBrowserThreadBundle::TestBrowserThreadBundle()
#7 0x000002063447 content::TestBrowserThreadBundle::TestBrowserThreadBundle()
#8 0x00000191965c content::(anonymous namespace)::WebContentsVideoCaptureDeviceTest::WebContentsVideoCaptureDeviceTest()
#9 0x0000019195ef content::(anonymous namespace)::WebContentsVideoCaptureDeviceTest_SafelyStartsUpAfterWebContentsHasGone_Test::WebContentsVideoCaptureDeviceTest_SafelyStartsUpAfterWebContentsHasGone_Test()
#10 0x0000019195b7 testing::internal::TestFactoryImpl<>::CreateTest()
#11 0x000000f0661e _ZN7testing8internal12InvokeHelperIdSt5tupleIJEEE12InvokeMethodIN5media23VirtualAudioInputStreamEMS7_FdvEEEdPT_T0_RKS3_
#12 0x00000227c4c2 testing::internal::HandleExceptionsInMethodIfSupported<>()
#13 0x000002271678 testing::TestInfo::Run()
#14 0x000002271c3f testing::TestCase::Run()
#15 0x000002276f8c testing::internal::UnitTestImpl::RunAllTests()
#16 0x000000f05a7e _ZN7testing8internal12InvokeHelperIbSt5tupleIJEEE12InvokeMethodIN5media23VirtualAudioInputStreamEMS7_FbvEEEbPT_T0_RKS3_
#17 0x00000227d972 testing::internal::HandleExceptionsInMethodIfSupported<>()
#18 0x000002276c2f testing::UnitTest::Run()
#19 0x0000020dadc1 RUN_ALL_TESTS()
#20 0x0000020d9cf2 base::TestSuite::Run()
#21 0x00000206e60d content::UnitTestTestSuite::Run()
#22 0x000000723825 _ZN4base8internal13FunctorTraitsIMN7content26AppCacheRequestHandlerTestEFvvEvE6InvokeIPS3_JEEEvS5_OT_DpOT0_
#23 0x000000723741 _ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIRKMN7content26AppCacheRequestHandlerTestEFvvEJPS5_EEEvOT_DpOT0_
#24 0x000001757367 _ZN4base8internal7InvokerINS0_9BindStateIMN7content17UnitTestTestSuiteEFivEJNS0_17UnretainedWrapperIS4_EEEEEFivEE7RunImplIRKS6_RKSt5tupleIJS8_EEJLm0EEEEiOT_OT0_NS_13IndexSequenceIJXspT1_EEEE
#25 0x0000017572ac _ZN4base8internal7InvokerINS0_9BindStateIMN7content17UnitTestTestSuiteEFivEJNS0_17UnretainedWrapperIS4_EEEEEFivEE3RunEPNS0_13BindStateBaseE
#26 0x00000072fb6b base::internal::RunMixin<>::Run()
#27 0x0000020fe3f9 base::(anonymous namespace)::LaunchUnitTestsInternal()
#28 0x0000020fe264 base::LaunchUnitTests()
#29 0x000001757084 main
#30 0x7ff190f44f45 __libc_start_main
#31 0x00000069d5a5 <unknown>
,
Mar 10 2017
Another possible solution: 4.) Remove the DCHECK() that requires |g_task_scheduler| to be nullptr when creating a ScopedTaskScheduler. However, it would not be clear whether or not ScopedTaskScheduler should then clean up |g_task_scheduler| in its destructor.
,
Mar 10 2017
The CL that lead to this issue is probably https://codereview.chromium.org/2627953002. + folks from that review jam@, gab@: Could you take a look and let me know what would be a good way to handle this?
,
Mar 13 2017
+ fdoray
,
Mar 14 2017
"2.) When ChildProcess is destroyed, its destructor does not clean up the global |g_task_scheduler|." This is intended. CONTINUE_ON_SHUTDOWN tasks can continue to run after ChildProcess has been destroyed. This CL fixes this issue: https://codereview.chromium.org/2749543003/
,
Mar 14 2017
#5: Thanks for preparing a CL so quickly. I believe the proposed CL will fix the issue for PepperToVideoTrackAdapterTest. However, this would only fix one symptom of the underlying problem. There may be other tests that create a ChildProcess instance and other tests that create ScopedTaskScheduler instances. With the current design, we may get the same kind of crash randomly depending on the order in which we execute the tests. Is there something we can do to fix the problem as a whole?
,
Mar 21 2017
fdoray@: Any updates on this? I am mostly interested in learning what your plans are.
,
Jun 13 2017
Bumping Priority up to 2, since I keep running into this and I think having unit tests fail depending on the order in which they are run is a serious issue. Running out/Default/content_unittests --gtest_filter="*Video*" on the latest trunk currently produces 4 crashing tests: MediaStreamVideoSourceTest.AddTrackAndStartSource MediaStreamRemoteVideoSourceTest.StartTrack HTMLVideoElementCapturerSourceTest.ConstructAndDestruct PepperToVideoTrackAdapterTest.Open [78102:78102:0613/094948.711195:2504928999760:FATAL:scoped_task_environment.cc(93)] Check failed: !TaskScheduler::GetInstance().
,
Jun 20 2017
I'll create a ChildProcessForTesting class that does the required cleanup.
,
Aug 16 2017
What's the status on the CL [1] ? I ran into this issue again today. Got 3 random tests failing while trying to verify some unrelated change I was working on. After [1] has landed, we still need to migrate all tests that use ChildProcess to ChildProcessForTesting. [1] https://chromium-review.googlesource.com/c/541595
,
Oct 12 2017
Issue 751093 has been merged into this issue.
,
Oct 13 2017
New solution under review: https://chromium-review.googlesource.com/c/chromium/src/+/716817
,
Oct 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/71d8c0a2541c85c818dc38faf7c13811267a284d commit 71d8c0a2541c85c818dc38faf7c13811267a284d Author: Francois Doray <fdoray@chromium.org> Date: Wed Oct 18 13:30:37 2017 Add ScopedTaskEnvironment to media unit tests that use ChildProcess. When ChildProcess is instantiated in a scope where there is no existing TaskScheduler instance, it creates one that will be leaked. Leaking allows CONTINUE_ON_SHUTDOWN tasks to continue running until the very end of the process' life. Unfortunately, leaking a TaskScheduler in a unit test because it prevents other unit tests to instantiate their own TaskScheduler. With this CL, a ScopedTaskEnvironment is instantiated before ChildProcess in media unit tests. That prevents ChildProcess from creating a TaskScheduler that will be leaked (instead, ChildProcess uses the TaskScheduler whose lifetime is managed by ScopedTaskEnvironment). Bug: 682370 Change-Id: I36d57a54e03c103871a596ca3f16eb4fc4307679 Reviewed-on: https://chromium-review.googlesource.com/716817 Commit-Queue: François Doray <fdoray@chromium.org> Reviewed-by: Dale Curtis <dalecurtis@chromium.org> Reviewed-by: Emircan Uysaler <emircan@chromium.org> Cr-Commit-Position: refs/heads/master@{#509758} [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/media_stream_video_capturer_source_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/media_stream_video_renderer_sink_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/media_stream_video_track_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/pepper_to_video_track_adapter_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/rtc_peer_connection_handler_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/user_media_client_impl_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/video_track_to_pepper_adapter_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/webrtc/media_stream_video_webrtc_sink_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/webrtc/webrtc_media_stream_adapter_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/webrtc/webrtc_video_capturer_adapter_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media_capture_from_element/canvas_capture_handler_unittest.cc [modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media_recorder/video_track_recorder_unittest.cc
,
Oct 18 2017
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by chfremer@chromium.org
, Mar 10 2017