New issue
Advanced search Search tips

Issue 682370 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug



Sign in to add a comment

Crash during run of content_unittests --gtest_filter="*Video*" on Linux Debug

Project Member Reported by chfremer@chromium.org, Jan 18 2017

Issue description

Repro steps:
1. Create a Linux Debug build of content_unittests
2. Run content_unittests --gtest_filter="*Video*"

Expected result:
All tests should pass

Actual result:
1 test crashed:
    WebContentsVideoCaptureDeviceTest.SafelyStartsUpAfterWebContentsHasGone (../../content/browser/media/capture/web_contents_video_capture_device_unittest.cc:637)

I run these tests all the time, and they were never flaky before. Will probably do a bisect to find out when it started happening.

Interestingly, the crash does not happen when running the test individually. Also does not happen on Release build.

Crash stack trace:
[3479:3479:0118/121024.645573:2598356820157:FATAL:scoped_task_scheduler.cc(246)] Check failed: !TaskScheduler::GetInstance(). 
#0 0x7ff1a2340afe base::debug::StackTrace::StackTrace()
#1 0x7ff1a23ae9df logging::LogMessage::~LogMessage()
#2 0x0000020c4c14 base::test::ScopedTaskScheduler::ScopedTaskScheduler()
#3 0x00000206443f _ZN4base10MakeUniqueINS_4test19ScopedTaskSchedulerEJPNS_11MessageLoopEEEENS_8internal16MakeUniqueResultIT_E6ScalarEDpOT0_
#4 0x000002063daf content::TestBrowserThreadBundle::CreateThreads()
#5 0x00000206374d content::TestBrowserThreadBundle::Init()
#6 0x0000020634f4 content::TestBrowserThreadBundle::TestBrowserThreadBundle()
#7 0x000002063447 content::TestBrowserThreadBundle::TestBrowserThreadBundle()
#8 0x00000191965c content::(anonymous namespace)::WebContentsVideoCaptureDeviceTest::WebContentsVideoCaptureDeviceTest()
#9 0x0000019195ef content::(anonymous namespace)::WebContentsVideoCaptureDeviceTest_SafelyStartsUpAfterWebContentsHasGone_Test::WebContentsVideoCaptureDeviceTest_SafelyStartsUpAfterWebContentsHasGone_Test()
#10 0x0000019195b7 testing::internal::TestFactoryImpl<>::CreateTest()
#11 0x000000f0661e _ZN7testing8internal12InvokeHelperIdSt5tupleIJEEE12InvokeMethodIN5media23VirtualAudioInputStreamEMS7_FdvEEEdPT_T0_RKS3_
#12 0x00000227c4c2 testing::internal::HandleExceptionsInMethodIfSupported<>()
#13 0x000002271678 testing::TestInfo::Run()
#14 0x000002271c3f testing::TestCase::Run()
#15 0x000002276f8c testing::internal::UnitTestImpl::RunAllTests()
#16 0x000000f05a7e _ZN7testing8internal12InvokeHelperIbSt5tupleIJEEE12InvokeMethodIN5media23VirtualAudioInputStreamEMS7_FbvEEEbPT_T0_RKS3_
#17 0x00000227d972 testing::internal::HandleExceptionsInMethodIfSupported<>()
#18 0x000002276c2f testing::UnitTest::Run()
#19 0x0000020dadc1 RUN_ALL_TESTS()
#20 0x0000020d9cf2 base::TestSuite::Run()
#21 0x00000206e60d content::UnitTestTestSuite::Run()
#22 0x000000723825 _ZN4base8internal13FunctorTraitsIMN7content26AppCacheRequestHandlerTestEFvvEvE6InvokeIPS3_JEEEvS5_OT_DpOT0_
#23 0x000000723741 _ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIRKMN7content26AppCacheRequestHandlerTestEFvvEJPS5_EEEvOT_DpOT0_
#24 0x000001757367 _ZN4base8internal7InvokerINS0_9BindStateIMN7content17UnitTestTestSuiteEFivEJNS0_17UnretainedWrapperIS4_EEEEEFivEE7RunImplIRKS6_RKSt5tupleIJS8_EEJLm0EEEEiOT_OT0_NS_13IndexSequenceIJXspT1_EEEE
#25 0x0000017572ac _ZN4base8internal7InvokerINS0_9BindStateIMN7content17UnitTestTestSuiteEFivEJNS0_17UnretainedWrapperIS4_EEEEEFivEE3RunEPNS0_13BindStateBaseE
#26 0x00000072fb6b base::internal::RunMixin<>::Run()
#27 0x0000020fe3f9 base::(anonymous namespace)::LaunchUnitTestsInternal()
#28 0x0000020fe264 base::LaunchUnitTests()
#29 0x000001757084 main
#30 0x7ff190f44f45 __libc_start_main
#31 0x00000069d5a5 <unknown>

 
Analysis of why this happens:

1.) PepperToVideoTrackAdapterTest creates an instance of ChildProcess, which, in turn, creates and sets the global |g_task_scheduler|.
2.) When ChildProcess is destroyed, its destructor does not clean up the global |g_task_scheduler|.
3.) WebContentsVideoCaptureDeviceTest.SafelyStartsUpAfterWebContentsHasGone creates a ScopedTaskScheduler. The constructor hits a DCHECK, because it expects the global |g_task_scheduler| to be nullptr.

Possible solutions:
1.) Do not use globals. Probably would require too much redesign/refactoring work to be practical.

2.) Have ChildProcess destructor clean up the |g_task_scheduler| if it set one. This may be dangerous, because (based on what the destructor currently does) it appears like certain things are allowed to keep running on |g_task_scheduler| even after it has shut down. Need to get input from someone intimately familiar with this to check whether or not this could be a solution.

3.) Have PepperToVideoTrackAdapterTest manually clean up the |g_task_scheduler| after its ChildProcess instance has been destroyed. Unfortunately, this is not a very scalable solution, since ChildProcess is created in many tests. The same issue may exist with other tests.

Another possible solution:
4.) Remove the DCHECK() that requires |g_task_scheduler| to be nullptr when creating a ScopedTaskScheduler. However, it would not be clear whether or not ScopedTaskScheduler should then clean up |g_task_scheduler| in its destructor.


Cc: gab@chromium.org jam@chromium.org
The CL that lead to this issue is probably https://codereview.chromium.org/2627953002.

+ folks from that review

jam@, gab@: Could you take a look and let me know what would be a good way to handle this?
Cc: fdoray@chromium.org
+ fdoray

Comment 5 by fdoray@chromium.org, Mar 14 2017

Cc: -fdoray@chromium.org chfremer@chromium.org
Owner: fdoray@chromium.org
"2.) When ChildProcess is destroyed, its destructor does not clean up the global |g_task_scheduler|."
This is intended. CONTINUE_ON_SHUTDOWN tasks can continue to run after ChildProcess has been destroyed.

This CL fixes this issue: https://codereview.chromium.org/2749543003/
#5: Thanks for preparing a CL so quickly. I believe the proposed CL will fix the issue for PepperToVideoTrackAdapterTest. However, this would only fix one symptom of the underlying problem. There may be other tests that create a ChildProcess instance and other tests that create ScopedTaskScheduler instances. With the current design, we may get the same kind of crash randomly depending on the order in which we execute the tests. Is there something we can do to fix the problem as a whole?

fdoray@: Any updates on this? I am mostly interested in learning what your plans are.
Labels: -OS-Linux -Pri-3 OS-All Pri-2
Bumping Priority up to 2, since I keep running into this and I think having unit tests fail depending on the order in which they are run is a serious issue.

Running out/Default/content_unittests --gtest_filter="*Video*" on the latest trunk currently produces 4 crashing tests:

MediaStreamVideoSourceTest.AddTrackAndStartSource
MediaStreamRemoteVideoSourceTest.StartTrack
HTMLVideoElementCapturerSourceTest.ConstructAndDestruct
PepperToVideoTrackAdapterTest.Open

[78102:78102:0613/094948.711195:2504928999760:FATAL:scoped_task_environment.cc(93)] Check failed: !TaskScheduler::GetInstance(). 

Comment 9 by fdoray@chromium.org, Jun 20 2017

Status: Started (was: Assigned)
I'll create a ChildProcessForTesting class that does the required cleanup.
What's the status on the CL [1] ?
I ran into this issue again today. Got 3 random tests failing while trying to verify some unrelated change I was working on.

After [1] has landed, we still need to migrate all tests that use ChildProcess to ChildProcessForTesting.

[1] https://chromium-review.googlesource.com/c/541595

Cc: wolenetz@chromium.org dalecur...@chromium.org guidou@chromium.org fdoray@chromium.org
 Issue 751093  has been merged into this issue.
Project Member

Comment 13 by bugdroid1@chromium.org, Oct 18 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/71d8c0a2541c85c818dc38faf7c13811267a284d

commit 71d8c0a2541c85c818dc38faf7c13811267a284d
Author: Francois Doray <fdoray@chromium.org>
Date: Wed Oct 18 13:30:37 2017

Add ScopedTaskEnvironment to media unit tests that use ChildProcess.

When ChildProcess is instantiated in a scope where there is no
existing TaskScheduler instance, it creates one that will be leaked.
Leaking allows CONTINUE_ON_SHUTDOWN tasks to continue running until
the very end of the process' life. Unfortunately, leaking a
TaskScheduler in a unit test because it prevents other unit tests
to instantiate their own TaskScheduler.

With this CL, a ScopedTaskEnvironment is instantiated before
ChildProcess in media unit tests. That prevents ChildProcess
from creating a TaskScheduler that will be leaked (instead,
ChildProcess uses the TaskScheduler whose lifetime is managed
by ScopedTaskEnvironment).

Bug:  682370 
Change-Id: I36d57a54e03c103871a596ca3f16eb4fc4307679
Reviewed-on: https://chromium-review.googlesource.com/716817
Commit-Queue: François Doray <fdoray@chromium.org>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Emircan Uysaler <emircan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509758}
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/media_stream_video_capturer_source_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/media_stream_video_renderer_sink_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/media_stream_video_track_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/pepper_to_video_track_adapter_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/rtc_peer_connection_handler_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/user_media_client_impl_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/video_track_to_pepper_adapter_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/webrtc/media_stream_video_webrtc_sink_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/webrtc/webrtc_media_stream_adapter_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media/webrtc/webrtc_video_capturer_adapter_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media_capture_from_element/canvas_capture_handler_unittest.cc
[modify] https://crrev.com/71d8c0a2541c85c818dc38faf7c13811267a284d/content/renderer/media_recorder/video_track_recorder_unittest.cc

Status: Fixed (was: Started)

Sign in to add a comment