New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 823318 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Use-after-free of WebRtcEventLogManager created by BrowserProcessImpl

Project Member Reported by gab@chromium.org, Mar 19 2018

Issue description

browser_process_impl.cc leaks the instance it creates in PreMainMessageLoopRun():
  WebRtcEventLogManager* const webrtc_event_log_manager =
      WebRtcEventLogManager::CreateSingletonInstance();
  content::WebRtcEventLogger::Set(webrtc_event_log_manager);

As such running out\Release\unit_tests --gtest_filter=*Browser* makes ChromeContentBrowserClientWindowTest.IsDataSaverEnabled flake but it's not at fault. The problem is that previous tests left an instance behind which points to a |task_runner_| belonging to a deleted TaskScheduler.

("out\Release\unit_tests --gtest_filter=*Browser*  --test-launcher-jobs=48" also works to repro faster)

https://chromium-review.googlesource.com/c/chromium/src/+/857515 made it such that WebRtcEventLogManager was a raw pointer managed manually rather than a LazyInstance::Leaky.

The CL documents that this is to address problems where "its task runner was not destroyed
between tests". Unfortunately the CL doesn't quite solve the issue either as if a caller doesn't do the manual invocation, WebRtcEventLogManager can still outlive its |task_runner_| (i.e. TaskScheduler == TestBrowserThreadBundle in this case).

To solve this there are a few options : 

1) Make the singleton a LazyInstance::DestructorAtExit (each unit tests forces recycling of DestructorAtExit LazyInstances).

2) Use a LazySequenceTaskRunner in webrtc_event_log_manager.cc instead of a member variable, this will make sure the TaskRunner instance is recycled between tests

I like (1) because it nicely scopes the instance to each environment (each test) without requiring additional code in BrowserProcessImpl but I know dcheng@ may not like it per wanting to get rid of non-Leaky LazyInstances.

I'm not a fan of (2) as it is merely working around the problem of the instance sticking around to the next tests



A few stack traces caused by this:

[ RUN      ] ChromeContentBrowserClientWindowTest.IsDataSaverEnabled
[20724:17268:0319/113349.717:611032937:FATAL:lock.cc(32)] Check failed: owning_thread_ref_.is_null().
Backtrace:
        base::debug::StackTrace::StackTrace [0x66AEF970+32] (D:\src\chrome\src\base\debug\stack_trace_win.cc:286)
        base::debug::StackTrace::StackTrace [0x66AEF08D+13] (D:\src\chrome\src\base\debug\stack_trace.cc:199)
        logging::LogMessage::~LogMessage [0x66B16FD4+84] (D:\src\chrome\src\base\logging.cc:595)
        base::Lock::CheckUnheldAndMark [0x66B8396E+78] (D:\src\chrome\src\base\synchronization\lock.cc:33)
        base::internal::SchedulerLockImpl::Acquire [0x66B8DB0E+46] (D:\src\chrome\src\base\task_scheduler\scheduler_lock_impl.cc:138)
        base::internal::TaskTracker::WillScheduleSequence [0x66B9A1CA+218] (D:\src\chrome\src\base\task_scheduler\task_tracker.cc:326)
        base::internal::SchedulerWorkerPool::PostTaskWithSequenceNow [0x66B926ED+317] (D:\src\chrome\src\base\task_scheduler\scheduler_worker_pool.cc:211)
        base::internal::SchedulerWorkerPool::PostTaskWithSequence [0x66B92564+372] (D:\src\chrome\src\base\task_scheduler\scheduler_worker_pool.cc:152)
        base::internal::SchedulerSequencedTaskRunner::PostDelayedTask [0x66B92E47+199] (D:\src\chrome\src\base\task_scheduler\scheduler_worker_pool.cc:105)
        base::TaskRunner::PostTask [0x66B89B10+64] (D:\src\chrome\src\base\task_runner.cc:44)
        WebRtcEventLogManager::DisableForBrowserContext [0x0381C1F3+389] (D:\src\chrome\src\chrome\browser\media\webrtc\webrtc_event_log_manager.cc:141)
        content::BrowserContext::~BrowserContext [0x64DCE4C6+240] (D:\src\chrome\src\content\browser\browser_context.cc:591)
        Profile::~Profile [0x03663204+148] (D:\src\chrome\src\chrome\browser\profiles\profile.cc:85)
        TestingProfile::~TestingProfile [0x02CE183A+478] (D:\src\chrome\src\chrome\test\base\testing_profile.cc:553)
        TestingProfile::~TestingProfile [0x02CE0F3B+11] (D:\src\chrome\src\chrome\test\base\testing_profile.cc:521)
        ProfileDestroyer::DestroyProfileWhenAppropriate [0x03CE9A46+426] (D:\src\chrome\src\chrome\browser\profiles\profile_destroyer.cc:72)
        ProfileManager::ProfileInfo::~ProfileInfo [0x03676872+20] (D:\src\chrome\src\chrome\browser\profiles\profile_manager.cc:1712)
        std::unique_ptr<ProfileManager::ProfileInfo,std::default_delete<ProfileManager::ProfileInfo> >::~unique_ptr [0x02CE32DB+17] (d:\src\chrome\src\third_party\depot_tools\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\vc\tools\msvc\14.11.25503\include\memory:2203)
        std::_Tree<std::_Tmap_traits<base::FilePath,std::unique_ptr<ProfileManager::ProfileInfo,std::default_delete<ProfileManager::ProfileInfo> >,std::less<base::FilePath>,std::allocator<std::pair<const base::FilePath,std::unique_ptr<ProfileManager::ProfileInfo, [0x02CE32A6+44] (d:\src\chrome\src\third_party\depot_tools\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\vc\tools\msvc\14.11.25503\include\xtree:1997)
        std::_Tree<std::_Tmap_traits<base::FilePath,std::unique_ptr<ProfileManager::ProfileInfo,std::default_delete<ProfileManager::ProfileInfo> >,std::less<base::FilePath>,std::allocator<std::pair<const base::FilePath,std::unique_ptr<ProfileManager::ProfileInfo, [0x02CE2EC4+50] (d:\src\chrome\src\third_party\depot_tools\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\vc\tools\msvc\14.11.25503\include\xtree:1366)
        ProfileManager::~ProfileManager [0x03677539+179] (D:\src\chrome\src\chrome\browser\profiles\profile_manager.cc:388)
        testing::ProfileManager::~ProfileManager [0x02CE2AED+11] (D:\src\chrome\src\chrome\test\base\testing_profile_manager.cc:32)
        TestingBrowserProcess::SetProfileManager [0x02CDDFAF+45] (D:\src\chrome\src\chrome\test\base\testing_browser_process.cc:192)
        TestingProfileManager::~TestingProfileManager [0x02CE18D7+29] (D:\src\chrome\src\chrome\test\base\testing_profile_manager.cc:55)
        std::unique_ptr<TestingProfileManager,std::default_delete<TestingProfileManager> >::reset [0x014EAA54+22] (d:\src\chrome\src\third_party\depot_tools\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\vc\tools\msvc\14.11.25503\include\memory:2238)
        BrowserWithTestWindowTest::TearDown [0x01A57FB1+163] (D:\src\chrome\src\chrome\test\base\browser_with_test_window_test.cc:117)
        testing::Test::Run [0x023A6E7C+242] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:2493)
        testing::TestInfo::Run [0x023A75EE+204] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:2665)
        testing::TestCase::Run [0x023A79D2+238] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:2778)
        testing::internal::UnitTestImpl::RunAllTests [0x023ADC72+632] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:5034)
        testing::UnitTest::Run [0x023AD8FC+156] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:4652)
        base::TestSuite::Run [0x02CECB28+104] (D:\src\chrome\src\base\test\test_suite.cc:284)
        base::OnceCallback<int ()>::Run [0x025AE57F+45] (D:\src\chrome\src\base\callback.h:95)
        base::`anonymous namespace'::LaunchUnitTestsInternal [0x02CEEC7F+274] (D:\src\chrome\src\base\test\launcher\unit_test_launcher.cc:225)
        base::LaunchUnitTests [0x02CEEB44+176] (D:\src\chrome\src\base\test\launcher\unit_test_launcher.cc:576)
        main [0x048A1480+324] (D:\src\chrome\src\chrome\test\base\run_all_unittests.cc:30)
        __scrt_common_main_seh [0x048A9F49+248] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
        BaseThreadInitThunk [0x767162C4+36]
        RtlSubscribeWnfStateChangeNotification [0x77690F79+1081]
        RtlSubscribeWnfStateChangeNotification [0x77690F44+1028]

[166/166] ChromeContentBrowserClientWindowTest.IsDataSaverEnabled (CRASHED)


[ RUN      ] ChromeContentBrowserClientWindowTest.IsDataSaverEnabled
Received fatal exception EXCEPTION_ACCESS_VIOLATION
Backtrace:
        std::vector<base::internal::TaskTracker::PreemptedBackgroundSequence,std::allocator<base::internal::TaskTracker::PreemptedBackgroundSequence> >::emplace_back<scoped_refptr<base::internal::Sequence>,base::TimeTicks,base::internal::CanScheduleSequenceObserv [0x66D4BB73+51] (d:\src\chrome\src\third_party\depot_tools\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\vc\tools\msvc\14.11.25503\include\vector:940)
        std::priority_queue<base::internal::TaskTracker::PreemptedBackgroundSequence,std::vector<base::internal::TaskTracker::PreemptedBackgroundSequence,std::allocator<base::internal::TaskTracker::PreemptedBackgroundSequence> >,std::greater<base::internal::TaskT [0x66D4A27C+44] (d:\src\chrome\src\third_party\depot_tools\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\vc\tools\msvc\14.11.25503\include\queue:337)
        base::internal::TaskTracker::WillScheduleSequence [0x66D4A208+280] (D:\src\chrome\src\base\task_scheduler\task_tracker.cc:332)
        base::internal::SchedulerWorkerPool::PostTaskWithSequenceNow [0x66D426ED+317] (D:\src\chrome\src\base\task_scheduler\scheduler_worker_pool.cc:211)
        base::internal::SchedulerWorkerPool::PostTaskWithSequence [0x66D42564+372] (D:\src\chrome\src\base\task_scheduler\scheduler_worker_pool.cc:152)
        base::internal::SchedulerSequencedTaskRunner::PostDelayedTask [0x66D42E47+199] (D:\src\chrome\src\base\task_scheduler\scheduler_worker_pool.cc:105)
        base::TaskRunner::PostTask [0x66D39B10+64] (D:\src\chrome\src\base\task_runner.cc:44)
        WebRtcEventLogManager::EnableForBrowserContext [0x0381BF1C+442] (D:\src\chrome\src\chrome\browser\media\webrtc\webrtc_event_log_manager.cc:126)
        content::BrowserContext::Initialize [0x64F7DFE5+1251] (D:\src\chrome\src\content\browser\browser_context.cc:534)
        TestingProfile::Init [0x02CDEC91+139] (D:\src\chrome\src\chrome\test\base\testing_profile.cc:409)
        TestingProfile::TestingProfile [0x02CDF436+384] (D:\src\chrome\src\chrome\test\base\testing_profile.cc:361)
        TestingProfile::Builder::Build [0x02CE0F12+224] (D:\src\chrome\src\chrome\test\base\testing_profile.cc:1048)
        TestingProfileManager::CreateTestingProfile [0x02CE1C39+279] (D:\src\chrome\src\chrome\test\base\testing_profile_manager.cc:99)
        BrowserWithTestWindowTest::CreateProfile [0x01A58298+138] (D:\src\chrome\src\chrome\test\base\browser_with_test_window_test.cc:180)
        BrowserWithTestWindowTest::SetUp [0x01A57E26+230] (D:\src\chrome\src\chrome\test\base\browser_with_test_window_test.cc:83)
        testing::Test::Run [0x023A6DF3+105] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:2481)
        testing::TestInfo::Run [0x023A75EE+204] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:2665)
        testing::TestCase::Run [0x023A79D2+238] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:2778)
        testing::internal::UnitTestImpl::RunAllTests [0x023ADC72+632] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:5034)
        testing::UnitTest::Run [0x023AD8FC+156] (D:\src\chrome\src\third_party\googletest\src\googletest\src\gtest.cc:4652)
        base::TestSuite::Run [0x02CECB28+104] (D:\src\chrome\src\base\test\test_suite.cc:284)
        base::OnceCallback<int ()>::Run [0x025AE57F+45] (D:\src\chrome\src\base\callback.h:95)
        base::`anonymous namespace'::LaunchUnitTestsInternal [0x02CEEC7F+274] (D:\src\chrome\src\base\test\launcher\unit_test_launcher.cc:225)
        base::LaunchUnitTests [0x02CEEB44+176] (D:\src\chrome\src\base\test\launcher\unit_test_launcher.cc:576)
        main [0x048A1480+324] (D:\src\chrome\src\chrome\test\base\run_all_unittests.cc:30)
        __scrt_common_main_seh [0x048A9F49+248] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
        BaseThreadInitThunk [0x767162C4+36]
        RtlSubscribeWnfStateChangeNotification [0x77690F79+1081]
        RtlSubscribeWnfStateChangeNotification [0x77690F44+1028]
[166/166] ChromeContentBrowserClientWindowTest.IsDataSaverEnabled (CRASHED)
 
Thanks you for bringing this up. :-)

I see you've dug through the code. For people who might read this a bit more quickly, let me just add a quick, explicit recap of the history after https://chromium-review.googlesource.com/c/chromium/src/+/857515:
1. WebRtcEventLogManager no longer destroyed on exit; now leaked again. 
2. WebRtcEventLogManager instantiation moved from BrowserMainLoop (content/browser/browser_main_loop.cc) to BrowserProcessImpl (chrome/browser/browser_process_impl.cc).

I don't see anything named "LazySequenceTaskRunner" in the codebase. Is there something with a similar name, or should we extrapolate from the name the new class that would be introduced, given that approach?
Found LazySequencedTaskRunner.
(More after I've had some time to think.)
Project Member

Comment 3 by bugdroid1@chromium.org, Mar 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7264d6a81f319b570fc24f7c2d20977da41d4ef3

commit 7264d6a81f319b570fc24f7c2d20977da41d4ef3
Author: Elad Alon <eladalon@chromium.org>
Date: Wed Mar 21 15:06:26 2018

Destroy WebRtcEventLogManager from ~BrowserProcessImpl()

Destroy WebRtcEventLogManager rather than leak it, so as to deflake
unit tests.

Bug:  823318 
Change-Id: Ibe255217eafe13569e1d86091dc71929adbaf646
Reviewed-on: https://chromium-review.googlesource.com/968874
Commit-Queue: Elad Alon <eladalon@chromium.org>
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: Tommi <tommi@chromium.org>
Reviewed-by: Gabriel Charette <gab@chromium.org>
Cr-Commit-Position: refs/heads/master@{#544704}
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/chrome/browser/browser_process_impl.cc
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/chrome/browser/browser_process_impl.h
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/chrome/browser/media/webrtc/webrtc_event_log_manager.cc
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/chrome/browser/media/webrtc/webrtc_event_log_manager.h
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/chrome/browser/media/webrtc/webrtc_event_log_manager_unittest.cc
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/chrome/browser/media/webrtc/webrtc_event_log_uploader.cc
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/content/public/browser/webrtc_event_logger.cc
[modify] https://crrev.com/7264d6a81f319b570fc24f7c2d20977da41d4ef3/content/public/browser/webrtc_event_logger.h

Status: Fixed (was: Assigned)

Sign in to add a comment