In-process NetworkService causes many unit_tests to fail when run in a batch |
||||||
Issue descriptionVarious unit_tests will fail on their first run, because they try to create a platform-specific NetworkChangeNotifier singleton, which some earlier test has already created, and leaked, e.g: [ RUN ] BackgroundSyncPermissionContextTest.TestSecureRequestingUrl [23782:23782:0921/182138.238032:4001550231500:FATAL:network_change_notifier.cc(672)] Check failed: !g_network_change_notifier. #0 0x7f7d57e1415d base::debug::StackTrace::StackTrace() #1 0x7f7d57b1b9ac base::debug::StackTrace::StackTrace() #2 0x7f7d57b8b47a logging::LogMessage::~LogMessage() #3 0x7f7d5c0f1f47 net::NetworkChangeNotifier::NetworkChangeNotifier() #4 0x7f7d5c0f1214 net::(anonymous namespace)::MockNetworkChangeNotifier::MockNetworkChangeNotifier() #5 0x7f7d5c0f11d2 net::NetworkChangeNotifier::CreateMock() #6 0x0000076501b0 content::RenderViewHostTestHarness::SetUp() #7 0x000004670a2e _ZN7testing8internal12InvokeHelperIvNSt3__15tupleIJEEEE12InvokeMethodIN10extensions27ExtensionDownloaderDelegateEMS8_FvvEEEvPT_T0_RKS4_ #8 0x00000560c392 testing::internal::HandleExceptionsInMethodIfSupported<>() #9 0x0000055ed434 testing::Test::Run() #10 0x0000055ede95 testing::TestInfo::Run() #11 0x0000055ee94f testing::TestCase::Run() #12 0x000005602148 testing::internal::UnitTestImpl::RunAllTests() #13 0x00000561578e testing::internal::HandleSehExceptionsInMethodIfSupported<>() #14 0x00000560da32 testing::internal::HandleExceptionsInMethodIfSupported<>() #15 0x000005601da7 testing::UnitTest::Run() #16 0x0000074fe8b1 RUN_ALL_TESTS() #17 0x0000074fab0b base::TestSuite::Run() #18 0x000007660520 content::UnitTestTestSuite::Run() #19 0x000002416c6d _ZN4base8internal13FunctorTraitsIMNS_7RunLoopEFvvEvE6InvokeIS4_PS2_JEEEvT_OT0_DpOT1_ #20 0x000002416be4 _ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIMNS_7RunLoopEFvvEJPS4_EEEvOT_DpOT0_ #21 0x000002416b95 _ZN4base8internal7InvokerINS0_9BindStateIMNS_7RunLoopEFvvEJNS0_17UnretainedWrapperIS3_EEEEEFvvEE7RunImplIS5_NSt3__15tupleIJS7_EEEJLm0EEEEvOT_OT0_NSC_16integer_sequenceImJXspT1_EEEE #22 0x0000029eb29c _ZN4base8internal7InvokerINS0_9BindStateIM17ThreadWatcherTestFvvEJNS0_17UnretainedWrapperIS3_EEEEEFvvEE3RunEPNS0_13BindStateBaseE #23 0x000005d2746e _ZNO4base12OnceCallbackIFivEE3RunEv #24 0x000007503d85 base::(anonymous namespace)::LaunchUnitTestsInternal() #25 0x000007503bd5 base::LaunchUnitTests() #26 0x0000074dbed1 main #27 0x7f7d305192b1 __libc_start_main #28 0x00000240102a _start The leaked instance of NetworkChangeNotifier appears to be created by the in-process NetworkService, which is itself a leaked singleton. Firstly, we should be tearing-down the NetworkServiceImpl after each test, to ensure proper isolation. Secondly, we remove the limitation of there being at-most-one NetworkChangeNotifier, and either allow multiple to be created, or have a leaked singleton. Note that we currently retry 40-50 tests in this suite on every run, and each retry takes ~3s to execute, i.e. these and other issues requiring retries waste ~3 minutes of every run of unit_tests.
,
Sep 25
,
Oct 25
,
Oct 25
,
Oct 26
> Secondly, we remove the limitation of there being at-most-one NetworkChangeNotifier, and either allow multiple to be created, or have a leaked singleton. I imagine this problem will go away when Issue 821009 is completed.
,
Oct 31
Any update on this? I'm hitting the DCHECK(!g_network_change_notifier) in almost all my local test runs, and as there's no automatic retry when using "--gtest_filter=Foo*" I need to scroll through the logs to see if it's a real issue or if it's this flakiness.
,
Oct 31
Can we get this resolved, or the tests disabled, please?
,
Oct 31
I don't think that there's a single test that we can disable here, the problem is that some random tests might indirectly instantiate g_network_change_notifier (e.g. unit_tests:TabDesktopMediaListTest.*) and leak it. This will break the next test that try to instantiate g_network_change_notifier (tests rarely create this explicitly, it's always a side effect of doing something else). We could patch these tests to make sure that they don't leak g_network_change_notifier , but this doesn't really scale. I've made an attempt at this for the TabDesktopMediaListTest here: https://chromium-review.googlesource.com/c/chromium/src/+/1290776 but I gave up because there was too many places where this has to be done. One simple way to detect the tests that leak this is to add some 'LOG(ERROR) << "Creating g_network_change_notifier";' / 'LOG(ERROR) << "Destroying g_network_change_notifier";' to network_change_notifier.cc and process the logs to see which tests leak it...
,
Oct 31
When I investigated this originally, the issue seemed to be that the in-process NetworkService was the one "leaking" the NetworkChangeNotifier, so I'd basically just disable tests that create an in-process NetworkService.
,
Oct 31
That's probably all tests that use TestingProfile...I think? And probably a bunch of others. The TestingProfile creates a StoragePartition by default, which I think will always cause the NetworkService to be instantiated, though I could be wrong.
,
Oct 31
Sorry for the delay on this, I'm working on it now.
,
Oct 31
Thanks! re c10: Yep, tests usually don't create the NetworkService explicitly (see the stack trace in https://chromium-review.googlesource.com/c/chromium/src/+/1290776), there's already a workaround in place for all the tests based on RenderViewTestHarness : https://cs.chromium.org/chromium/src/content/public/test/test_renderer_host.cc?l=267&rcl=c2d8da6cf7ddfd085ac9c92ff86401b1f7651bc6
,
Nov 2
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/de0b6f6c532914dd6e9dc331a004470a36b3de58 commit de0b6f6c532914dd6e9dc331a004470a36b3de58 Author: Robbie McElrath <rmcelrath@chromium.org> Date: Fri Nov 02 01:18:26 2018 Disable any existing NetworkChangeNotifier before all unit tests. This is a quick fix for crbug.com/888215 . crbug.com/901092 will track the longer term fix, which will likely require quite a bit more work. Bug: 888215 Change-Id: I071ea0a1508e486211789073c43050f4cd56accb Reviewed-on: https://chromium-review.googlesource.com/c/1313414 Commit-Queue: Robbie McElrath <rmcelrath@chromium.org> Reviewed-by: Reilly Grant <reillyg@chromium.org> Reviewed-by: Scott Violet <sky@chromium.org> Cr-Commit-Position: refs/heads/master@{#604797} [modify] https://crrev.com/de0b6f6c532914dd6e9dc331a004470a36b3de58/content/public/test/content_test_suite_base.cc
,
Nov 2
I landed a quick fix for this, and filed crbug.com/901092 about a proper fix, which will take a lot longer. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by dougt@chromium.org
, Sep 24Owner: rmcelrath@chromium.org
Status: Assigned (was: Untriaged)