New issue
Advanced search Search tips

Issue 888215 link

Starred by 9 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Chrome , Mac , Fuchsia
Pri: 1
Type: Bug



Sign in to add a comment

In-process NetworkService causes many unit_tests to fail when run in a batch

Project Member Reported by w...@chromium.org, Sep 22

Issue description

Various unit_tests will fail on their first run, because they try to create a platform-specific NetworkChangeNotifier singleton, which some earlier test has already created, and leaked, e.g:

[ RUN      ] BackgroundSyncPermissionContextTest.TestSecureRequestingUrl
[23782:23782:0921/182138.238032:4001550231500:FATAL:network_change_notifier.cc(672)] Check failed: !g_network_change_notifier. 
#0 0x7f7d57e1415d base::debug::StackTrace::StackTrace()
#1 0x7f7d57b1b9ac base::debug::StackTrace::StackTrace()
#2 0x7f7d57b8b47a logging::LogMessage::~LogMessage()
#3 0x7f7d5c0f1f47 net::NetworkChangeNotifier::NetworkChangeNotifier()
#4 0x7f7d5c0f1214 net::(anonymous namespace)::MockNetworkChangeNotifier::MockNetworkChangeNotifier()
#5 0x7f7d5c0f11d2 net::NetworkChangeNotifier::CreateMock()
#6 0x0000076501b0 content::RenderViewHostTestHarness::SetUp()
#7 0x000004670a2e _ZN7testing8internal12InvokeHelperIvNSt3__15tupleIJEEEE12InvokeMethodIN10extensions27ExtensionDownloaderDelegateEMS8_FvvEEEvPT_T0_RKS4_
#8 0x00000560c392 testing::internal::HandleExceptionsInMethodIfSupported<>()
#9 0x0000055ed434 testing::Test::Run()
#10 0x0000055ede95 testing::TestInfo::Run()
#11 0x0000055ee94f testing::TestCase::Run()
#12 0x000005602148 testing::internal::UnitTestImpl::RunAllTests()
#13 0x00000561578e testing::internal::HandleSehExceptionsInMethodIfSupported<>()
#14 0x00000560da32 testing::internal::HandleExceptionsInMethodIfSupported<>()
#15 0x000005601da7 testing::UnitTest::Run()
#16 0x0000074fe8b1 RUN_ALL_TESTS()
#17 0x0000074fab0b base::TestSuite::Run()
#18 0x000007660520 content::UnitTestTestSuite::Run()
#19 0x000002416c6d _ZN4base8internal13FunctorTraitsIMNS_7RunLoopEFvvEvE6InvokeIS4_PS2_JEEEvT_OT0_DpOT1_
#20 0x000002416be4 _ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIMNS_7RunLoopEFvvEJPS4_EEEvOT_DpOT0_
#21 0x000002416b95 _ZN4base8internal7InvokerINS0_9BindStateIMNS_7RunLoopEFvvEJNS0_17UnretainedWrapperIS3_EEEEEFvvEE7RunImplIS5_NSt3__15tupleIJS7_EEEJLm0EEEEvOT_OT0_NSC_16integer_sequenceImJXspT1_EEEE
#22 0x0000029eb29c _ZN4base8internal7InvokerINS0_9BindStateIM17ThreadWatcherTestFvvEJNS0_17UnretainedWrapperIS3_EEEEEFvvEE3RunEPNS0_13BindStateBaseE
#23 0x000005d2746e _ZNO4base12OnceCallbackIFivEE3RunEv
#24 0x000007503d85 base::(anonymous namespace)::LaunchUnitTestsInternal()
#25 0x000007503bd5 base::LaunchUnitTests()
#26 0x0000074dbed1 main
#27 0x7f7d305192b1 __libc_start_main
#28 0x00000240102a _start

The leaked instance of NetworkChangeNotifier appears to be created by the in-process NetworkService, which is itself a leaked singleton.

Firstly, we should be tearing-down the NetworkServiceImpl after each test, to ensure proper isolation.

Secondly, we remove the limitation of there being at-most-one NetworkChangeNotifier, and either allow multiple to be created, or have a leaked singleton.

Note that we currently retry 40-50 tests in this suite on every run, and each retry takes ~3s to execute, i.e. these and other issues requiring retries waste ~3 minutes of every run of unit_tests.
 
Labels: Proj-Servicification
Owner: rmcelrath@chromium.org
Status: Assigned (was: Untriaged)
Labels: Hotlist-KnownIssue
Cc: sebmarchand@chromium.org
Cc: mmenke@chromium.org
 Issue 867414  has been merged into this issue.
> Secondly, we remove the limitation of there being at-most-one NetworkChangeNotifier, and either allow multiple to be created, or have a leaked singleton.

I imagine this problem will go away when Issue 821009 is completed.
Any update on this? I'm hitting the DCHECK(!g_network_change_notifier) in almost all my local test runs, and as there's no automatic retry when using "--gtest_filter=Foo*" I need to scroll through the logs to see if it's a real issue or if it's this flakiness.
Labels: -M-71 M-72
Can we get this resolved, or the tests disabled, please?
I don't think that there's a single test that we can disable here, the problem is that some random tests might indirectly instantiate g_network_change_notifier (e.g. unit_tests:TabDesktopMediaListTest.*) and leak it. This will break the next test that try to instantiate g_network_change_notifier (tests rarely create this explicitly, it's always a side effect of doing something else).

We could patch these tests to make sure that they don't leak g_network_change_notifier , but this doesn't really scale. I've made an attempt at this for the TabDesktopMediaListTest here: https://chromium-review.googlesource.com/c/chromium/src/+/1290776 but I gave up because there was too many places where this has to be done.

One simple way to detect the tests that leak this is to add some 'LOG(ERROR) << "Creating g_network_change_notifier";' / 'LOG(ERROR) << "Destroying g_network_change_notifier";' to network_change_notifier.cc and process the logs to see which tests leak it...
When I investigated this originally, the issue seemed to be that the
in-process NetworkService was the one "leaking" the NetworkChangeNotifier,
so I'd basically just disable tests that create an in-process
NetworkService.
That's probably all tests that use TestingProfile...I think?  And probably a bunch of others.  The TestingProfile creates a StoragePartition by default, which I think will always cause the NetworkService to be instantiated, though I could be wrong.
Status: Started (was: Assigned)
Sorry for the delay on this, I'm working on it now.
Thanks!

re c10: Yep, tests usually don't create the NetworkService explicitly (see the stack trace in https://chromium-review.googlesource.com/c/chromium/src/+/1290776), there's already a workaround in place for all the tests based on RenderViewTestHarness : https://cs.chromium.org/chromium/src/content/public/test/test_renderer_host.cc?l=267&rcl=c2d8da6cf7ddfd085ac9c92ff86401b1f7651bc6
Project Member

Comment 13 by bugdroid1@chromium.org, Nov 2

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/de0b6f6c532914dd6e9dc331a004470a36b3de58

commit de0b6f6c532914dd6e9dc331a004470a36b3de58
Author: Robbie McElrath <rmcelrath@chromium.org>
Date: Fri Nov 02 01:18:26 2018

Disable any existing NetworkChangeNotifier before all unit tests.

This is a quick fix for  crbug.com/888215 . crbug.com/901092 will track
the longer term fix, which will likely require quite a bit more work.

Bug:  888215 
Change-Id: I071ea0a1508e486211789073c43050f4cd56accb
Reviewed-on: https://chromium-review.googlesource.com/c/1313414
Commit-Queue: Robbie McElrath <rmcelrath@chromium.org>
Reviewed-by: Reilly Grant <reillyg@chromium.org>
Reviewed-by: Scott Violet <sky@chromium.org>
Cr-Commit-Position: refs/heads/master@{#604797}
[modify] https://crrev.com/de0b6f6c532914dd6e9dc331a004470a36b3de58/content/public/test/content_test_suite_base.cc

Status: Fixed (was: Started)
I landed a quick fix for this, and filed crbug.com/901092 about a proper fix, which will take a lot longer.

Sign in to add a comment