New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 789442 link

Starred by 3 users

Issue metadata

Status: Fixed
Merged: issue 517420
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

DCHECK hit in DnsConfigServiceWin in some tests

Project Member Reported by grt@chromium.org, Nov 29 2017

Issue description

TaskScheduler::GetInstance is returning an empty ref (see screenshot). Perhaps it's a shutdown race and the DnsConfigServiceThread isn't being torn down before the tests's ScopedTaskEnvironment? Just a guess. I don't have a repro, only screenshots from the bots like the one attached. I also don't know which test binary is causing it -- the "Fatal error" message box is left over from some previous test run on the bot when interactive_ui_tests is run (which dumps the snapshot).

This is presumably causing some other test to fail. It's interesting that said other test is even generating this dialog. It means that the test is calling logging::SetShowErrorDialogs(true), which should not be happening on the bots at all, as they have CHROME_HEADLESS=1 in their environment (which suppresses them).

Assigning to a random member of net/OWNERS for triage and assignment.
 
ss_20171128173029_063.png
74.7 KB View Download

Comment 1 by mmenke@chromium.org, Nov 29 2017

Cc: cbentzel@chromium.org
Owner: mge...@chromium.org
[+mgersh]:  Mind investigating?

Comment 2 by mge...@chromium.org, Nov 29 2017

Do you have links to bot runs where this happened?

The DnsConfigService conversion to TaskScheduler was only a few months ago, I wouldn't be surprised to still find some bugs. It looks like it's failing in initialization, not shutdown.

Comment 3 by grt@chromium.org, Nov 29 2017

Cc: mar...@chromium.org
No. I only have these screenshots from tests that run *after* the failing test on the same bot. I don't know what it was that ran before interactive_ui_tests on this particular swarming bot. I've seen this same error message on many runs.

Maybe someone from infra could help figure out what is popping the dialog. +maruel, who added some logging recently that might help.

Comment 4 by grt@chromium.org, Dec 4 2017

Labels: -Pri-2 Pri-1
Now that CHROME_HEADLESS=1 is properly being set (see issue 790985), we're seeing this crash in failing browser_tests; see, for example, https://chromium-swarm.appspot.com/task?id=3a2d610a75456610&refresh=10&show_raw=1:

[ RUN      ] ServiceProcessControlBrowserTest.DieOnDisconnect
...
[4372:6280:1201/161614.652:FATAL:post_task.cc(72)] Check failed: TaskScheduler::GetInstance(). Ref. Prerequisite section of post_task.h.

Hint: if this is in a unit test, you're likely merely missing a base::test::ScopedTaskEnvironment member in your fixture.

Backtrace:
	base::debug::StackTrace::StackTrace [0x02BBD2B0+32]
	base::debug::StackTrace::StackTrace [0x02B7D2ED+13]
	logging::LogMessage::~LogMessage [0x02B2801E+78]
	base::PostDelayedTaskWithTraits [0x02B52F01+97]
	base::PostTaskWithTraits [0x02B53081+65]
	base::CreateCOMSTATaskRunnerWithTraits [0x02B53455+277]
	base::internal::PostTaskAndReplyImpl::PostTaskAndReply [0x02BC6D51+625]
	base::PostTaskWithTraitsAndReply [0x02B53011+97]
	net::SerialWorker::WorkNow [0x02E87709+329]
	net::internal::DnsConfigServiceWin::ReadNow [0x02DF5A17+23]
	net::DnsConfigService::WatchConfig [0x02D771CB+217]
	net::NetworkChangeNotifierWin::DnsConfigServiceThread::Init [0x02D76949+169]
	base::Thread::ThreadMain [0x02B6EBD4+548]
	base::PlatformThread::GetCurrentThreadPriority [0x02B4B1D3+467]
	BaseThreadInitThunk [0x75D3337A+18]
	RtlInitializeExceptionChain [0x77C692B2+99]
	RtlInitializeExceptionChain [0x77C69285+54]

Received fatal exception EXCEPTION_BREAKPOINT
Backtrace:
	base::debug::BreakDebugger [0x02BC656C+12]
	?Run@?$Invoker@U?$BindState@P6AXPBDHV?$BasicStringPiece@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@base@@1@Z$$V@internal@base@@$$A6AXPBDHV?$BasicStringPiece@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@3@1@Z@internal@ [0x0353081F+31]
	logging::LogMessage::~LogMessage [0x02B283DE+1038]
	base::PostDelayedTaskWithTraits [0x02B52F01+97]
	base::PostTaskWithTraits [0x02B53081+65]
	base::CreateCOMSTATaskRunnerWithTraits [0x02B53455+277]
	base::internal::PostTaskAndReplyImpl::PostTaskAndReply [0x02BC6D51+625]
	base::PostTaskWithTraitsAndReply [0x02B53011+97]
	net::SerialWorker::WorkNow [0x02E87709+329]
	net::internal::DnsConfigServiceWin::ReadNow [0x02DF5A17+23]
	net::DnsConfigService::WatchConfig [0x02D771CB+217]
	net::NetworkChangeNotifierWin::DnsConfigServiceThread::Init [0x02D76949+169]
	base::Thread::ThreadMain [0x02B6EBD4+548]
	base::PlatformThread::GetCurrentThreadPriority [0x02B4B1D3+467]
	BaseThreadInitThunk [0x75D3337A+18]
	RtlInitializeExceptionChain [0x77C692B2+99]
	RtlInitializeExceptionChain [0x77C69285+54]
../../chrome/browser/service_process/service_process_control_browsertest.cc(126): error: Failed
../../chrome/browser/service_process/service_process_control_browsertest.cc(314): error: Value of: ServiceProcessControl::GetInstance()->IsConnected()
  Actual: false
Expected: true
[264:7424:1201/161616.580:INFO:chrome_cryptauth_service.cc(222)] Profile is not authenticated yet; waiting before starting CryptAuth managers.
[264:7424:1201/161616.890:WARNING:url_request_context_getter.cc(43)] URLRequestContextGetter leaking due to no owning thread.
[  FAILED  ] ServiceProcessControlBrowserTest.DieOnDisconnect, where TypeParam =  and GetParam() =  (2635 ms)

Without CHROME_HEADLESS=1, the dialog in the initial report is shown, which was likely leading to test timeouts. Now we're at least seeing the error where it happens. This is leading to flaky browser_tests, so please fix as soon as possible. I have bumped this to P1 since it should now be quite actionable.
Cc: mge...@chromium.org
Components: -Internals>Network>DNS Services>CloudPrint
Mergedinto: 517420
Owner: ----
Status: Duplicate (was: Assigned)
It sounds like this is a problem with initialization order in a specific set of tests, which already has a bug filed. The DnsConfigService initialization just happens to be something that needs the TaskScheduler early on. I don't think I'm the right person to debug this, but let me know if you see it anywhere other than ServiceProcessControl tests.

Comment 6 by grt@chromium.org, Dec 4 2017

Owner: fdoray@chromium.org
Status: Assigned (was: Duplicate)
The crash stacks for the initial reports in issue 517420 have nothing to do with those in comment 4 here. I don't think it's helpful to lump unrelated problems into the same bug.

I just poked around in codesearch for a few minutes, and it seems pretty clear that the problem is in chrome/service/service_process.cc:

bool ServiceProcess::Initialize(base::MessageLoopForUI* message_loop,
                                const base::CommandLine& command_line,
                                ServiceProcessState* state) {
...
  network_change_notifier_.reset(net::NetworkChangeNotifier::Create());
...
  base::TaskScheduler::Create("CloudPrintServiceProcess");
...
}

those two lines are out-of-order. Francois, could you take this?

Comment 7 by grt@chromium.org, Dec 4 2017

Oh, and if I'm reading it right, this is a product problem, not a test problem. That is, it's not initialization order in a specific set of tests,
I can believe it's unrelated to the Mac flakiness, I didn't look. But it's definitely the cause of the Windows flakiness in comments 18-20 on the other bug, I did check those links, and I don't think it's helpful to not inform the people working on a bug that you've decided to separate it out. I'll go comment there.

Comment 9 by grt@chromium.org, Dec 5 2017

Good point. Thanks for updating the other bug. Cheers.

Comment 10 by grt@chromium.org, Dec 5 2017

Upon re-reading comment 6, I think I came across as more gruff than is appropriate. My apologies.
Status: Started (was: Assigned)
https://chromium-review.googlesource.com/c/chromium/src/+/810925
Cc: thestig@chromium.org vitalyb...@chromium.org rbpotter@chromium.org
 Issue 791791  has been merged into this issue.
Project Member

Comment 13 by bugdroid1@chromium.org, Dec 7 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d7a91284b1ea021daaeeea39c8f6a242782f670

commit 1d7a91284b1ea021daaeeea39c8f6a242782f670
Author: Francois Doray <fdoray@chromium.org>
Date: Thu Dec 07 00:01:50 2017

Initialize NetworkChangeNotifier after TaskScheduler in service process.

The NetworkChangeNotifier must be created after TaskScheduler because it
posts tasks to it.

Bug:  789442 
Change-Id: Ie6066064a5f840666160bac76d95ad16d3363708
Reviewed-on: https://chromium-review.googlesource.com/810925
Reviewed-by: Vitaly Buka <vitalybuka@chromium.org>
Commit-Queue: François Doray <fdoray@chromium.org>
Cr-Commit-Position: refs/heads/master@{#522262}
[modify] https://crrev.com/1d7a91284b1ea021daaeeea39c8f6a242782f670/chrome/service/service_process.cc

Comment 14 by grt@chromium.org, Dec 8 2017

Thanks for the fix! Would you also please re-enable the tests that were disabled for this bug? https://bugs.chromium.org/p/chromium/issues/detail?id=517420#c18 and https://bugs.chromium.org/p/chromium/issues/detail?id=517420#c20 might both be okay now.
CL to re-enabled the tests waiting for review https://chromium-review.googlesource.com/c/chromium/src/+/817574
Project Member

Comment 16 by bugdroid1@chromium.org, Dec 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/29c78ed184f0b7c2a3e71462450030d72ce0bc70

commit 29c78ed184f0b7c2a3e71462450030d72ce0bc70
Author: Francois Doray <fdoray@chromium.org>
Date: Fri Dec 08 23:23:38 2017

Re-enable ServiceProcessControlBrowserTest on Windows.

This reverts commits
d70e91da314d7d5b28bcfe97007933dc0aec9dfd and
448f299e4fb9825df68bad77b8f2352b288f5a63.

Issue on Windows was fixed by
https://chromium-review.googlesource.com/810925.

Issue on Mac seems to have a different root cause.
We should consider re-enabling the tests, which were
disabled in 2015, and see if the issue is still there.

Bug:  789442 , 517420
Change-Id: Id4ce64b5ec1ffa6d97c25b7967a92ac0ad293979
Reviewed-on: https://chromium-review.googlesource.com/817574
Reviewed-by: Vitaly Buka <vitalybuka@chromium.org>
Commit-Queue: Vitaly Buka <vitalybuka@chromium.org>
Cr-Commit-Position: refs/heads/master@{#522915}
[modify] https://crrev.com/29c78ed184f0b7c2a3e71462450030d72ce0bc70/chrome/browser/service_process/service_process_control_browsertest.cc

Status: Fixed (was: Started)

Comment 18 by grt@chromium.org, Dec 13 2017

Thanks!

Sign in to add a comment