New issue
Advanced search Search tips

Issue 833401 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug



Sign in to add a comment

cronet_tests failing on ToTWin64 with crash in at_exit

Project Member Reported by thakis@chromium.org, Apr 16 2018

Issue description

Probably started when wez made cronet_tests run on more bots in https://chromium-review.googlesource.com/c/chromium/src/+/999932

List of failing tests:

BufferTest.TestInitWithAlloc
BufferTest.TestInitWithHugeAllocFails
ExecutorsTest.TestTestExecutor
UrlRequestTest.CancelRequest
UrlRequestTest.SimpleRequest
UrlRequestTest.InitChecks
UrlRequestTest.PerfTest
EngineTest.InvalidPkpParams
UrlRequestTest.FailedRequestHostNotFound
EngineTest.InitDifferentEngines
ExecutorsTest.TestCustom
EngineTest.ValidPkpParams
EngineTest.StartResults
EngineTest.StartCronetEngine
UrlRequestTest.MultiRedirect
EngineTest.StartNetLogToFile
UrlRequestTest.SimpleGet
EngineTest.CronetEngineDefaultUserAgent
BufferTest.TestInitWithDataAndCallback
BufferTest.TestCronetBufferAsync

All stacks look like this example though:

https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FToTWin64_dll_%2F445%2F%2B%2Frecipes%2Fsteps%2Fcronet_tests%2F0%2Flogs%2FUrlRequestTest.MultiRedirect%2F0

[4968:5328:0406/121118.956:24841864:FATAL:at_exit.cc(42)] Check failed: this == g_top_manager (0000000000343CE0 vs. 000000000034DFB0)
Backtrace:
	base::debug::StackTrace::StackTrace [0x000007FEF09FE734+36] (C:\b\c\b\ToTWin64_dll_\src\base\debug\stack_trace_win.cc:286)
	logging::LogMessage::~LogMessage [0x000007FEF0A3485F+95] (C:\b\c\b\ToTWin64_dll_\src\base\logging.cc:595)
	base::AtExitManager::~AtExitManager [0x000007FEF09D3271+129] (C:\b\c\b\ToTWin64_dll_\src\base\at_exit.cc:44)
	std::unique_ptr<base::AtExitManager,std::default_delete<base::AtExitManager> >::~unique_ptr [0x000000013F1668CA+22] (C:\b\c\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\VC\Tools\MSVC\14.11.25503\include\memory:2203)
	main [0x000000013F0F9AF2+182] (C:\b\c\b\ToTWin64_dll_\src\components\cronet\run_all_unittests.cc:14)
	__scrt_common_main_seh [0x000000013F21DAB1+285] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
	BaseThreadInitThunk [0x0000000076C859CD+13]
	RtlUserThreadStart [0x0000000076DBA561+33]



So the binary crashes after all tests have run. This may be a compiler bug or a code bug, not sure.
 

Comment 1 by mef@chromium.org, Apr 16 2018

Is ToTWin64 doing component builds? If so, it is probably  https://crbug.com/816705 

Comment 2 by mef@chromium.org, Apr 16 2018

Is there a trybot with same configuration? 
I'd like to try https://chromium-review.googlesource.com/c/chromium/src/+/1014167

Comment 3 by thakis@chromium.org, Apr 16 2018

Whoops, it's ToTWin64(dll), and yes, that is doing component builds. (From the error output you can click through to the bot, to a build, to generate build files -> https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FToTWin64_dll_%2F491%2F%2B%2Frecipes%2Fsteps%2Fgenerate_build_files%2F0%2Fstdout -> 'is_component_build = true' to check yourself).


Can we then disable cronet_tests on all bots doing component builds until this is fixed, please?



There isn't a ready-made trybot, but you can add a change to your CL that sets https://cs.chromium.org/chromium/src/build/config/BUILDCONFIG.gn?type=cs&q=is_component_build%5C+%3D&sq=package:chromium&l=169 to true unconditionally and then send a regular release try job. That should then test release component builds.

Comment 4 by mef@chromium.org, Apr 16 2018

Cc: mef@chromium.org
Owner: w...@chromium.org
I'm fine with disabling cronet tests on bots doing component builds as this configuration is not supported and we don't expect it to be supported.

Wez@, do you know how to disable cronet_tests from running on such bots?

Comment 5 by w...@chromium.org, Apr 16 2018

Cc: jbudorick@chromium.org
Owner: mef@chromium.org
Status: Assigned (was: Untriaged)
Re #4: There is no clean way to disable particular tests on all component builders.  You will need to manually add entries to https://cs.chromium.org/chromium/src/testing/buildbot/test_suite_exceptions.pyl?l=1269 for each bot on which you want to disable "cronet_tests" - you'll see we have already done that for Fuchsia/x64/Debug.

Comment 6 by w...@chromium.org, Apr 16 2018

Cc: thakis@chromium.org
Owner: thakis@chromium.org
Note that we already have a hack https://cs.chromium.org/chromium/src/components/cronet/cronet_global_state_stubs.cc?l=28 which disables the creation of the Cronet library AtExitManager, which is the cause of the CHECK stack that thakis@ listed.

IIUC, that means that the ToTWin64 bot must be building a Release/component build, which I thought was a build configuration we explicitly don't support?

(Assigning to thakis@ to clarify)

Comment 7 by thakis@chromium.org, Apr 16 2018

Owner: w...@chromium.org
Lots of devs do release component builds. No main waterfall bots do, but some off waterfall bots do since they need to check we don't break devs.

Comment 8 by w...@chromium.org, Apr 16 2018

Status: Started (was: Assigned)
Re #7: I am behind the times, it seems. OK, I'll send out a CL shortly to disable the cronet_tests on 'ToTWin64(dll)'.

Comment 9 by thakis@chromium.org, Apr 16 2018

Also the other release dll bots on chromium.clang please (32 bit, and lld 32 and 64 bit)

Comment 10 by r...@chromium.org, Apr 16 2018

Would it be cleaner to change the hack in cronet_global_state_stubs.cc to check for component builds rather than debug builds? Then we don't need to have exceptions for tests.

Comment 11 by w...@chromium.org, Apr 16 2018

Re #10: Yes, that is actually the intent of the check, after all; will try that now.
Project Member

Comment 12 by bugdroid1@chromium.org, Apr 18 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5fe14697a2fc922c26818612a3e3f6ba22daaa99

commit 5fe14697a2fc922c26818612a3e3f6ba22daaa99
Author: Wez <wez@chromium.org>
Date: Wed Apr 18 19:45:18 2018

[cronet] Apply AtExitManager workaround to Component builds.

We work-around shared global state in component builds of the Cronet
native library by avoiding creating the AtExitManager in those builds.
The condition for that is fixed here, from being based on Debug build
to depending directly on whether this is a component build.

The tests are also enabled on Fuchsia/Debug, where they now pass.

Bug:  833401 ,  816705 
Cq-Include-Trybots: master.tryserver.chromium.android:android_cronet_tester;master.tryserver.chromium.mac:ios-simulator-cronet
Change-Id: I1401c0dcaa8e893c6cd4bcbf75f8d4d30ccd0116
Reviewed-on: https://chromium-review.googlesource.com/1014642
Commit-Queue: Wez <wez@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Misha Efimov <mef@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551788}
[modify] https://crrev.com/5fe14697a2fc922c26818612a3e3f6ba22daaa99/components/cronet/cronet_global_state_stubs.cc
[modify] https://crrev.com/5fe14697a2fc922c26818612a3e3f6ba22daaa99/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/5fe14697a2fc922c26818612a3e3f6ba22daaa99/testing/buildbot/test_suite_exceptions.pyl

Comment 13 by w...@chromium.org, Apr 18 2018

Status: Fixed (was: Started)

Sign in to add a comment