cronet_tests failing on ToTWin64 with crash in at_exit |
|||||||
Issue descriptionProbably started when wez made cronet_tests run on more bots in https://chromium-review.googlesource.com/c/chromium/src/+/999932 List of failing tests: BufferTest.TestInitWithAlloc BufferTest.TestInitWithHugeAllocFails ExecutorsTest.TestTestExecutor UrlRequestTest.CancelRequest UrlRequestTest.SimpleRequest UrlRequestTest.InitChecks UrlRequestTest.PerfTest EngineTest.InvalidPkpParams UrlRequestTest.FailedRequestHostNotFound EngineTest.InitDifferentEngines ExecutorsTest.TestCustom EngineTest.ValidPkpParams EngineTest.StartResults EngineTest.StartCronetEngine UrlRequestTest.MultiRedirect EngineTest.StartNetLogToFile UrlRequestTest.SimpleGet EngineTest.CronetEngineDefaultUserAgent BufferTest.TestInitWithDataAndCallback BufferTest.TestCronetBufferAsync All stacks look like this example though: https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FToTWin64_dll_%2F445%2F%2B%2Frecipes%2Fsteps%2Fcronet_tests%2F0%2Flogs%2FUrlRequestTest.MultiRedirect%2F0 [4968:5328:0406/121118.956:24841864:FATAL:at_exit.cc(42)] Check failed: this == g_top_manager (0000000000343CE0 vs. 000000000034DFB0) Backtrace: base::debug::StackTrace::StackTrace [0x000007FEF09FE734+36] (C:\b\c\b\ToTWin64_dll_\src\base\debug\stack_trace_win.cc:286) logging::LogMessage::~LogMessage [0x000007FEF0A3485F+95] (C:\b\c\b\ToTWin64_dll_\src\base\logging.cc:595) base::AtExitManager::~AtExitManager [0x000007FEF09D3271+129] (C:\b\c\b\ToTWin64_dll_\src\base\at_exit.cc:44) std::unique_ptr<base::AtExitManager,std::default_delete<base::AtExitManager> >::~unique_ptr [0x000000013F1668CA+22] (C:\b\c\win_toolchain\vs_files\1180cb75833ea365097e279efb2d5d7a42dee4b0\VC\Tools\MSVC\14.11.25503\include\memory:2203) main [0x000000013F0F9AF2+182] (C:\b\c\b\ToTWin64_dll_\src\components\cronet\run_all_unittests.cc:14) __scrt_common_main_seh [0x000000013F21DAB1+285] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283) BaseThreadInitThunk [0x0000000076C859CD+13] RtlUserThreadStart [0x0000000076DBA561+33] So the binary crashes after all tests have run. This may be a compiler bug or a code bug, not sure.
,
Apr 16 2018
Is there a trybot with same configuration? I'd like to try https://chromium-review.googlesource.com/c/chromium/src/+/1014167
,
Apr 16 2018
Whoops, it's ToTWin64(dll), and yes, that is doing component builds. (From the error output you can click through to the bot, to a build, to generate build files -> https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FToTWin64_dll_%2F491%2F%2B%2Frecipes%2Fsteps%2Fgenerate_build_files%2F0%2Fstdout -> 'is_component_build = true' to check yourself). Can we then disable cronet_tests on all bots doing component builds until this is fixed, please? There isn't a ready-made trybot, but you can add a change to your CL that sets https://cs.chromium.org/chromium/src/build/config/BUILDCONFIG.gn?type=cs&q=is_component_build%5C+%3D&sq=package:chromium&l=169 to true unconditionally and then send a regular release try job. That should then test release component builds.
,
Apr 16 2018
I'm fine with disabling cronet tests on bots doing component builds as this configuration is not supported and we don't expect it to be supported. Wez@, do you know how to disable cronet_tests from running on such bots?
,
Apr 16 2018
Re #4: There is no clean way to disable particular tests on all component builders. You will need to manually add entries to https://cs.chromium.org/chromium/src/testing/buildbot/test_suite_exceptions.pyl?l=1269 for each bot on which you want to disable "cronet_tests" - you'll see we have already done that for Fuchsia/x64/Debug.
,
Apr 16 2018
Note that we already have a hack https://cs.chromium.org/chromium/src/components/cronet/cronet_global_state_stubs.cc?l=28 which disables the creation of the Cronet library AtExitManager, which is the cause of the CHECK stack that thakis@ listed. IIUC, that means that the ToTWin64 bot must be building a Release/component build, which I thought was a build configuration we explicitly don't support? (Assigning to thakis@ to clarify)
,
Apr 16 2018
Lots of devs do release component builds. No main waterfall bots do, but some off waterfall bots do since they need to check we don't break devs.
,
Apr 16 2018
Re #7: I am behind the times, it seems. OK, I'll send out a CL shortly to disable the cronet_tests on 'ToTWin64(dll)'.
,
Apr 16 2018
Also the other release dll bots on chromium.clang please (32 bit, and lld 32 and 64 bit)
,
Apr 16 2018
Would it be cleaner to change the hack in cronet_global_state_stubs.cc to check for component builds rather than debug builds? Then we don't need to have exceptions for tests.
,
Apr 16 2018
Re #10: Yes, that is actually the intent of the check, after all; will try that now.
,
Apr 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5fe14697a2fc922c26818612a3e3f6ba22daaa99 commit 5fe14697a2fc922c26818612a3e3f6ba22daaa99 Author: Wez <wez@chromium.org> Date: Wed Apr 18 19:45:18 2018 [cronet] Apply AtExitManager workaround to Component builds. We work-around shared global state in component builds of the Cronet native library by avoiding creating the AtExitManager in those builds. The condition for that is fixed here, from being based on Debug build to depending directly on whether this is a component build. The tests are also enabled on Fuchsia/Debug, where they now pass. Bug: 833401 , 816705 Cq-Include-Trybots: master.tryserver.chromium.android:android_cronet_tester;master.tryserver.chromium.mac:ios-simulator-cronet Change-Id: I1401c0dcaa8e893c6cd4bcbf75f8d4d30ccd0116 Reviewed-on: https://chromium-review.googlesource.com/1014642 Commit-Queue: Wez <wez@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Reid Kleckner <rnk@chromium.org> Reviewed-by: Misha Efimov <mef@chromium.org> Cr-Commit-Position: refs/heads/master@{#551788} [modify] https://crrev.com/5fe14697a2fc922c26818612a3e3f6ba22daaa99/components/cronet/cronet_global_state_stubs.cc [modify] https://crrev.com/5fe14697a2fc922c26818612a3e3f6ba22daaa99/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/5fe14697a2fc922c26818612a3e3f6ba22daaa99/testing/buildbot/test_suite_exceptions.pyl
,
Apr 18 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by mef@chromium.org
, Apr 16 2018