NonThreadSafeDeathTest is Flaky on Mac |
|||
Issue description
Chrome Version: HEAD
OS: Mac OSX
What steps will reproduce the problem?
(1) build base_unittests (Debug or Release with DCHECK)
(2) .../base_unittests --test-launcher-retry-limit=0 -
-gtest_filter=*NonThreadSafeDeathTest*
What is the expected result?
It should pass.
What happens instead?
Death test: { NonThreadSafeClass::MethodOnDifferentThreadImpl(); }
Result: died but not with expected error.
Expected: Check failed
Actual msg:
[ DEATH ] The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
[ DEATH ] Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
[ DEATH ] The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
[ DEATH ] Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
[ DEATH ] Received signal 11 SEGV_MAPERR 000000000110
[ DEATH ] [0x000100e0479c]
[ DEATH ] [0x000100e04691]
[ DEATH ] [0x7fff956b5b3a]
[ DEATH ] [0x000000000000]
[ DEATH ] [0x7fff7fd06b98]
[ DEATH ] [0x7fff7fd0687e]
[ DEATH ] [0x7fff8175b7af]
[ DEATH ] [0x7fff956bf9af]
[ DEATH ] [0x7fff956bf8fb]
[ DEATH ] [0x7fff956bf101]
[ DEATH ] [end of stack trace]
[ DEATH ]
[ FAILED ] NonThreadSafeDeathTest.MethodNotAllowedOnDifferentThreadInDebug (3 ms)
Though this isn't generally causing failures on the build bots, it can apparently be triggered by the submission of new, unrelated tests such as this one:
https://codereview.chromium.org/2850233002/
,
May 4 2017
No, this is because the test uses CoreFoundation and then forks without execing. We can't use posix_spawn in death tests because those tests rely on the cloned parent/child state that is created when forking.
The main test binary, as part of setup, calls into CoreFoundation:
* frame #0: 0x00007fff91d3551a CoreFoundation`__CFNotificationCenterGetLocalCenter_block_invoke + 122
frame #1: 0x00007fffa78e78fc libdispatch.dylib`_dispatch_client_callout + 8
frame #2: 0x00007fffa78e78b9 libdispatch.dylib`dispatch_once_f + 38
frame #3: 0x00007fff91d35489 CoreFoundation`CFNotificationCenterGetLocalCenter + 41
frame #4: 0x00007fff91d342c0 CoreFoundation`_CFLocaleCopyCurrentGuts + 768
frame #5: 0x00007fff91d33f89 CoreFoundation`+[NSLocale currentLocale] + 9
frame #6: 0x00007fff93787346 Foundation`-[NSUserDefaults(NSUserDefaults) init] + 1589
frame #7: 0x00007fff93786cb5 Foundation`+[NSUserDefaults(NSUserDefaults) standardUserDefaults] + 81
frame #8: 0x00007fff9378eb30 Foundation`-[NSBundle URLForResource:withExtension:subdirectory:] + 192
frame #9: 0x00007fff93798573 Foundation`-[NSBundle pathForResource:ofType:] + 40
frame #10: 0x0000000102b4acfc libbase.dylib`base::mac::PathForFrameworkBundleResource(resourceName=0x002e828823a904a5) + 76 at foundation_util.mm:96
frame #11: 0x0000000103447d92 libbase_i18n.dylib`base::i18n::(anonymous namespace)::LazyInitIcuDataFile() + 290 at icu_util.cc:129
frame #12: 0x0000000103447c15 libbase_i18n.dylib`base::i18n::InitializeICU() + 197 at icu_util.cc:277
frame #13: 0x000000010108b93e base_unittests`base::test::InitializeICUForTesting() + 46 at icu_test_util.cc:27
frame #14: 0x00000001010b9019 base_unittests`base::TestSuite::Initialize(this=0x00007fff5fbff9b0) + 1641 at test_suite.cc:378
frame #15: 0x00000001010b789a base_unittests`base::TestSuite::Run(this=0x00007fff5fbff9b0) + 74 at test_suite.cc:259
This causes CF to set its flag for "used-CoreFoundation" in CHECK_FOR_FORK() [1]. Most calls into CF guard themselves with CHECK_FOR_FORK(). If a process does fork(), its pthread_atfork handler runs [2] and sets another flag for "did-fork".
In the forked death test child process, if anything else calls into CF and both the "used-CF" and "did-fork" flags are set, then it aborts with the __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() sentinel.
Looking at the child side of the death test, it appears that the call into CF is happening here:
Thread 1 Crashed:
0 libbase.dylib 0x0000000102aa3644 base::debug::BreakDebugger() + 20 (debugger_posix.cc:262)
1 libbase.dylib 0x0000000102aa4cee base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 78
2 libsystem_platform.dylib 0x00007fffa7b2cb3a _sigtramp + 26
3 ??? 000000000000000000 0 + 0
4 com.apple.CoreFoundation 0x00007fff91d45188 __CFRunLoopCreate + 520
5 com.apple.CoreFoundation 0x00007fff91d44e5e _CFRunLoopGet0 + 110
6 com.apple.Foundation 0x00007fff937b67af __NSThread__start__ + 189
7 libsystem_pthread.dylib 0x00007fffa7b369af _pthread_body + 180
8 libsystem_pthread.dylib 0x00007fffa7b368fb _pthread_start + 286
9 libsystem_pthread.dylib 0x00007fffa7b36101 thread_start + 13
That looks like its from the InitThreading() hack [3] we do to tell Cocoa that it's going to be "multi-threaded." I did confirm that making that function a noop that the crash stops happening. But that's probably not a real fix.
[1] https://opensource.apple.com/source/CF/CF-1153.18/CFInternal.h.auto.html
[2] https://opensource.apple.com/source/CF/CF-1153.18/CFRuntime.c.auto.html
[3] https://cs.chromium.org/chromium/src/base/threading/platform_thread_mac.mm?type=cs&q=NSThread+file:base&sq=package:chromium&l=39
,
May 4 2017
Would it make sense to implement gtest-death-test.cc in a way that doesn't use fork? Or does an exec after the fork? I guess another option would be to have some kind of flag to disable the InitThreading() code when doing this.
,
May 4 2017
GTest thankfully already has an option to not use fork: --gtest_death_test_style=threadsafe. This can be set on a per-test basis, so I'll do that for these.
,
May 4 2017
Sweet - thanks!
,
May 4 2017
https://chromium.googlesource.com/chromium/src/+/b4704531f34265eb7fa58c2e08fa6435519f45ee |
|||
►
Sign in to add a comment |
|||
Comment 1 by lgrey@chromium.org
, May 4 2017