various crash handling tests fail under win/asan |
|||||
Issue descriptionI added crashpad_test to the win/asan bots recently. Some of the tests don't pass: https://ci.chromium.org/buildbot/chromium.clang/CrWinAsan/623 CrashpadClient.HandlerLaunchFailureCrash CrashpadClient.HandlerLaunchFailureDumpAndCrash CrashpadHandler.ExtensibilityCalloutsWork SimulateCrash.ChildDumpWithoutCrashing ExceptionSnapshotWinTest.ChildCrash [ RUN ] CrashpadClient.HandlerLaunchFailureCrash ================================================================= ==2724==ERROR: AddressSanitizer: breakpoint on unknown address 0x00000000 (pc 0x0025e0f3 bp 0x0019f178 sp 0x0019f0c0 T0) ==2724==The signal is caused by a READ memory access. ==2724==Hint: address points to the zero page. [0520/062249.577:ERROR:crashpad_client_win.cc(491)] CreateProcess: The system cannot find the file specified. (0x2) #0 0x25e0f2 in crashpad::test::`anonymous namespace'::HandlerLaunchFailureCrash::WinMultiprocessChild+0x82 (e:\b\s\w\ir\out\Release\crashpad_tests.exe+0x4ae0f2) #1 0x25e022 in crashpad::test::WinMultiprocess::ChildProcessHelper<crashpad::test::(anonymous namespace)::HandlerLaunchFailureCrash>::Run C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\win\win_multiprocess.h:152 #2 0x25defe in crashpad::test::`anonymous namespace'::CrashpadClient_HandlerLaunchFailureCrash_Test::TestBody C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\client\crashpad_client_win_test.cc:131 #3 0x627379 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402 #4 0x5fda19 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472 #5 0x5fd588 in testing::Test::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2491 #6 0x5ffb32 in testing::TestInfo::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2667 #7 0x600ddf in testing::TestCase::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2785 #8 0x61b59d in testing::internal::UnitTestImpl::RunAllTests C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:5047 #9 0x627e29 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402 #10 0x61ad29 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472 #11 0x61a9d0 in testing::UnitTest::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4663 #12 0x8fe4b4 in main C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\gtest_main.cc:95 #13 0xa6dceb in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283 #14 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369) #15 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901) #16 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: breakpoint C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\client\crashpad_client_win_test.cc:123 in crashpad::test::`anonymous namespace'::HandlerLaunchFailureCrash::WinMultiprocessChild ==2724==ABORTING ../../third_party/crashpad/crashpad\test/win/win_multiprocess.h(68): error: Expected equality of these values: exit_code Which is: 1 parent_process.exit_code_ Which is: 4294930433 Stack trace: Backtrace: StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22) testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810) testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386) crashpad::test::`anonymous namespace'::CrashpadClient_HandlerLaunchFailureCrash_Test::TestBody [0x0025DA9C+2364] [ FAILED ] CrashpadClient.HandlerLaunchFailureCrash (172 ms) [ RUN ] CrashpadClient.HandlerLaunchFailureDumpAndCrash ================================================================= ==4148==ERROR: AddressSanitizer: access-violation on unknown address 0x00000004 (pc 0x005ed75c bp 0x016bf9e4 sp 0x016bf81c T0) ==4148==The signal is caused by a READ memory access. ==4148==Hint: address points to the zero page. #0 0x5ed75b in testing::UnitTest::AddTestPartResult C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4538 #1 0x5fc47f in testing::internal::ReportFailureInUnknownLocation C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2269 #2 0x627eae in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2410 #3 0x61ad29 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472 #4 0x61a9d0 in testing::UnitTest::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4663 #5 0x8fe4b4 in main C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\gtest_main.cc:95 #6 0xa6dceb in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283 #7 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369) #8 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901) #9 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: access-violation C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4538 in testing::UnitTest::AddTestPartResult ==4148==ABORTING ../../third_party/crashpad/crashpad\test/win/win_multiprocess.h(68): error: Expected equality of these values: exit_code Which is: 1 parent_process.exit_code_ Which is: 4294930433 Stack trace: Backtrace: StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22) testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810) testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386) crashpad::test::`anonymous namespace'::CrashpadClient_HandlerLaunchFailureDumpAndCrash_Test::TestBody [0x0025F1BC+2364] (C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\client\crashpad_client_win_test.cc:155) [ FAILED ] CrashpadClient.HandlerLaunchFailureDumpAndCrash (109 ms) [ RUN ] CrashpadHandler.ExtensibilityCalloutsWork ================================================================= ==4600==ERROR: AddressSanitizer: unknown exception on unknown address 0x00000000 (pc 0x00000000 bp 0x0162ebf8 sp 0x0162e4fc T0) ==4600==Hint: pc points to the zero page. ==4600==The signal is caused by a READ memory access. ==4600==Hint: address points to the zero page. #0 0x26d7f8 in crashpad::test::`anonymous namespace'::CrashWithExtendedHandler::WinMultiprocessChild+0x5e8 (e:\b\s\w\ir\out\Release\crashpad_tests.exe+0x4bd7f8) #1 0x26d09b in crashpad::test::WinMultiprocess::ChildProcessHelper<crashpad::test::(anonymous namespace)::CrashWithExtendedHandler>::Run C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\win\win_multiprocess.h:152 #2 0x26cf66 in crashpad::test::`anonymous namespace'::CrashpadHandler_ExtensibilityCalloutsWork_Test::TestBody C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\handler\crashpad_handler_test.cc:138 #3 0x627379 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402 #4 0x5fda19 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472 #5 0x5fd588 in testing::Test::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2491 #6 0x5ffb32 in testing::TestInfo::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2667 #7 0x600ddf in testing::TestCase::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2785 #8 0x61b59d in testing::internal::UnitTestImpl::RunAllTests C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:5047 #9 0x627e29 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2402 #10 0x61ad29 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:2472 #11 0x61a9d0 in testing::UnitTest::Run C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:4663 #12 0x8fe4b4 in main C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\test\gtest_main.cc:95 #13 0xa6dceb in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283 #14 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369) #15 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901) #16 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: unknown exception C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\handler\crashpad_handler_test.cc:76 in crashpad::test::`anonymous namespace'::CrashWithExtendedHandler::WinMultiprocessChild ==4600==ABORTING ../../third_party/crashpad/crashpad\test/win/win_multiprocess.h(68): error: Expected equality of these values: exit_code Which is: 1 parent_process.exit_code_ Which is: 485163226 Stack trace: Backtrace: StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22) testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810) testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386) crashpad::test::`anonymous namespace'::CrashpadHandler_ExtensibilityCalloutsWork_Test::TestBody [0x0026CA71+2385] [ FAILED ] CrashpadHandler.ExtensibilityCalloutsWork (140 ms) [ RUN ] SimulateCrash.ChildDumpWithoutCrashing ../../third_party/crashpad/crashpad/snapshot/win/exception_snapshot_win_test.cc(211): error: Expected: (snapshot.Exception()->Context()->InstructionPointer()) < (dump_near_ + kAllowedOffset), actual: 15275245 vs 15275188 Stack trace: Backtrace: StackTraceGetter::CurrentStackTrace [0x005C894A+330] (C:\b\c\b\CrWinAsan\src\third_party\googletest\custom\gtest\internal\custom\stack_trace_getter.cc:22) testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x005EDDDB+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:810) testing::internal::AssertHelper::operator= [0x005ECCF9+201] (C:\b\c\b\CrWinAsan\src\third_party\googletest\src\googletest\src\gtest.cc:386) crashpad::test::`anonymous namespace'::SimulateDelegate::ExceptionHandlerServerException [0x003C5B47+1959] crashpad::ExceptionHandlerServer::OnNonCrashDumpEvent [0x008122D6+374] (C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\util\win\exception_handler_server.cc:565) RtlRegisterWait [0x77A31CF2+788] RtlIsCriticalSectionLockedByThread [0x77A10CE8+766] TpCallbackIndependent [0x77A10951+1808] BaseThreadInitThunk [0x7539336A+18] RtlInitializeExceptionChain [0x779F9902+99] RtlInitializeExceptionChain [0x779F98D5+54] [ FAILED ] SimulateCrash.ChildDumpWithoutCrashing (94 ms) [ RUN ] ExceptionSnapshotWinTest.ChildCrash ================================================================= ==4632==ERROR: AddressSanitizer: breakpoint on unknown address 0x00000000 (pc 0x001a148b bp 0x0091f9c4 sp 0x0091f240 T0) ==4632==The signal is caused by a READ memory access. ==4632==Hint: address points to the zero page. #0 0x1a148a in wmain C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\snapshot\win\crashpad_snapshot_test_crashing_child.cc:49 #1 0x20535b in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283 #2 0x75393369 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73369) #3 0x779f9901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901) #4 0x779f98d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: breakpoint C:\b\c\b\CrWinAsan\src\third_party\crashpad\crashpad\snapshot\win\crashpad_snapshot_test_crashing_child.cc:49 in wmain ==4632==ABORTING Probably should disable these tests under asan for now and then investigate later.
,
May 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/92d07a3931ffebea0ce216b9b0f28277de8ce977 commit 92d07a3931ffebea0ce216b9b0f28277de8ce977 Author: Nico Weber <thakis@chromium.org> Date: Sun May 20 15:24:17 2018 disable failing crashpad_test tests under win/asan TBR=scottmg Bug: 845011 Change-Id: I06891b947b6eb444ab8a6af7b2f9a574e04d40f0 Reviewed-on: https://chromium-review.googlesource.com/1067151 Reviewed-by: Nico Weber <thakis@chromium.org> Commit-Queue: Nico Weber <thakis@chromium.org> Cr-Commit-Position: refs/heads/master@{#560200} [modify] https://crrev.com/92d07a3931ffebea0ce216b9b0f28277de8ce977/third_party/crashpad/crashpad/client/crashpad_client_win_test.cc [modify] https://crrev.com/92d07a3931ffebea0ce216b9b0f28277de8ce977/third_party/crashpad/crashpad/handler/crashpad_handler_test.cc [modify] https://crrev.com/92d07a3931ffebea0ce216b9b0f28277de8ce977/third_party/crashpad/crashpad/snapshot/win/exception_snapshot_win_test.cc
,
May 28 2018
I just looked into these tests. These four CrashpadClient.HandlerLaunchFailureCrash CrashpadClient.HandlerLaunchFailureDumpAndCrash CrashpadHandler.ExtensibilityCalloutsWork ExceptionSnapshotWinTest.ChildCrash are failing because ASAN captures an expected crash without letting crashpad see it, and either terminates the process with an unexpected error code, or causes a hang because the test fixture doesn't notified. So I think these are fine to leave disabled when running under ASAN. Unless there's a way to tag specific crash locations as uninteresting so that ASAN will let those through and we can check the rest of the code in those tests? SimulateCrash.ChildDumpWithoutCrashing is doing an instruction pointer location compare, and the ASAN instrumentation made the code larger than expected. I posted a CL to bump up the value when under ASAN. https://chromium-review.googlesource.com/c/crashpad/crashpad/+/1075715 I also upstreamed the rest of the disabling CL https://chromium-review.googlesource.com/1067151 there.
,
May 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/crashpad/crashpad.git/+/5191992ae54c5e349130e0e9394349568ee11b8b commit 5191992ae54c5e349130e0e9394349568ee11b8b Author: Scott Graham <scottmg@chromium.org> Date: Tue May 29 17:04:18 2018 win: Fix SimulateCrash.ChildDumpWithoutCrashing under ASAN, disable others SimulateCrash.ChildDumpWithoutCrashing needed a larger threshold due to ASAN instrumentation. These tests expect children to crash, but ASAN captures the exception before letting Crashpad handle it: CrashpadClient.HandlerLaunchFailureCrash CrashpadClient.HandlerLaunchFailureDumpAndCrash CrashpadHandler.ExtensibilityCalloutsWork ExceptionSnapshotWinTest.ChildCrash (which is an upstreaming of https://chromium-review.googlesource.com/1067151). Additionally, because Chrome doesn't build all, I noticed a missing dependency on a test binary which is added here. Bug: chromium:845011 Change-Id: I5c3ae5673512be29edad21e7d20dd57b8b5ce2bf Reviewed-on: https://chromium-review.googlesource.com/1075715 Reviewed-by: Joshua Peraza <jperaza@chromium.org> Commit-Queue: Scott Graham <scottmg@chromium.org> [modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/client/crashpad_client_win_test.cc [modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/handler/BUILD.gn [modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/handler/crashpad_handler_test.cc [modify] https://crrev.com/5191992ae54c5e349130e0e9394349568ee11b8b/snapshot/win/exception_snapshot_win_test.cc
,
May 29 2018
rnk: Do you have an opinion on comment 3? Do we disable "child crashes" tests in chromium's tests under asan too?
,
May 29 2018
Maybe the easiest way forward would be to disable ASan's crash handler in all crashpad_tests. Vitaly, I remember ASan recently changed all the defaults around allow_user_segv_handler/handle_segv, etc, because it was a problem for Go or some other customer. I remember that Clusterfuzz wants all fuzz targets to use the same crash report format for ASan error reports and regular crashes, which are typically null pointer derefs. Should we try to handle access violation exceptions by default on Windows? Should clusterfuzz set their own custom ASAN_OPTIONS to get the behavior they want? If we go that way, ASan will violate fewer application expectations, and most crash handler tests should start to pass without modification. In the short term, disabling tests seems fine.
,
May 30 2018
Yes, but not on Windows: http://llvm-cs.pcc.me.uk/projects/compiler-rt/lib/sanitizer_common/sanitizer_win.cc#852 Default is to set Asan as signal handler, but let application to override it. Clusterfuzz wants Asan as the only signal handler, blocking overrides from application. Clusterfuzz sets behavior with ASAN_OPTIONS. But these should be irrelevant to Windows tests?
,
Jun 14 2018
Issue 782907 has been merged into this issue.
,
Jun 20 2018
I don't think we can do what ASan on Linux does, where ASan sets the unhandled exception filter, and then lets the user override it, because the CRT always overrides it. We inherently have to fight the CRT to install our "signal handler" if we want it to work at all.
I think we should add this code to src/build/sanitizers/sanitizer_options.cc:
#if defined(OS_WIN)
SANITIZER_HOOK_ATTRIBUTE const char *__asan_default_options() {
// Disable ASan's access violation handler by default. It interferes too much
// with crash testing. ClusterFuzz and other fuzzing tools should explicitly
// override these in the ASAN_OPTIONS environment variable for uniform crash
// reporting.
return "handle_segv=0 handle_abort=0 handle_sigill=0 handle_sigfpe=0";
}
#endif // OS_WIN
That will cause ASan to install its exception filter, but always delegate to Chrome's (or crashpad's) exception filter by default.
I verified that ClusterFuzz always sets ASAN_OPTIONS=handle_segv=1 handle_sigill=1... in the environment, so its behavior won't be affected. ClusterFuzz will get ASan reports for both access violation crashes and ASan UAF reports.
,
Jun 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d7685a5fd09bcf5144008cec5d8f3396dda6eacd commit d7685a5fd09bcf5144008cec5d8f3396dda6eacd Author: Reid Kleckner <rnk@google.com> Date: Wed Jun 20 20:34:31 2018 Change headless crash handler test expectations under ASan ASan on Windows changes the exit code to 1, which base categorizes as the process being killed. R=dgozman@chromium.org BUG=845011 Change-Id: I42149965f2904a2a8349a00f4a2dd70351d2eee1 Reviewed-on: https://chromium-review.googlesource.com/1108448 Reviewed-by: Dmitry Gozman <dgozman@chromium.org> Commit-Queue: Reid Kleckner <rnk@chromium.org> Cr-Commit-Position: refs/heads/master@{#568994} [modify] https://crrev.com/d7685a5fd09bcf5144008cec5d8f3396dda6eacd/headless/lib/headless_devtools_client_browsertest.cc |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by thakis@chromium.org
, May 20 2018