Test assertion failures on Windows masquerade as timeouts. |
||||
Issue description
Chrome Version: 63
OS: Win10
What steps will reproduce the problem?
(1) Add a test that fails MSVC Debug Assertion. Easiest to do to piggy back it on an existing test, e.g., encoded_program_unittest.cc under courgette_unittests, with:
#include <vector>
...
TEST(BlahTest, Fail) {
std::vector<int> v;
int* ptr = &v[0];
EXPECT_EQ(ptr, ptr);
}
(2) Run the test in Windows debug trybot, e.g., win_chromium_dbg_ng.
What is the expected result?
Test failure due to assert error.
What happens instead?
Test failure due to timeout. This is misleading.
When running the test directly in Windows, a "Debug Assertion Failed!" modal dialog appears. After 45 seconds the dialog automatically disappears, and timeout is registered.
This failure mode was discovered by mheikal@.
,
Nov 10 2017
Thanks for the report. It looks like crrev.com/c/762106 should fix this. I guess some build settings drift caused the original solution to not work anymore. I tested in the context of base_unittests where I could reproduce the problem and verify the fix. I assume that the fix will work for all test binaries.
,
Nov 10 2017
Thanks for taking care of this! Assigning it to you.
,
Nov 10 2017
I'll also check a few other failure modes such as purevirt to make sure they don't cause hangs. I don't think that they do, but now is a good time to revisit this.
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b0ab776717a9b8b164e1e11b3b0d14ed412a2fd5 commit b0ab776717a9b8b164e1e11b3b0d14ed412a2fd5 Author: Bruce Dawson <brucedawson@chromium.org> Date: Wed Nov 15 01:12:35 2017 Fix SuppressErrorDialogs SuppressErrorDialogs is supposed to configure the VC++ CRT so that it doesn't bring up error dialogs that can cause test hangs. However it was only enabled when building with exceptions enabled which means that it wasn't doing anything. This restriction may have been appropriate when the code was written but it doesn't seem necessary now. The need for this change can be seen by adding this to any test file and running the test in a debug build: TEST(BlahTest, Fail) { std::vector<int> v; int* ptr = &v[0]; EXPECT_EQ(ptr, ptr); } Without this change an error dialog appears and the test hangs. With this change there is no error dialog and the output looks something like this: ...include\vector(1) : Assertion failed: vector subscript out of range [1/1] BlahTest.Fail (CRASHED) 1 test crashed: BlahTest.Fail (../../base/win/enum_variant_unittest.cc:13) Bug: 783040 Change-Id: I37b04de2f4df093164bd7a4ea281ea02c5e79007 Reviewed-on: https://chromium-review.googlesource.com/762106 Commit-Queue: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Mark Mentovai <mark@chromium.org> Cr-Commit-Position: refs/heads/master@{#516540} [modify] https://crrev.com/b0ab776717a9b8b164e1e11b3b0d14ed412a2fd5/base/test/test_suite.cc
,
Nov 15 2017
I tested purevirt and I verified that it crashes rather than bringing up a dialog. It prints:
abort() has been called
This is okay, but not great. Looking at my favorite reference on this topic (https://randomascii.wordpress.com/2012/07/22/more-adventures-in-failing-to-crash-properly/) I see the following recommendations that we are doing for Chrome but not for tests:
_set_purecall_handler
_set_invalid_parameter_handler
signal(SIGABRT, &AbortHandler);
Testing with a pure-call handler gives this result on that type of failure - *much* easier to see and use:
Received fatal exception EXCEPTION_BREAKPOINT
Backtrace:
base::`anonymous namespace'::PureCall [0x0195ABC9+25] (d:\src\chromium\src\base\test\test_suite.cc:334)
purecall [0x533E9F85+37]
MyBase::Blah [0x017DA0CE+14] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:18)
MyBase::~MyBase [0x017DA0A7+23] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:14)
MyDerived::~MyDerived [0x017D4CEF+15] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:21)
BlahTest_Fail_Test::TestBody [0x00466A57+87] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:35)
testing::internal::HandleExceptionsInMethodIfSupported<...> [0x018CB569+105] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2458)
testing::Test::Run [0x018CB476+182] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2479)
testing::TestInfo::Run [0x018CC13A+250] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2658)
testing::TestCase::Run [0x018CCE4B+267] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2771)
testing::internal::UnitTestImpl::RunAllTests [0x018D4F92+786] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:4676)
testing::internal::HandleExceptionsInMethodIfSupported<...> [0x018D4C2E+110] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2456)
testing::UnitTest::Run [0x018D4A24+324] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:4285)
RUN_ALL_TESTS [0x0195A0EF+15] (d:\src\chromium\src\third_party\googletest\src\googletest\include\gtest\gtest.h:2237)
base::TestSuite::Run [0x01959406+150] (d:\src\chromium\src\base\test\test_suite.cc:270)
base::internal::FunctorTraits<...> [0x01958D4C+28] (D:\src\chromium\src\base\bind_internal.h:194)
base::internal::InvokeHelper<0,int>::MakeItSo<...> [0x01958C6D+77] (D:\src\chromium\src\base\bind_internal.h:277)
base::internal::Invoker<...>::RunImpl<... [0x01958BF3+83] (D:\src\chromium\src\base\bind_internal.h:351)
base::internal::Invoker<...> >,int ()>::Run [0x01958ABE+62] (D:\src\chromium\src\base\bind_internal.h:333)
base::RepeatingCallback<int ()>::Run [0x004A7452+50] (D:\src\chromium\src\base\callback.h:94)
base::`anonymous namespace'::LaunchUnitTestsInternal [0x0195D145+549] (D:\src\chromium\src\base\test\launcher\unit_test_launcher.cc:217)
base::LaunchUnitTests [0x0195CEC0+192] (D:\src\chromium\src\base\test\launcher\unit_test_launcher.cc:555)
main [0x0195881E+142] (D:\src\chromium\src\base\test\run_all_base_unittests.cc:12)
invoke_main [0x01AF4BAE+30] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:78)
__scrt_common_main_seh [0x01AF4A50+336] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
__scrt_common_main [0x01AF48ED+13] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:326)
mainCRTStartup [0x01AF4C28+8] (f:\dd\vctools\crt\vcstartup\src\startup\exe_main.cpp:17)
BaseThreadInitThunk [0x765162C4+36]
RtlSubscribeWnfStateChangeNotification [0x77130F79+1081]
RtlSubscribeWnfStateChangeNotification [0x77130F44+1028]
[1/1] BlahTest.Fail (CRASHED)
1 test crashed:
BlahTest.Fail (../../base/win/enum_variant_unittest.cc:28)
Here's my purevirt test code. I'll land hooks for the remaining three cases tomorrow.
class MyBase {
public:
~MyBase() {
Blah();
}
virtual void Foo() = 0;
void Blah() {
Foo();
}
};
class MyDerived : MyBase {
public:
void Foo() override {
printf("Hello.\n");
}
};
TEST(BlahTest, Fail) {
MyDerived x;
(void)x;
}
,
Nov 15 2017
Here is complete test code for three types of failures - abort(), pure virtual function call, and invalid parameter handler. All of these failures were "caught" before as crashes, but with no call stacks. I have a CL ready that will turn them into real crashes which therefore give us call stacks and make these failures much easier to understand and investigate. Test code:
class MyBase {
public:
~MyBase() {
Blah();
}
virtual void Foo() = 0;
void Blah() {
Foo();
}
};
class MyDerived : MyBase {
public:
void Foo() override {
printf("Hello.\n");
}
};
TEST(BlahTest, Fail) {
abort();
//MyDerived x; (void)x;
//char buffer[5];
//sprintf_s(buffer, "Too long");
}
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/da276c95c4643ab2ddbbd9a3fe9c84aa16600fa2 commit da276c95c4643ab2ddbbd9a3fe9c84aa16600fa2 Author: Bruce Dawson <brucedawson@chromium.org> Date: Thu Nov 16 01:24:08 2017 Add handlers to crash tests more informatively The VC++ runtime "helpfully" catches various types of failures (pure virtual function calls, abort, and invalid parameters) and turns them into calls to abort. This is good for security but is bad for testing because it means that no call stack is printed on these "crashes". This CL changes the behavior so that these become actual crashes, with call stacks printed by the test framework. Messages printed (tested in debug builds) before the back trace now are: abort() has been called Received fatal exception EXCEPTION_BREAKPOINT Backtrace: ...\output.cpp(273) : Assertion failed: ("Buffer too small", 0) Received fatal exception EXCEPTION_BREAKPOINT Backtrace: Pure-virtual function call. Terminating. Received fatal exception EXCEPTION_BREAKPOINT Backtrace: Bug: 783040 Change-Id: I2f3d666af338e9b52b557ebbfb076b353888d6ff Reviewed-on: https://chromium-review.googlesource.com/772740 Commit-Queue: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Mark Mentovai <mark@chromium.org> Cr-Commit-Position: refs/heads/master@{#516939} [modify] https://crrev.com/da276c95c4643ab2ddbbd9a3fe9c84aa16600fa2/base/test/test_suite.cc
,
Nov 16 2017
I can't think of anything else that we're missing. Closing as fixed. |
||||
►
Sign in to add a comment |
||||
Comment 1 by huangs@google.com
, Nov 9 2017