New issue
Advanced search Search tips

Issue 783040 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Test assertion failures on Windows masquerade as timeouts.

Project Member Reported by huangs@google.com, Nov 9 2017

Issue description

Chrome Version: 63
OS: Win10

What steps will reproduce the problem?
(1) Add a test that fails MSVC Debug Assertion. Easiest to do to piggy back it on an existing test, e.g., encoded_program_unittest.cc under courgette_unittests, with:

#include <vector>

...

TEST(BlahTest, Fail) {
  std::vector<int> v;
  int* ptr = &v[0];
  EXPECT_EQ(ptr, ptr);
}

(2) Run the test in Windows debug trybot, e.g., win_chromium_dbg_ng.

What is the expected result?
Test failure due to assert error.

What happens instead?
Test failure due to timeout. This is misleading.

When running the test directly in Windows, a "Debug Assertion Failed!" modal dialog appears. After 45 seconds the dialog automatically disappears, and timeout is registered.

This failure mode was discovered by mheikal@.

 

Comment 1 by huangs@google.com, Nov 9 2017

Sample CL that triggers this failure:

https://chromium-review.googlesource.com/c/chromium/src/+/758923

Thanks for the report.

It looks like crrev.com/c/762106 should fix this. I guess some build settings drift caused the original solution to not work anymore.

I tested in the context of base_unittests where I could reproduce the problem and verify the fix. I assume that the fix will work for all test binaries.

Comment 3 by huangs@google.com, Nov 10 2017

Owner: brucedaw...@chromium.org
Status: Assigned (was: Untriaged)
Thanks for taking care of this! Assigning it to you.
Status: Started (was: Assigned)
I'll also check a few other failure modes such as purevirt to make sure they don't cause hangs. I don't think that they do, but now is a good time to revisit this.
Project Member

Comment 5 by bugdroid1@chromium.org, Nov 15 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b0ab776717a9b8b164e1e11b3b0d14ed412a2fd5

commit b0ab776717a9b8b164e1e11b3b0d14ed412a2fd5
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Wed Nov 15 01:12:35 2017

Fix SuppressErrorDialogs

SuppressErrorDialogs is supposed to configure the VC++ CRT so that it
doesn't bring up error dialogs that can cause test hangs. However it was
only enabled when building with exceptions enabled which means that it
wasn't doing anything. This restriction may have been appropriate when
the code was written but it doesn't seem necessary now.

The need for this change can be seen by adding this to any test file
and running the test in a debug build:

  TEST(BlahTest, Fail) {
    std::vector<int> v;
    int* ptr = &v[0];
    EXPECT_EQ(ptr, ptr);
  }

Without this change an error dialog appears and the test hangs. With
this change there is no error dialog and the output looks something like
this:

  ...include\vector(1) : Assertion failed: vector subscript out of range
  [1/1] BlahTest.Fail (CRASHED)
  1 test crashed:
      BlahTest.Fail (../../base/win/enum_variant_unittest.cc:13)

Bug:  783040 
Change-Id: I37b04de2f4df093164bd7a4ea281ea02c5e79007
Reviewed-on: https://chromium-review.googlesource.com/762106
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Mark Mentovai <mark@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516540}
[modify] https://crrev.com/b0ab776717a9b8b164e1e11b3b0d14ed412a2fd5/base/test/test_suite.cc

I tested purevirt and I verified that it crashes rather than bringing up a dialog. It prints:

    abort() has been called

This is okay, but not great. Looking at my favorite reference on this topic (https://randomascii.wordpress.com/2012/07/22/more-adventures-in-failing-to-crash-properly/) I see the following recommendations that we are doing for Chrome but not for tests:

    _set_purecall_handler
    _set_invalid_parameter_handler
    signal(SIGABRT, &AbortHandler);

Testing with a pure-call handler gives this result on that type of failure - *much* easier to see and use:

Received fatal exception EXCEPTION_BREAKPOINT
Backtrace:
        base::`anonymous namespace'::PureCall [0x0195ABC9+25] (d:\src\chromium\src\base\test\test_suite.cc:334)
        purecall [0x533E9F85+37]
        MyBase::Blah [0x017DA0CE+14] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:18)
        MyBase::~MyBase [0x017DA0A7+23] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:14)
        MyDerived::~MyDerived [0x017D4CEF+15] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:21)
        BlahTest_Fail_Test::TestBody [0x00466A57+87] (d:\src\chromium\src\base\win\enum_variant_unittest.cc:35)
        testing::internal::HandleExceptionsInMethodIfSupported<...> [0x018CB569+105] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2458)
        testing::Test::Run [0x018CB476+182] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2479)
        testing::TestInfo::Run [0x018CC13A+250] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2658)
        testing::TestCase::Run [0x018CCE4B+267] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2771)
        testing::internal::UnitTestImpl::RunAllTests [0x018D4F92+786] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:4676)
        testing::internal::HandleExceptionsInMethodIfSupported<...> [0x018D4C2E+110] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:2456)
        testing::UnitTest::Run [0x018D4A24+324] (d:\src\chromium\src\third_party\googletest\src\googletest\src\gtest.cc:4285)
        RUN_ALL_TESTS [0x0195A0EF+15] (d:\src\chromium\src\third_party\googletest\src\googletest\include\gtest\gtest.h:2237)
        base::TestSuite::Run [0x01959406+150] (d:\src\chromium\src\base\test\test_suite.cc:270)
        base::internal::FunctorTraits<...> [0x01958D4C+28] (D:\src\chromium\src\base\bind_internal.h:194)
        base::internal::InvokeHelper<0,int>::MakeItSo<...> [0x01958C6D+77] (D:\src\chromium\src\base\bind_internal.h:277)
        base::internal::Invoker<...>::RunImpl<... [0x01958BF3+83] (D:\src\chromium\src\base\bind_internal.h:351)
        base::internal::Invoker<...> >,int ()>::Run [0x01958ABE+62] (D:\src\chromium\src\base\bind_internal.h:333)
        base::RepeatingCallback<int ()>::Run [0x004A7452+50] (D:\src\chromium\src\base\callback.h:94)
        base::`anonymous namespace'::LaunchUnitTestsInternal [0x0195D145+549] (D:\src\chromium\src\base\test\launcher\unit_test_launcher.cc:217)
        base::LaunchUnitTests [0x0195CEC0+192] (D:\src\chromium\src\base\test\launcher\unit_test_launcher.cc:555)
        main [0x0195881E+142] (D:\src\chromium\src\base\test\run_all_base_unittests.cc:12)
        invoke_main [0x01AF4BAE+30] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:78)
        __scrt_common_main_seh [0x01AF4A50+336] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
        __scrt_common_main [0x01AF48ED+13] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:326)
        mainCRTStartup [0x01AF4C28+8] (f:\dd\vctools\crt\vcstartup\src\startup\exe_main.cpp:17)
        BaseThreadInitThunk [0x765162C4+36]
        RtlSubscribeWnfStateChangeNotification [0x77130F79+1081]
        RtlSubscribeWnfStateChangeNotification [0x77130F44+1028]
[1/1] BlahTest.Fail (CRASHED)
1 test crashed:
    BlahTest.Fail (../../base/win/enum_variant_unittest.cc:28)

Here's my purevirt test code. I'll land hooks for the remaining three cases tomorrow.

class MyBase {
 public:
   ~MyBase() {
     Blah();
   }
   virtual void Foo() = 0;
   void Blah() {
     Foo();
   }
};

class MyDerived : MyBase {
 public:
  void Foo() override {
    printf("Hello.\n");
  }
};

TEST(BlahTest, Fail) {
  MyDerived x;
  (void)x;
}

Here is complete test code for three types of failures - abort(), pure virtual function call, and invalid parameter handler. All of these failures were "caught" before as crashes, but with no call stacks. I have a CL ready that will turn them into real crashes which therefore give us call stacks and make these failures much easier to understand and investigate. Test code:

class MyBase {
 public:
   ~MyBase() {
     Blah();
   }
   virtual void Foo() = 0;
   void Blah() {
     Foo();
   }
};

class MyDerived : MyBase {
 public:
  void Foo() override {
    printf("Hello.\n");
  }
};

TEST(BlahTest, Fail) {
  abort();

  //MyDerived x; (void)x;

  //char buffer[5];
  //sprintf_s(buffer, "Too long");
}

Project Member

Comment 8 by bugdroid1@chromium.org, Nov 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da276c95c4643ab2ddbbd9a3fe9c84aa16600fa2

commit da276c95c4643ab2ddbbd9a3fe9c84aa16600fa2
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Thu Nov 16 01:24:08 2017

Add handlers to crash tests more informatively

The VC++ runtime "helpfully" catches various types of failures (pure
virtual function calls, abort, and invalid parameters) and turns them
into calls to abort. This is good for security but is bad for testing
because it means that no call stack is printed on these "crashes". This
CL changes the behavior so that these become actual crashes, with call
stacks printed by the test framework.

Messages printed (tested in debug builds) before the back trace now
are:

  abort() has been called
  Received fatal exception EXCEPTION_BREAKPOINT
  Backtrace:

  ...\output.cpp(273) : Assertion failed: ("Buffer too small", 0)
  Received fatal exception EXCEPTION_BREAKPOINT
  Backtrace:

  Pure-virtual function call. Terminating.
  Received fatal exception EXCEPTION_BREAKPOINT
  Backtrace:

Bug:  783040 
Change-Id: I2f3d666af338e9b52b557ebbfb076b353888d6ff
Reviewed-on: https://chromium-review.googlesource.com/772740
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Mark Mentovai <mark@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516939}
[modify] https://crrev.com/da276c95c4643ab2ddbbd9a3fe9c84aa16600fa2/base/test/test_suite.cc

Status: Fixed (was: Started)
I can't think of anything else that we're missing. Closing as fixed.

Sign in to add a comment