HotwordServiceTest fails under ubsan |
||||||||
Issue descriptionVersion: tip OS: Linux x86-64 Per 'UBSanVptr Linux' buildbot, HotwordServiceTest is failing: https://build.chromium.org/p/chromium.fyi/builders/UBSanVptr%20Linux/builds/408/steps/unit_tests/logs/HotwordServiceTests_HotwordServiceTest.AudioHistorySyncOccurs_0 HotwordServiceTests/HotwordServiceTest.AudioHistorySyncOccurs/0 (run #1): [ RUN ] HotwordServiceTests/HotwordServiceTest.AudioHistorySyncOccurs/0 ../../base/bind_internal.h:187:12: runtime error: member call on address 0x010fe4858200 which does not point to an object of type 'HotwordService' 0x010fe4858200: note: object has invalid vptr 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^~~~~~~~~~~~~~~~~~~~~~~ invalid vptr The stack trace is not symbolized there, but based on the message it seems like another case of 'use-before-new'. Reproducing this locally...
,
Jun 7 2016
Okay, I figured out what happened. The fix is https://codereview.chromium.org/2045833003/
,
Jun 7 2016
Ah, not a real fix, as the additions to component_loader_unittest.cc look completely arbitrary. Input from owners required.
Kendra, can you please take a look?
To reproduce the failures:
$ gn gen //out/gn-vptr '--args=is_ubsan_vptr=true is_ubsan_no_recover=true is_debug=false is_component_build=false symbol_level=1 dcheck_always_on=true' --check
$ ninja -C out/gn-vptr unit_tests
$ ./out/gn-vptr/unit_tests --gtest_filter=ComponentLoaderTest.LoadAll
...
../../base/bind_internal.h:187:12: runtime error: member call on address 0x208aa54b96c0 which does not point to an object of type 'HotwordService'
0x208aa54b96c0: note: object has invalid vptr
00 00 00 00 09 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0 79 67 a5
^~~~~~~~~~~~~~~~~~~~~~~
invalid vptr
#0 0x4d1fe77 in Run<HotwordService *> out/gn-vptr/../../base/bind_internal.h:187:12
#1 0x4d1fe77 in MakeItSo<base::internal::RunnableAdapter<void (HotwordService::*)()> &, HotwordService *> out/gn-vptr/../../base/bind_internal.h:312
#2 0x4d1fe77 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (HotwordService::*)()>, void (HotwordService*), base::internal::UnretainedWrapper<HotwordService> >, false, void ()>::Run(base::internal::BindStateBase*) out/gn-vptr/../../base/bind_internal.h:364
#3 0xee2a21d in Run out/gn-vptr/../../base/callback.h:397:12
#4 0xee2a21d in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) out/gn-vptr/../../base/debug/task_annotator.cc:51
#5 0xee59987 in base::MessageLoop::RunTask(base::PendingTask const&) out/gn-vptr/../../base/message_loop/message_loop.cc:475:19
#6 0xee5a1d8 in base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&) out/gn-vptr/../../base/message_loop/message_loop.cc:484:5
#7 0xee5ac03 in base::MessageLoop::DoWork() out/gn-vptr/../../base/message_loop/message_loop.cc:601:13
#8 0xee5fae3 in base::MessagePumpGlib::HandleDispatch() out/gn-vptr/../../base/message_loop/message_pump_glib.cc:267:25
#9 0xee60a90 in base::(anonymous namespace)::WorkSourceDispatch(_GSource*, int (*)(void*), void*) out/gn-vptr/../../base/message_loop/message_pump_glib.cc:109:43
#10 0x7f4c246e3e03 in g_main_context_dispatch (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x48e03)
#11 0x7f4c246e4047 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x49047)
#12 0x7f4c246e40eb in g_main_context_iteration (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x490eb)
#13 0xee60429 in base::MessagePumpGlib::Run(base::MessagePump::Delegate*) out/gn-vptr/../../base/message_loop/message_pump_glib.cc:309:30
#14 0xee58f10 in base::MessageLoop::RunHandler() out/gn-vptr/../../base/message_loop/message_loop.cc:439:10
#15 0xeea78d8 in base::RunLoop::Run() out/gn-vptr/../../base/run_loop.cc:35:10
#16 0x330c9d2 in content::RunThisRunLoop(base::RunLoop*) out/gn-vptr/../../content/public/test/test_utils.cc:135:13
#17 0x330cc39 in content::RunAllPendingInMessageLoop() out/gn-vptr/../../content/public/test/test_utils.cc:144:3
#18 0x330ccb1 in content::RunAllPendingInMessageLoop(content::BrowserThread::ID) out/gn-vptr/../../content/public/test/test_utils.cc:149:5
#19 0x320951f in TestingProfile::~TestingProfile() out/gn-vptr/../../chrome/test/base/testing_profile.cc:528:5
#20 0x1e97af3 in extensions::ComponentLoaderTest::~ComponentLoaderTest() out/gn-vptr/../../chrome/browser/extensions/component_loader_unittest.cc:80:7
#21 0x1e97c6f in extensions::ComponentLoaderTest_LoadAll_Test::~ComponentLoaderTest_LoadAll_Test() out/gn-vptr/../../chrome/browser/extensions/component_loader_unittest.cc:230:1
#22 0xebbee7e in testing::TestInfo::Run() out/gn-vptr/../../testing/gtest/src/gtest.cc:2661:3
#23 0xebbfd01 in testing::TestCase::Run() out/gn-vptr/../../testing/gtest/src/gtest.cc:2774:28
#24 0xebcce07 in testing::internal::UnitTestImpl::RunAllTests() out/gn-vptr/../../testing/gtest/src/gtest.cc:4647:43
#25 0xebcc1d8 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) out/gn-vptr/../../testing/gtest/src/gtest.cc:2458:12
#26 0xebcbfcc in testing::UnitTest::Run() out/gn-vptr/../../testing/gtest/src/gtest.cc:4255:10
#27 0x322075f in RUN_ALL_TESTS out/gn-vptr/../../testing/gtest/include/gtest/gtest.h:2237:46
#28 0x322075f in base::TestSuite::Run() out/gn-vptr/../../base/test/test_suite.cc:230
#29 0x3231301 in Run out/gn-vptr/../../base/callback.h:397:12
#30 0x3231301 in base::(anonymous namespace)::LaunchUnitTestsInternal(base::Callback<int (), (base::internal::CopyMode)1> const&, int, int, bool, base::Callback<void (), (base::internal::CopyMode)1> const&) out/gn-vptr/../../base/test/launcher/unit_test_launcher.cc:206
#31 0x32311b8 in base::LaunchUnitTests(int, char**, base::Callback<int (), (base::internal::CopyMode)1> const&) out/gn-vptr/../../base/test/launcher/unit_test_launcher.cc:445:10
#32 0x321504b in main out/gn-vptr/../../chrome/test/base/run_all_unittests.cc:21:10
#33 0x7f4c1d623f44 in __libc_start_main /build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287
#34 0x79582c in _start (/usr/local/google/home/krasin/chr22/src/out/gn-vptr/unit_tests+0x79582c)
Adding base::MessageLoop::current()->RunUntilIdle(); to the end of the test case fixes the problem as the callback gets executed while HotwordService is still alive.
,
Jun 7 2016
Kendra, friendly ping. There's a chance that my fix is actually correct, but I need a look from a code owner. Note that this issue keeps a buildbot red: https://build.chromium.org/p/chromium.fyi/builders/UBSanVptr%20Linux/builds/441/steps/unit_tests
,
Jun 7 2016
,
Jun 8 2016
Hi, Kendra is OOO this week. I'm not familiar with the code here; I am an owner so I can LG simple changes but I don't deeply understand what's going on here. Does your CL not work? What do you mean "not a real fix"?
,
Jun 8 2016
It does not seem to be a real fix, as I had to add a completely random line of code into a test unrelated to HotwordService. While this line fixes the crash, it does not prevent the very same issue to appear in other tests (or, worse, in real Chrome). The proper fix would be inside HotwordService with no changes in unrelated tests. re: Kendra is OOO: sorry, I didn't know that. Unless, there's someone around who knows the code, it seems that the best course of action would be just to wait for Kendra.
,
Jun 9 2016
abodenha: Is there anyone who can take over on eng and look at HotwordService?
,
Jun 13 2016
Today is the last day before I go on vacation. While I plan to check out mail from time to time, my responsibility will be reduced. To summarize: the problem still breaks the UBSan bots; I have a shallow fix: https://codereview.chromium.org/2045833003/ The code owners should take a look and decide on the proper way to fix the issue.
,
Jun 14 2016
,
Jun 14 2016
Hi, sorry, I've been OOO. Adding amistry in case he has any insight on this.
,
Jun 14 2016
krasin@, is this failure new? As far as I know this code hasn't been touched in years.
,
Jun 14 2016
It might not be new. Hard to say, as UBSan Vptr bot started to run this test just recently.
,
Jun 15 2016
Looks like it might be HotwordService::InitializeMicrophoneObserver which is bound using a base::Unretained instead of via the WeakPtr like most other things. It's created and posted to the UI thread on Init, so it's probably still in the task queue when the test is done.
,
Jun 15 2016
amistry@, Thanks for the analysis! I have a CL proposed and all the unit tests has passed. https://codereview.chromium.org/2067183003/
,
Jun 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4a66954b1826d4fde1b4eae316f4512be17a0a02 commit 4a66954b1826d4fde1b4eae316f4512be17a0a02 Author: xiaoyinh <xiaoyinh@chromium.org> Date: Thu Jun 16 17:51:53 2016 Use WeakPtr to ensure automatic cancellation of tasks. Use WeakPtr in HotwordService Constructor to ensure that, invokes to InitializeMicrophoneObserver cannot outlive HotwordService Object. BUG= 617332 TEST=Unit Tests Review-Url: https://codereview.chromium.org/2067183003 Cr-Commit-Position: refs/heads/master@{#400196} [modify] https://crrev.com/4a66954b1826d4fde1b4eae316f4512be17a0a02/chrome/browser/search/hotword_service.cc
,
Jun 16 2016
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by krasin@chromium.org
, Jun 6 2016Better stack trace and diagnostics: [ RUN ] HotwordServiceTests/HotwordServiceTest.AudioHistorySyncOccurs/0 ../../base/bind_internal.h:187:12: runtime error: member call on address 0x2f724432b780 which does not point to an object of type 'HotwordService' 0x2f724432b780: note: object has invalid vptr 00 00 00 00 8f 44 ba bb 8d d0 ff ff 0f d6 ff ff ff ff ff ff f3 55 c1 b5 65 00 00 00 40 75 40 44 ^~~~~~~~~~~~~~~~~~~~~~~ invalid vptr #0 0x4d1fe77 in Run<HotwordService *> out/gn-vptr/../../base/bind_internal.h:187:12 #1 0x4d1fe77 in MakeItSo<base::internal::RunnableAdapter<void (HotwordService::*)()> &, HotwordService *> out/gn-vptr/../../base/bind_internal.h:312 #2 0x4d1fe77 in base::internal::Invoker<base::IndexSequence<0ul>, base::internal::BindState<base::internal::RunnableAdapter<void (HotwordService::*)()>, void (HotwordService*), base::internal::UnretainedWrapper<HotwordService> >, false, void ()>::Run(base::internal::BindStateBase*) out/gn-vptr/../../base/bind_internal.h:364 #3 0xee2a21d in Run out/gn-vptr/../../base/callback.h:397:12 #4 0xee2a21d in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) out/gn-vptr/../../base/debug/task_annotator.cc:51 #5 0xee59987 in base::MessageLoop::RunTask(base::PendingTask const&) out/gn-vptr/../../base/message_loop/message_loop.cc:475:19 #6 0xee5a1d8 in base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&) out/gn-vptr/../../base/message_loop/message_loop.cc:484:5 #7 0xee5ac03 in base::MessageLoop::DoWork() out/gn-vptr/../../base/message_loop/message_loop.cc:601:13 #8 0xee62d75 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) out/gn-vptr/../../base/message_loop/message_pump_libevent.cc:217:31 #9 0xee58f10 in base::MessageLoop::RunHandler() out/gn-vptr/../../base/message_loop/message_loop.cc:439:10 #10 0xeea78d8 in base::RunLoop::Run() out/gn-vptr/../../base/run_loop.cc:35:10 #11 0x330c9d2 in content::RunThisRunLoop(base::RunLoop*) out/gn-vptr/../../content/public/test/test_utils.cc:135:13 #12 0x330cc39 in content::RunAllPendingInMessageLoop() out/gn-vptr/../../content/public/test/test_utils.cc:144:3 #13 0x330ccb1 in content::RunAllPendingInMessageLoop(content::BrowserThread::ID) out/gn-vptr/../../content/public/test/test_utils.cc:149:5 #14 0x320951f in TestingProfile::~TestingProfile() out/gn-vptr/../../chrome/test/base/testing_profile.cc:528:5 #15 0x3209daf in TestingProfile::~TestingProfile() out/gn-vptr/../../chrome/test/base/testing_profile.cc:503:35 #16 0x202ee43 in operator() out/gn-vptr/../../build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2 #17 0x202ee43 in reset out/gn-vptr/../../build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245 #18 0x202ee43 in extensions::ExtensionServiceTestBase::~ExtensionServiceTestBase() out/gn-vptr/../../chrome/browser/extensions/extension_service_test_base.cc:108 #19 0x22e8b51 in HotwordServiceTest::~HotwordServiceTest() out/gn-vptr/../../chrome/browser/search/hotword_service_unittest.cc:131:34 #20 0x22e92df in HotwordServiceTest_AudioHistorySyncOccurs_Test::~HotwordServiceTest_AudioHistorySyncOccurs_Test() out/gn-vptr/../../chrome/browser/search/hotword_service_unittest.cc:506:1 #21 0xebbee7e in testing::TestInfo::Run() out/gn-vptr/../../testing/gtest/src/gtest.cc:2661:3 #22 0xebbfd01 in testing::TestCase::Run() out/gn-vptr/../../testing/gtest/src/gtest.cc:2774:28 #23 0xebcce07 in testing::internal::UnitTestImpl::RunAllTests() out/gn-vptr/../../testing/gtest/src/gtest.cc:4647:43 #24 0xebcc1d8 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) out/gn-vptr/../../testing/gtest/src/gtest.cc:2458:12 #25 0xebcbfcc in testing::UnitTest::Run() out/gn-vptr/../../testing/gtest/src/gtest.cc:4255:10 #26 0x322075f in RUN_ALL_TESTS out/gn-vptr/../../testing/gtest/include/gtest/gtest.h:2237:46 #27 0x322075f in base::TestSuite::Run() out/gn-vptr/../../base/test/test_suite.cc:230 #28 0x3231301 in Run out/gn-vptr/../../base/callback.h:397:12 #29 0x3231301 in base::(anonymous namespace)::LaunchUnitTestsInternal(base::Callback<int (), (base::internal::CopyMode)1> const&, int, int, bool, base::Callback<void (), (base::internal::CopyMode)1> const&) out/gn-vptr/../../base/test/launcher/unit_test_launcher.cc:206 #30 0x32311b8 in base::LaunchUnitTests(int, char**, base::Callback<int (), (base::internal::CopyMode)1> const&) out/gn-vptr/../../base/test/launcher/unit_test_launcher.cc:445:10 #31 0x321504b in main out/gn-vptr/../../chrome/test/base/run_all_unittests.cc:21:10 #32 0x7fa53b2bbf44 in __libc_start_main /build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287 #33 0x79582c in _start (/usr/local/google/home/krasin/chr22/src/out/gn-vptr/unit_tests+0x79582c) [1/1] HotwordServiceTests/HotwordServiceTest.AudioHistorySyncOccurs/0 (CRASHED)