New issue
Advanced search Search tips

Issue 907371 link

Starred by 1 user
Status: Available
Owner: ----
Cc:
pwnall@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

ThreadSanitizer crashes on SQLite's mutex hold checks

Project Member Reported by pwnall@chromium.org, Nov 21 
When SQLITE_DEBUG is defined, SQLite checks for mutex acquisition. ThreadSanitizer correctly flags the pthreadMutexHeld / pthreadMutexUnheld functions [1] as racy.

While it'd be impossible to build a meaningful pthreadMutexUnheld, it is possible to obtain thread-safe versions of these functions using pthread_mutex_trylock, which should work for the purpose of SQLite's assertions.

We need thread-safe versions so we can enable SQLITE_DEBUG on CQ -- the current versions would result in flaky assertions, at least on ARM.

[1] https://cs.chromium.org/chromium/src/third_party/sqlite/src/src/mutex_unix.c?q=pthreadMutexHeld


Example TSAN output:

WARNING: ThreadSanitizer: data race (pid=23686)
  Read of size 4 at 0x55a91b900ecc by main thread (mutexes: write M587010635091870848):
    #0 pthreadMutexNotheld third_party/sqlite/amalgamation/sqlite3.c:25217:13 (unit_tests+0xdb76d45)
    #1 pthreadMutexEnter third_party/sqlite/amalgamation/sqlite3.c:25387 (unit_tests+0xdb76d45)
    #2 chrome_sqlite3_mutex_enter third_party/sqlite/amalgamation/sqlite3.c:24869:5 (unit_tests+0xda38401)
    #3 chrome_sqlite3_free third_party/sqlite/amalgamation/sqlite3.c:26542 (unit_tests+0xda38401)
    #4 sqlite3DbFreeNN third_party/sqlite/amalgamation/sqlite3.c:26588:3 (unit_tests+0xda61d55)
    #5 sqlite3DbFree third_party/sqlite/amalgamation/sqlite3.c:26592:11 (unit_tests+0xda373af)
    #6 sqlite3VdbeClearObject third_party/sqlite/amalgamation/sqlite3.c:78839 (unit_tests+0xda373af)
    #7 sqlite3VdbeDelete third_party/sqlite/amalgamation/sqlite3.c:78864:3 (unit_tests+0xda8cba4)
    #8 sqlite3VdbeFinalize third_party/sqlite/amalgamation/sqlite3.c:78779:3 (unit_tests+0xda46c60)
    #9 chrome_sqlite3_finalize third_party/sqlite/amalgamation/sqlite3.c:80662 (unit_tests+0xda46c60)
    #10 Close sql/database.cc:241:5 (unit_tests+0xda25716)
    #11 sql::Database::StatementRef::~StatementRef() sql/database.cc:228 (unit_tests+0xda25716)
    #12 DeleteInternal<sql::Database::StatementRef> base/memory/ref_counted.h:352:5 (unit_tests+0xda272b9)
    #13 Destruct base/memory/ref_counted.h:318 (unit_tests+0xda272b9)
    #14 Release base/memory/ref_counted.h:341 (unit_tests+0xda272b9)
    #15 Release base/memory/scoped_refptr.h:284 (unit_tests+0xda272b9)
    #16 ~scoped_refptr base/memory/scoped_refptr.h:208 (unit_tests+0xda272b9)
    #17 ~pair buildtools/third_party/libc++/trunk/include/utility:315 (unit_tests+0xda272b9)
    #18 destroy buildtools/third_party/libc++/trunk/include/memory:1867 (unit_tests+0xda272b9)
    #19 __destroy<std::__1::pair<sql::StatementID, scoped_refptr<sql::Database::StatementRef> > > buildtools/third_party/libc++/trunk/include/memory:1729 (unit_tests+0xda272b9)
    #20 destroy<std::__1::pair<sql::StatementID, scoped_refptr<sql::Database::StatementRef> > > buildtools/third_party/libc++/trunk/include/memory:1597 (unit_tests+0xda272b9)
    #21 __destruct_at_end buildtools/third_party/libc++/trunk/include/vector:422 (unit_tests+0xda272b9)
    #22 clear buildtools/third_party/libc++/trunk/include/vector:365 (unit_tests+0xda272b9)
    #23 clear buildtools/third_party/libc++/trunk/include/vector:768 (unit_tests+0xda272b9)
    #24 clear base/containers/flat_tree.h:588 (unit_tests+0xda272b9)
    #25 sql::Database::CloseInternal(bool) sql/database.cc:373 (unit_tests+0xda272b9)
    #26 Close sql/database.cc:424:3 (unit_tests+0xda25ba3)
    #27 sql::Database::~Database() sql/database.cc:279 (unit_tests+0xda25ba3)
    #28 ~InMemoryDatabase components/history/core/browser/in_memory_database.cc:19:1 (unit_tests+0xe861160)
    #29 history::InMemoryDatabase::~InMemoryDatabase() components/history/core/browser/in_memory_database.cc:18 (unit_tests+0xe861160)
    #30 operator() buildtools/third_party/libc++/trunk/include/memory:2325:5 (unit_tests+0xe8608ec)
    #31 reset buildtools/third_party/libc++/trunk/include/memory:2638 (unit_tests+0xe8608ec)
    #32 ~unique_ptr buildtools/third_party/libc++/trunk/include/memory:2592 (unit_tests+0xe8608ec)
    #33 ~InMemoryHistoryBackend components/history/core/browser/in_memory_history_backend.cc:24 (unit_tests+0xe8608ec)
    #34 history::InMemoryHistoryBackend::~InMemoryHistoryBackend() components/history/core/browser/in_memory_history_backend.cc:23 (unit_tests+0xe8608ec)
    #35 operator() buildtools/third_party/libc++/trunk/include/memory:2325:5 (unit_tests+0xe8182c9)
    #36 reset buildtools/third_party/libc++/trunk/include/memory:2638 (unit_tests+0xe8182c9)
    #37 history::HistoryService::Cleanup() components/history/core/browser/history_service.cc:895 (unit_tests+0xe8182c9)
    #38 Shutdown components/history/core/browser/history_service.cc:238:3 (unit_tests+0xe819042)
    #39 non-virtual thunk to history::HistoryService::Shutdown() components/history/core/browser/history_service.cc (unit_tests+0xe819042)
    #40 KeyedServiceFactory::ContextShutdown(base::SupportsUserData*) components/keyed_service/core/keyed_service_factory.cc:112:23 (unit_tests+0xe6731a4)
    #41 BrowserContextKeyedServiceFactory::BrowserContextShutdown(content::BrowserContext*) components/keyed_service/content/browser_context_keyed_service_factory.cc:91:24 (unit_tests+0xef089e0)
    #42 BrowserContextKeyedServiceFactory::ContextShutdown(base::SupportsUserData*) components/keyed_service/content/browser_context_keyed_service_factory.cc:125:3 (unit_tests+0xef08b7c)
    #43 DependencyManager::DestroyContextServices(base::SupportsUserData*) components/keyed_service/core/dependency_manager.cc:91:14 (unit_tests+0xe66e85e)
    #44 BrowserContextDependencyManager::DestroyBrowserContextServices(content::BrowserContext*) components/keyed_service/content/browser_context_dependency_manager.cc:52:22 (unit_tests+0xef07cf0)
    #45 TestingProfile::~TestingProfile() chrome/test/base/testing_profile.cc:510:40 (unit_tests+0xac473c6)
    #46 BookmarkHTMLWriterTest_Test_Test::TestBody() chrome/browser/bookmarks/bookmark_html_writer_unittest.cc:308:1 (unit_tests+0x4b22fb4)
    #47 HandleExceptionsInMethodIfSupported<testing::Test, void> third_party/googletest/src/googletest/src/gtest.cc (unit_tests+0x672f7af)
    #48 testing::Test::Run() third_party/googletest/src/googletest/src/gtest.cc:2522 (unit_tests+0x672f7af)
    #49 testing::TestInfo::Run() third_party/googletest/src/googletest/src/gtest.cc:2703:11 (unit_tests+0x67309c8)
    #50 testing::TestCase::Run() third_party/googletest/src/googletest/src/gtest.cc:2825:28 (unit_tests+0x6731256)
    #51 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5227:43 (unit_tests+0x6742836)
    #52 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/src/googletest/src/gtest.cc (unit_tests+0x674211a)
    #53 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:4835 (unit_tests+0x674211a)
    #54 RUN_ALL_TESTS third_party/googletest/src/googletest/include/gtest/gtest.h:2369:46 (unit_tests+0xac5fea5)
    #55 base::TestSuite::Run() base/test/test_suite.cc:294 (unit_tests+0xac5fea5)
    #56 content::UnitTestTestSuite::Run() content/public/test/unittest_test_suite.cc:64:23 (unit_tests+0xada64e1)
    #57 Invoke<int (content::UnitTestTestSuite::*)(), content::UnitTestTestSuite *> base/bind_internal.h:516:12 (unit_tests+0xac4ec28)
    #58 MakeItSo<int (content::UnitTestTestSuite::*const &)(), content::UnitTestTestSuite *> base/bind_internal.h:616 (unit_tests+0xac4ec28)
    #59 RunImpl<int (content::UnitTestTestSuite::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::UnitTestTestSuite> > &, 0> base/bind_internal.h:689 (unit_tests+0xac4ec28)
    #60 base::internal::Invoker<base::internal::BindState<int (content::UnitTestTestSuite::*)(), base::internal::UnretainedWrapper<content::UnitTestTestSuite> >, int ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:671 (unit_tests+0xac4ec28)
    #61 Run base/callback.h:99:12 (unit_tests+0xac63c88)
    #62 base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned long, int, bool, base::OnceCallback<void ()>) base/test/launcher/unit_test_launcher.cc:225 (unit_tests+0xac63c88)
    #63 base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>) base/test/launcher/unit_test_launcher.cc:575:10 (unit_tests+0xac63a97)
    #64 main chrome/test/base/run_all_unittests.cc:30:10 (unit_tests+0xac4eb54)

  Previous write of size 4 at 0x55a91b900ecc by thread T23 (mutexes: write M596017834346506368, write M548688914409328288):
    #0 pthreadMutexEnter third_party/sqlite/amalgamation/sqlite3.c:25418:10 (unit_tests+0xdb76db2)
    #1 chrome_sqlite3_mutex_enter third_party/sqlite/amalgamation/sqlite3.c:24869:5 (unit_tests+0xda38401)
    #2 chrome_sqlite3_free third_party/sqlite/amalgamation/sqlite3.c:26542 (unit_tests+0xda38401)
    #3 sqlite3DbFreeNN third_party/sqlite/amalgamation/sqlite3.c:26588:3 (unit_tests+0xda61d55)
    #4 vdbeMemClear third_party/sqlite/amalgamation/sqlite3.c:74280:5 (unit_tests+0xda913b0)
    #5 sqlite3VdbeMemRelease third_party/sqlite/amalgamation/sqlite3.c:74299:5 (unit_tests+0xda474f9)
    #6 chrome_sqlite3_clear_bindings third_party/sqlite/amalgamation/sqlite3.c:80707 (unit_tests+0xda474f9)
    #7 sql::Statement::Reset(bool) sql/statement.cc:104:7 (unit_tests+0xda31baf)
    #8 sql::Statement::~Statement() sql/statement.cc:35:3 (unit_tests+0xda319ee)
    #9 history::ThumbnailDatabase::GetIconMappingsForPageURL(GURL const&, std::__1::vector<history::IconMapping, std::__1::allocator<history::IconMapping> >*) components/history/core/browser/thumbnail_database.cc:815:1 (unit_tests+0xe876d25)
    #10 history::HistoryBackend::SetFaviconMappingsForPage(GURL const&, favicon_base::IconType, long) components/history/core/browser/history_backend.cc:2280:18 (unit_tests+0xe84267e)
    #11 SetFaviconMappingsForPages components/history/core/browser/history_backend.cc:2259:9 (unit_tests+0xe83f6d4)
    #12 history::HistoryBackend::SetFaviconMappingsForPageAndRedirects(GURL const&, favicon_base::IconType, long) components/history/core/browser/history_backend.cc:2247 (unit_tests+0xe83f6d4)
    #13 history::HistoryBackend::SetFaviconsImpl(base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> > const&, favicon_base::IconType, GURL const&, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > const&, history::FaviconBitmapType) components/history/core/browser/history_backend.cc:2000:9 (unit_tests+0xe840870)
    #14 history::HistoryBackend::SetFavicons(base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> > const&, favicon_base::IconType, GURL const&, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > const&) components/history/core/browser/history_backend.cc:1804:3 (unit_tests+0xe84061c)
    #15 Invoke<void (history::HistoryBackend::*)(const base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> > &, favicon_base::IconType, const GURL &, const std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > &), scoped_refptr<history::HistoryBackend>, base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> >, favicon_base::IconType, GURL, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > > base/bind_internal.h:516:12 (unit_tests+0xe82bc63)
    #16 MakeItSo<void (history::HistoryBackend::*)(const base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> > &, favicon_base::IconType, const GURL &, const std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > &), scoped_refptr<history::HistoryBackend>, base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> >, favicon_base::IconType, GURL, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > > base/bind_internal.h:616 (unit_tests+0xe82bc63)
    #17 RunImpl<void (history::HistoryBackend::*)(const base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> > &, favicon_base::IconType, const GURL &, const std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > &), std::__1::tuple<scoped_refptr<history::HistoryBackend>, base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> >, favicon_base::IconType, GURL, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > >, 0, 1, 2, 3, 4> base/bind_internal.h:689 (unit_tests+0xe82bc63)
    #18 base::internal::Invoker<base::internal::BindState<void (history::HistoryBackend::*)(base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> > const&, favicon_base::IconType, GURL const&, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > const&), scoped_refptr<history::HistoryBackend>, base::internal::flat_tree<GURL, GURL, base::internal::GetKeyFromValueIdentity<GURL>, std::__1::less<void> >, favicon_base::IconType, GURL, std::__1::vector<SkBitmap, std::__1::allocator<SkBitmap> > >, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:658 (unit_tests+0xe82bc63)
    #19 Run base/callback.h:99:12 (unit_tests+0xbcd9f2d)
    #20 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:99 (unit_tests+0xbcd9f2d)
    #21 base::MessageLoopImpl::RunTask(base::PendingTask*) base/message_loop/message_loop_impl.cc:462:46 (unit_tests+0xbcd7f84)
    #22 DeferOrRunPendingTask base/message_loop/message_loop_impl.cc:473:5 (unit_tests+0xbcd88ab)
    #23 base::MessageLoopImpl::DoWork() base/message_loop/message_loop_impl.cc:561 (unit_tests+0xbcd88ab)
    #24 non-virtual thunk to base::MessageLoopImpl::DoWork() base/message_loop/message_loop_impl.cc (unit_tests+0xbcd88ed)
    #25 base::MessagePumpDefault::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_default.cc:39:31 (unit_tests+0xbcdcdc6)
    #26 base::MessageLoopImpl::Run(bool) base/message_loop/message_loop_impl.cc:414:12 (unit_tests+0xbcd7838)
    #27 non-virtual thunk to base::MessageLoopImpl::Run(bool) base/message_loop/message_loop_impl.cc (unit_tests+0xbcd7934)
    #28 base::RunLoop::Run() base/run_loop.cc:102:14 (unit_tests+0xbd1f6ff)
    #29 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:250:13 (unit_tests+0xbdab9cb)
    #30 base::Thread::ThreadMain() base/threading/thread.cc:332:3 (unit_tests+0xbdabe6c)
    #31 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:81:13 (unit_tests+0xbe13c41)

  Location is global 'pthreadMutexAlloc.staticMutexes' of size 768 at 0x55a91b900e60 (unit_tests+0x00001409becc)

  Mutex M587010635091870848 is already destroyed.

  Mutex M596017834346506368 is already destroyed.

  Mutex M548688914409328288 is already destroyed.

  Thread T23 'Chrome_HistoryThread' (tid=23824, running) created by main thread at:
    #0 pthread_create /b/swarming/w/ir/kitchen-workdir/src/third_party/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:968:3 (unit_tests+0x3f5b8b5)
    #1 base::(anonymous namespace)::CreateThread(unsigned long, bool, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) base/threading/platform_thread_posix.cc:120:13 (unit_tests+0xbe13676)
    #2 base::PlatformThread::CreateWithPriority(unsigned long, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) base/threading/platform_thread_posix.cc:238:10 (unit_tests+0xbe13535)
    #3 base::Thread::StartWithOptions(base::Thread::Options const&) base/threading/thread.cc:112:15 (unit_tests+0xbdab015)
    #4 history::HistoryService::Init(bool, history::HistoryDatabaseParams const&) components/history/core/browser/history_service.cc:946:21 (unit_tests+0xe8205e0)
    #5 TestingProfile::CreateHistoryService(bool, bool) chrome/test/base/testing_profile.cc:553:25 (unit_tests+0xac47a94)
    #6 BookmarkHTMLWriterTest_Test_Test::TestBody() chrome/browser/bookmarks/bookmark_html_writer_unittest.cc:155:3 (unit_tests+0x4b21346)
    #7 HandleExceptionsInMethodIfSupported<testing::Test, void> third_party/googletest/src/googletest/src/gtest.cc (unit_tests+0x672f7af)
    #8 testing::Test::Run() third_party/googletest/src/googletest/src/gtest.cc:2522 (unit_tests+0x672f7af)
    #9 testing::TestInfo::Run() third_party/googletest/src/googletest/src/gtest.cc:2703:11 (unit_tests+0x67309c8)
    #10 testing::TestCase::Run() third_party/googletest/src/googletest/src/gtest.cc:2825:28 (unit_tests+0x6731256)
    #11 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5227:43 (unit_tests+0x6742836)
    #12 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/src/googletest/src/gtest.cc (unit_tests+0x674211a)
    #13 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:4835 (unit_tests+0x674211a)
    #14 RUN_ALL_TESTS third_party/googletest/src/googletest/include/gtest/gtest.h:2369:46 (unit_tests+0xac5fea5)
    #15 base::TestSuite::Run() base/test/test_suite.cc:294 (unit_tests+0xac5fea5)
    #16 content::UnitTestTestSuite::Run() content/public/test/unittest_test_suite.cc:64:23 (unit_tests+0xada64e1)
    #17 Invoke<int (content::UnitTestTestSuite::*)(), content::UnitTestTestSuite *> base/bind_internal.h:516:12 (unit_tests+0xac4ec28)
    #18 MakeItSo<int (content::UnitTestTestSuite::*const &)(), content::UnitTestTestSuite *> base/bind_internal.h:616 (unit_tests+0xac4ec28)
    #19 RunImpl<int (content::UnitTestTestSuite::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::UnitTestTestSuite> > &, 0> base/bind_internal.h:689 (unit_tests+0xac4ec28)
    #20 base::internal::Invoker<base::internal::BindState<int (content::UnitTestTestSuite::*)(), base::internal::UnretainedWrapper<content::UnitTestTestSuite> >, int ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:671 (unit_tests+0xac4ec28)
    #21 Run base/callback.h:99:12 (unit_tests+0xac63c88)
    #22 base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned long, int, bool, base::OnceCallback<void ()>) base/test/launcher/unit_test_launcher.cc:225 (unit_tests+0xac63c88)
    #23 base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>) base/test/launcher/unit_test_launcher.cc:575:10 (unit_tests+0xac63a97)
    #24 main chrome/test/base/run_all_unittests.cc:30:10 (unit_tests+0xac4eb54)

SUMMARY: ThreadSanitizer: data race third_party/sqlite/amalgamation/sqlite3.c:25217:13 in pthreadMutexNotheld

Comment 1 by pwnall@chromium.org, Nov 21

Description: Show this description

Comment 2 by pwnall@chromium.org, Nov 29

Cc: pwnall@chromium.org
Labels: -Pri-1 Pri-2
Owner: ----
Status: Available (was: Started)
We were able to diagnose the current SQLite crash without enabling SQLITE's asserts, so the issue is not urgent anymore.
►

Sign in to add a comment