New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 919945 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 917380
Owner:
Closed: Jan 11
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

False rejects caused by TSAN sqlite races in sync_integration_tests

Project Member Reported by erikc...@chromium.org, Jan 8

Issue description

Test:
SingleClientWalletSyncTest.DownloadAccountStorage_Card

Example build:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_tsan_rel_ng/174770

Race:
"""
WARNING: ThreadSanitizer: data race (pid=28756)
  Write of size 4 at 0x562fe224ba40 by thread T22 (mutexes: write M98489):
    #0 pcache1FetchStage2 third_party/sqlite/amalgamation/sqlite3.c (sync_integration_tests+0xaec46ce)
    #1 pcache1FetchNoMutex third_party/sqlite/amalgamation/sqlite3.c:49514:12 (sync_integration_tests+0xaec2dd5)
    #2 pcache1Fetch third_party/sqlite/amalgamation/sqlite3.c:49556 (sync_integration_tests+0xaec2dd5)
    #3 sqlite3PcacheFetch third_party/sqlite/amalgamation/sqlite3.c:48011:10 (sync_integration_tests+0xad7ab2e)
    #4 getPageNormal third_party/sqlite/amalgamation/sqlite3.c:55899 (sync_integration_tests+0xad7ab2e)
    #5 sqlite3PagerGet third_party/sqlite/amalgamation/sqlite3.c:56078:10 (sync_integration_tests+0xad8b865)
    #6 btreeGetPage third_party/sqlite/amalgamation/sqlite3.c:65059 (sync_integration_tests+0xad8b865)
    #7 btreeGetUnusedPage third_party/sqlite/amalgamation/sqlite3.c:65203 (sync_integration_tests+0xad8b865)
    #8 allocateBtreePage third_party/sqlite/amalgamation/sqlite3.c:69099 (sync_integration_tests+0xad8b865)
    #9 btreeCreateTable third_party/sqlite/amalgamation/sqlite3.c:71903:10 (sync_integration_tests+0xadb548e)
    #10 sqlite3BtreeCreateTable third_party/sqlite/amalgamation/sqlite3.c:71922 (sync_integration_tests+0xadb548e)
    #11 sqlite3VdbeExec third_party/sqlite/amalgamation/sqlite3.c:88755:8 (sync_integration_tests+0xada4807)
    #12 sqlite3Step third_party/sqlite/amalgamation/sqlite3.c:81415:10 (sync_integration_tests+0xad54af8)
    #13 chrome_sqlite3_step third_party/sqlite/amalgamation/sqlite3.c:81478 (sync_integration_tests+0xad54af8)
    #14 sql::Database::ExecuteAndReturnErrorCode(char const*) sql/database.cc:1304:18 (sync_integration_tests+0xad36db9)
    #15 sql::Database::Execute(char const*) sql/database.cc:1341:15 (sync_integration_tests+0xad337d6)
    #16 autofill::AutofillTable::InitMainTable() components/autofill/core/browser/webdata/autofill_table.cc:2843:15 (sync_integration_tests+0xc0d8cf6)
    #17 autofill::AutofillTable::CreateTablesIfNecessary() components/autofill/core/browser/webdata/autofill_table.cc:408:11 (sync_integration_tests+0xc0d878e)
    #18 WebDatabase::Init(base::FilePath const&) components/webdata/common/web_database.cc:136:22 (sync_integration_tests+0xbc029cd)
    #19 WebDatabaseBackend::LoadDatabaseIfNecessary() components/webdata/common/web_database_backend.cc:110:23 (sync_integration_tests+0xbc06215)
    #20 WebDatabaseBackend::InitDatabase() components/webdata/common/web_database_backend.cc:38:3 (sync_integration_tests+0xbc05f8d)
    #21 Invoke<void (WebDatabaseBackend::*)(), scoped_refptr<WebDatabaseBackend>> base/bind_internal.h:516:12 (sync_integration_tests+0xbc04fb0)
    #22 MakeItSo<void (WebDatabaseBackend::*)(), scoped_refptr<WebDatabaseBackend> > base/bind_internal.h:616 (sync_integration_tests+0xbc04fb0)
    #23 RunImpl<void (WebDatabaseBackend::*)(), std::__1::tuple<scoped_refptr<WebDatabaseBackend> >, 0> base/bind_internal.h:689 (sync_integration_tests+0xbc04fb0)
    #24 base::internal::Invoker<base::internal::BindState<void (WebDatabaseBackend::*)(), scoped_refptr<WebDatabaseBackend> >, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:658 (sync_integration_tests+0xbc04fb0)
    #25 Run base/callback.h:99:12 (sync_integration_tests+0x86cc06d)
    #26 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:99 (sync_integration_tests+0x86cc06d)
    #27 base::internal::TaskTracker::RunBlockShutdown(base::internal::Task*) base/task/task_scheduler/task_tracker.cc:950:19 (sync_integration_tests+0x878ddbc)
    #28 RunTaskWithShutdownBehavior base/task/task_scheduler/task_tracker.cc:965:7 (sync_integration_tests+0x878d302)
    #29 base::internal::TaskTracker::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, base::TaskTraits const&, bool) base/task/task_scheduler/task_tracker.cc:649 (sync_integration_tests+0x878d302)
    #30 base::internal::TaskTrackerPosix::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, base::TaskTraits const&, bool) base/task/task_scheduler/task_tracker_posix.cc:24:16 (sync_integration_tests+0x880d334)
    #31 base::internal::TaskTracker::RunAndPopNextTask(scoped_refptr<base::internal::Sequence>, base::internal::CanScheduleSequenceObserver*) base/task/task_scheduler/task_tracker.cc:505:3 (sync_integration_tests+0x878bfcd)
    #32 base::internal::SchedulerWorker::RunWorker() base/task/task_scheduler/scheduler_worker.cc:337:24 (sync_integration_tests+0x87859a5)
    #33 base::internal::SchedulerWorker::RunSharedWorker() base/task/task_scheduler/scheduler_worker.cc:241:3 (sync_integration_tests+0x8785531)
    #34 base::internal::SchedulerWorker::ThreadMain() base/task/task_scheduler/scheduler_worker.cc:211:7 (sync_integration_tests+0x8785395)
    #35 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:81:13 (sync_integration_tests+0x880e131)

  Previous write of size 4 at 0x562fe224ba40 by thread T29 (mutexes: write M98547):
    #0 pcache1FetchStage2 third_party/sqlite/amalgamation/sqlite3.c (sync_integration_tests+0xaec46ce)
    #1 pcache1FetchNoMutex third_party/sqlite/amalgamation/sqlite3.c:49514:12 (sync_integration_tests+0xaec2dd5)
    #2 pcache1Fetch third_party/sqlite/amalgamation/sqlite3.c:49556 (sync_integration_tests+0xaec2dd5)
    #3 sqlite3PcacheFetch third_party/sqlite/amalgamation/sqlite3.c:48011:10 (sync_integration_tests+0xad7ab2e)
    #4 getPageNormal third_party/sqlite/amalgamation/sqlite3.c:55899 (sync_integration_tests+0xad7ab2e)
    #5 sqlite3PagerGet third_party/sqlite/amalgamation/sqlite3.c:56078:10 (sync_integration_tests+0xad8b865)
    #6 btreeGetPage third_party/sqlite/amalgamation/sqlite3.c:65059 (sync_integration_tests+0xad8b865)
    #7 btreeGetUnusedPage third_party/sqlite/amalgamation/sqlite3.c:65203 (sync_integration_tests+0xad8b865)
    #8 allocateBtreePage third_party/sqlite/amalgamation/sqlite3.c:69099 (sync_integration_tests+0xad8b865)
    #9 btreeCreateTable third_party/sqlite/amalgamation/sqlite3.c:71825:10 (sync_integration_tests+0xadb535d)
    #10 sqlite3BtreeCreateTable third_party/sqlite/amalgamation/sqlite3.c:71922 (sync_integration_tests+0xadb535d)
    #11 sqlite3VdbeExec third_party/sqlite/amalgamation/sqlite3.c:88755:8 (sync_integration_tests+0xada4807)
    #12 sqlite3Step third_party/sqlite/amalgamation/sqlite3.c:81415:10 (sync_integration_tests+0xad54af8)
    #13 chrome_sqlite3_step third_party/sqlite/amalgamation/sqlite3.c:81478 (sync_integration_tests+0xad54af8)
    #14 sql::Database::ExecuteAndReturnErrorCode(char const*) sql/database.cc:1304:18 (sync_integration_tests+0xad36db9)
    #15 sql::Database::Execute(char const*) sql/database.cc:1341:15 (sync_integration_tests+0xad337d6)
    #16 history::URLDatabase::CreateKeywordSearchTermsIndices() components/history/core/browser/url_database.cc:441:16 (sync_integration_tests+0xbdb4989)
    #17 history::InMemoryDatabase::InitFromDisk(base::FilePath const&) components/history/core/browser/in_memory_database.cc:156:3 (sync_integration_tests+0xbd85f71)
    #18 history::InMemoryHistoryBackend::Init(base::FilePath const&) components/history/core/browser/in_memory_history_backend.cc:28:15 (sync_integration_tests+0xbd84fa9)
    #19 history::HistoryBackend::InitImpl(history::HistoryDatabaseParams const&) components/history/core/browser/history_backend.cc:721:22 (sync_integration_tests+0xbd5a5a0)
    #20 history::HistoryBackend::Init(bool, history::HistoryDatabaseParams const&) components/history/core/browser/history_backend.cc:242:5 (sync_integration_tests+0xbd59fad)
    #21 Invoke<void (history::HistoryBackend::*)(bool, const history::HistoryDatabaseParams &), scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams> base/bind_internal.h:516:12 (sync_integration_tests+0xbd56ec9)
    #22 MakeItSo<void (history::HistoryBackend::*)(bool, const history::HistoryDatabaseParams &), scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams> base/bind_internal.h:616 (sync_integration_tests+0xbd56ec9)
    #23 RunImpl<void (history::HistoryBackend::*)(bool, const history::HistoryDatabaseParams &), std::__1::tuple<scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams>, 0, 1, 2> base/bind_internal.h:689 (sync_integration_tests+0xbd56ec9)
    #24 base::internal::Invoker<base::internal::BindState<void (history::HistoryBackend::*)(bool, history::HistoryDatabaseParams const&), scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams>, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:658 (sync_integration_tests+0xbd56ec9)
    #25 Run base/callback.h:99:12 (sync_integration_tests+0x86cc06d)
    #26 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:99 (sync_integration_tests+0x86cc06d)
    #27 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::TimeTicks*) base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:244:21 (sync_integration_tests+0x87635e7)
    #28 DoWork base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:214:10 (sync_integration_tests+0x8763cb9)
    #29 non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() base/task/sequence_manager/thread_controller_with_message_pump_impl.cc (sync_integration_tests+0x8763cb9)
    #30 base::MessagePumpDefault::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_default.cc:39:31 (sync_integration_tests+0x86cee09)
    #31 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool) base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:342:12 (sync_integration_tests+0x876409a)
    #32 non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool) base/task/sequence_manager/thread_controller_with_message_pump_impl.cc (sync_integration_tests+0x87641d4)
    #33 base::RunLoop::Run() base/run_loop.cc:102:14 (sync_integration_tests+0x871368f)
    #34 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:251:13 (sync_integration_tests+0x879fe0b)
    #35 base::Thread::ThreadMain() base/threading/thread.cc:333:3 (sync_integration_tests+0x87a02e3)
    #36 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:81:13 (sync_integration_tests+0x880e131)

  Location is global 'pcache1Create.dummyCurrentPage' of size 4 at 0x562fe224ba40 (sync_integration_tests+0x000012ab1a40)

  Mutex M98489 (0x7b0c000fe8b0) created at:
    #0 pthread_mutex_init /b/swarming/w/ir/kitchen-workdir/src/third_party/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1228:3 (sync_integration_tests+0x37c7c43)
    #1 pthreadMutexAlloc third_party/sqlite/amalgamation/sqlite3.c:25487:9 (sync_integration_tests+0xaeb1deb)
    #2 sqlite3MutexAlloc third_party/sqlite/amalgamation/sqlite3.c:25019:10 (sync_integration_tests+0xad6701b)
    #3 openDatabase third_party/sqlite/amalgamation/sqlite3.c:155439 (sync_integration_tests+0xad6701b)
    #4 chrome_sqlite3_open_v2 third_party/sqlite/amalgamation/sqlite3.c:155754:10 (sync_integration_tests+0xad688c0)
    #5 sql::Database::OpenInternal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, sql::Database::Retry) sql/database.cc:1597:13 (sync_integration_tests+0xad31bab)
    #6 sql::Database::OpenInMemory() sql/database.cc:358:10 (sync_integration_tests+0xad32349)
    #7 WebDatabase::Init(base::FilePath const&) components/webdata/common/web_database.cc:95:49 (sync_integration_tests+0xbc0288f)
    #8 WebDatabaseBackend::LoadDatabaseIfNecessary() components/webdata/common/web_database_backend.cc:110:23 (sync_integration_tests+0xbc06215)
    #9 WebDatabaseBackend::InitDatabase() components/webdata/common/web_database_backend.cc:38:3 (sync_integration_tests+0xbc05f8d)
    #10 Invoke<void (WebDatabaseBackend::*)(), scoped_refptr<WebDatabaseBackend>> base/bind_internal.h:516:12 (sync_integration_tests+0xbc04fb0)
    #11 MakeItSo<void (WebDatabaseBackend::*)(), scoped_refptr<WebDatabaseBackend> > base/bind_internal.h:616 (sync_integration_tests+0xbc04fb0)
    #12 RunImpl<void (WebDatabaseBackend::*)(), std::__1::tuple<scoped_refptr<WebDatabaseBackend> >, 0> base/bind_internal.h:689 (sync_integration_tests+0xbc04fb0)
    #13 base::internal::Invoker<base::internal::BindState<void (WebDatabaseBackend::*)(), scoped_refptr<WebDatabaseBackend> >, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:658 (sync_integration_tests+0xbc04fb0)
    #14 Run base/callback.h:99:12 (sync_integration_tests+0x86cc06d)
    #15 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:99 (sync_integration_tests+0x86cc06d)
    #16 base::internal::TaskTracker::RunBlockShutdown(base::internal::Task*) base/task/task_scheduler/task_tracker.cc:950:19 (sync_integration_tests+0x878ddbc)
    #17 RunTaskWithShutdownBehavior base/task/task_scheduler/task_tracker.cc:965:7 (sync_integration_tests+0x878d302)
    #18 base::internal::TaskTracker::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, base::TaskTraits const&, bool) base/task/task_scheduler/task_tracker.cc:649 (sync_integration_tests+0x878d302)
    #19 base::internal::TaskTrackerPosix::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, base::TaskTraits const&, bool) base/task/task_scheduler/task_tracker_posix.cc:24:16 (sync_integration_tests+0x880d334)
    #20 base::internal::TaskTracker::RunAndPopNextTask(scoped_refptr<base::internal::Sequence>, base::internal::CanScheduleSequenceObserver*) base/task/task_scheduler/task_tracker.cc:505:3 (sync_integration_tests+0x878bfcd)
    #21 base::internal::SchedulerWorker::RunWorker() base/task/task_scheduler/scheduler_worker.cc:337:24 (sync_integration_tests+0x87859a5)
    #22 base::internal::SchedulerWorker::RunSharedWorker() base/task/task_scheduler/scheduler_worker.cc:241:3 (sync_integration_tests+0x8785531)
    #23 base::internal::SchedulerWorker::ThreadMain() base/task/task_scheduler/scheduler_worker.cc:211:7 (sync_integration_tests+0x8785395)
    #24 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:81:13 (sync_integration_tests+0x880e131)
"""

I built tip of tree on Linux with TSAN and was able to repro this race as well [>10% frequency]. This race appears to be quite rare when the test is run as part of the whole suite of tests:
https://findit-for-me.appspot.com/flake/occurrences?key=ag9zfmZpbmRpdC1mb3ItbWVyYQsSBUZsYWtlIlZjaHJvbWl1bUBzeW5jX2ludGVncmF0aW9uX3Rlc3RzQFNpbmdsZUNsaWVudFdhbGxldFN5bmNUZXN0LkRvd25sb2FkQWNjb3VudFN0b3JhZ2VfQ2FyZAw

This suggests that state is inappropriately carried between tests for sync_integration_tests.
 
Owner: jkrcal@chromium.org
Status: Assigned (was: Untriaged)
Over to jkrcal who's looking into this. Time to dedupe some of these bugs?
Components: Services>Sync
Labels: Stability-ThreadSanitizer
Mergedinto: 917380
Status: Duplicate (was: Assigned)
This is being looked at. If it does not get fixed until Monday, I'll find a way to disable all sync_integration_tests on TSan.

Sign in to add a comment