interactive_ui_tests fail on tsan bot during roll attempt |
||||
Issue descriptionOn clang roll try job: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_tsan_rel_ng/162443 nteractive_ui_tests (retry summary) Failed with patch, succeeded without patch: OmniboxViewTest.CutURLToClipboard InstantExtendedTest.NavigateBackToNTP PopupBlockerBrowserTest.UnblockedPopupShowsInHistoryAndOmnibox OmniboxViewViewsTest.SelectionClipboard OmniboxViewViewsTest.DoNotNavigateOnDrop OmniboxViewViewsTest.SelectAllOnTap OmniboxViewTest.AltEnter OmniboxViewTest.FocusSearchLongUrl InstantExtendedTest.Referrer Tests ignored as they also fail without patch: OmniboxViewTest.CopyTextToClipboard OmniboxApiTest.PopupStaysClosed One stack: [ RUN ] OmniboxViewTest.FocusSearchLongUrl Xlib: extension "RANDR" missing on display ":99". [19276:19276:1210/135502.207934:ERROR:browser_dm_token_storage_linux.cc(101)] Error: /etc/machine-id contains 0 characters (32 were expected). [19276:19309:1210/135502.638567:ERROR:bus.cc(396)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix") [19276:19276:1210/135502.727056:WARNING:password_store_factory.cc(261)] Using basic (unencrypted) store for password storage. See https://chromium.googlesource.com/chromium/src/+/master/docs/linux_password_storage.md for more information about password storage options. (interactive_ui_tests:19276): LIBDBUSMENU-GLIB-WARNING **: Unable to get session bus: Unknown or unsupported transport 'disabled' for address 'disabled:' [19276:19276:1210/135504.094560:WARNING:gaia_auth_fetcher.cc(891)] Could not reach Google Accounts servers: errno -11 ================== WARNING: ThreadSanitizer: data race (pid=19276) Write of size 4 at 0x56051a90b610 by thread T25 (mutexes: write M990364259539016336): #0 pcache1FreePage third_party/sqlite/amalgamation/sqlite3.c:48979:25 (interactive_ui_tests+0xb3ba209) #1 pcache1TruncateUnsafe third_party/sqlite/amalgamation/sqlite3.c:49173 (interactive_ui_tests+0xb3ba209) #2 pcache1Truncate third_party/sqlite/amalgamation/sqlite3.c:49645 (interactive_ui_tests+0xb3ba209) #3 sqlite3PcacheTruncate third_party/sqlite/amalgamation/sqlite3.c:48292:5 (interactive_ui_tests+0xb27777b) #4 sqlite3PcacheClear third_party/sqlite/amalgamation/sqlite3.c:48309 (interactive_ui_tests+0xb27777b) #5 pager_reset third_party/sqlite/amalgamation/sqlite3.c:52205 (interactive_ui_tests+0xb27777b) #6 sqlite3PagerClose third_party/sqlite/amalgamation/sqlite3.c:54607:3 (interactive_ui_tests+0xb26976c) #7 sqlite3BtreeClose third_party/sqlite/amalgamation/sqlite3.c:65685:5 (interactive_ui_tests+0xb28af76) #8 sqlite3VdbeFreeCursor third_party/sqlite/amalgamation/sqlite3.c:78109:25 (interactive_ui_tests+0xb28ad7e) #9 closeCursorsInFrame third_party/sqlite/amalgamation/sqlite3.c:78140:9 (interactive_ui_tests+0xb286d70) #10 closeAllCursors third_party/sqlite/amalgamation/sqlite3.c:78190 (interactive_ui_tests+0xb286d70) #11 sqlite3VdbeHalt third_party/sqlite/amalgamation/sqlite3.c:78677 (interactive_ui_tests+0xb286d70) #12 sqlite3VdbeExec third_party/sqlite/amalgamation/sqlite3.c:84072:8 (interactive_ui_tests+0xb2aacea) #13 sqlite3Step third_party/sqlite/amalgamation/sqlite3.c:81415:10 (interactive_ui_tests+0xb24ba28) #14 chrome_sqlite3_step third_party/sqlite/amalgamation/sqlite3.c:81478 (interactive_ui_tests+0xb24ba28) #15 sql::Statement::StepInternal(bool) sql/statement.cc:71:13 (interactive_ui_tests+0xb233f6a) #16 sql::Statement::Step() sql/statement.cc:97:10 (interactive_ui_tests+0xb23452e) #17 history::URLDatabase::AutocompleteForPrefix(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool, std::__1::vector<history::URLRow, std::__1::allocator<history::URLRow> >*) components/history/core/browser/url_database.cc:309:20 (interactive_ui_tests+0xc06d90c) #18 HistoryURLProvider::DoAutocomplete(history::HistoryBackend*, history::URLDatabase*, HistoryURLProviderParams*) components/omnibox/browser/history_url_provider.cc:767:11 (interactive_ui_tests+0xd1f8c0a) #19 HistoryURLProvider::ExecuteWithDB(HistoryURLProviderParams*, history::HistoryBackend*, history::URLDatabase*) components/omnibox/browser/history_url_provider.cc:685:5 (interactive_ui_tests+0xd1f9729) #20 Invoke<void (HistoryURLProvider::*)(HistoryURLProviderParams *, history::HistoryBackend *, history::URLDatabase *), const scoped_refptr<HistoryURLProvider> &, HistoryURLProviderParams *const &, history::HistoryBackend *, history::URLDatabase *> base/bind_internal.h:516:12 (interactive_ui_tests+0xd2036ff) #21 MakeItSo<void (HistoryURLProvider::*const &)(HistoryURLProviderParams *, history::HistoryBackend *, history::URLDatabase *), const scoped_refptr<HistoryURLProvider> &, HistoryURLProviderParams *const &, history::HistoryBackend *, history::URLDatabase *> base/bind_internal.h:616 (interactive_ui_tests+0xd2036ff) #22 RunImpl<void (HistoryURLProvider::*const &)(HistoryURLProviderParams *, history::HistoryBackend *, history::URLDatabase *), const std::__1::tuple<scoped_refptr<HistoryURLProvider>, HistoryURLProviderParams *> &, 0, 1> base/bind_internal.h:689 (interactive_ui_tests+0xd2036ff) #23 base::internal::Invoker<base::internal::BindState<void (HistoryURLProvider::*)(HistoryURLProviderParams*, history::HistoryBackend*, history::URLDatabase*), scoped_refptr<HistoryURLProvider>, HistoryURLProviderParams*>, void (history::HistoryBackend*, history::URLDatabase*)>::Run(base::internal::BindStateBase*, history::HistoryBackend*, history::URLDatabase*) base/bind_internal.h:671 (interactive_ui_tests+0xd2036ff) #24 Run base/callback.h:129:12 (interactive_ui_tests+0xc015716) #25 history::HistoryBackend::ScheduleAutocomplete(base::RepeatingCallback<void (history::HistoryBackend*, history::URLDatabase*)> const&) components/history/core/browser/history_backend.cc:1457 (interactive_ui_tests+0xc015716) #26 Invoke<void (history::HistoryBackend::*)(const base::RepeatingCallback<void (history::HistoryBackend *, history::URLDatabase *)> &), scoped_refptr<history::HistoryBackend>, base::RepeatingCallback<void (history::HistoryBackend *, history::URLDatabase *)> > base/bind_internal.h:516:12 (interactive_ui_tests+0xc00a4a9) #27 MakeItSo<void (history::HistoryBackend::*)(const base::RepeatingCallback<void (history::HistoryBackend *, history::URLDatabase *)> &), scoped_refptr<history::HistoryBackend>, base::RepeatingCallback<void (history::HistoryBackend *, history::URLDatabase *)> > base/bind_internal.h:616 (interactive_ui_tests+0xc00a4a9) #28 RunImpl<void (history::HistoryBackend::*)(const base::RepeatingCallback<void (history::HistoryBackend *, history::URLDatabase *)> &), std::__1::tuple<scoped_refptr<history::HistoryBackend>, base::RepeatingCallback<void (history::HistoryBackend *, history::URLDatabase *)> >, 0, 1> base/bind_internal.h:689 (interactive_ui_tests+0xc00a4a9) #29 base::internal::Invoker<base::internal::BindState<void (history::HistoryBackend::*)(base::RepeatingCallback<void (history::HistoryBackend*, history::URLDatabase*)> const&), scoped_refptr<history::HistoryBackend>, base::RepeatingCallback<void (history::HistoryBackend*, history::URLDatabase*)> >, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:658 (interactive_ui_tests+0xc00a4a9) #30 Run base/callback.h:99:12 (interactive_ui_tests+0x9a7795d) #31 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:99 (interactive_ui_tests+0x9a7795d) #32 base::MessageLoopImpl::RunTask(base::PendingTask*) base/message_loop/message_loop_impl.cc:374:46 (interactive_ui_tests+0x9a75ab5) #33 DeferOrRunPendingTask base/message_loop/message_loop_impl.cc:385:5 (interactive_ui_tests+0x9a7640b) #34 base::MessageLoopImpl::DoWork() base/message_loop/message_loop_impl.cc:473 (interactive_ui_tests+0x9a7640b) #35 non-virtual thunk to base::MessageLoopImpl::DoWork() base/message_loop/message_loop_impl.cc (interactive_ui_tests+0x9a7644d) #36 base::MessagePumpDefault::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_default.cc:39:31 (interactive_ui_tests+0x9a7a7c1) #37 base::MessageLoopImpl::Run(bool) base/message_loop/message_loop_impl.cc:326:12 (interactive_ui_tests+0x9a75368) #38 non-virtual thunk to base::MessageLoopImpl::Run(bool) base/message_loop/message_loop_impl.cc (interactive_ui_tests+0x9a75464) #39 base::RunLoop::Run() base/run_loop.cc:102:14 (interactive_ui_tests+0x9abda4f) #40 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:250:13 (interactive_ui_tests+0x9b4c22b) #41 base::Thread::ThreadMain() base/threading/thread.cc:332:3 (interactive_ui_tests+0x9b4c6cc) #42 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:81:13 (interactive_ui_tests+0x9bbba31) Previous write of size 4 at 0x56051a90b610 by main thread (mutexes: write M17530): #0 pcache1FetchStage2 third_party/sqlite/amalgamation/sqlite3.c (interactive_ui_tests+0xb3bb4be) #1 pcache1FetchNoMutex third_party/sqlite/amalgamation/sqlite3.c:49514:12 (interactive_ui_tests+0xb3b9bc5) #2 pcache1Fetch third_party/sqlite/amalgamation/sqlite3.c:49556 (interactive_ui_tests+0xb3b9bc5) #3 sqlite3PcacheFetch third_party/sqlite/amalgamation/sqlite3.c:48011:10 (interactive_ui_tests+0xb271a6e) #4 getPageNormal third_party/sqlite/amalgamation/sqlite3.c:55899 (interactive_ui_tests+0xb271a6e) #5 sqlite3PagerGet third_party/sqlite/amalgamation/sqlite3.c:56078:10 (interactive_ui_tests+0xb282805) #6 btreeGetPage third_party/sqlite/amalgamation/sqlite3.c:65059 (interactive_ui_tests+0xb282805) #7 btreeGetUnusedPage third_party/sqlite/amalgamation/sqlite3.c:65203 (interactive_ui_tests+0xb282805) #8 allocateBtreePage third_party/sqlite/amalgamation/sqlite3.c:69099 (interactive_ui_tests+0xb282805) #9 btreeCreateTable third_party/sqlite/amalgamation/sqlite3.c:71903:10 (interactive_ui_tests+0xb2ac42e) #10 sqlite3BtreeCreateTable third_party/sqlite/amalgamation/sqlite3.c:71922 (interactive_ui_tests+0xb2ac42e) #11 sqlite3VdbeExec third_party/sqlite/amalgamation/sqlite3.c:86723:12 (interactive_ui_tests+0xb2917bb) #12 sqlite3Step third_party/sqlite/amalgamation/sqlite3.c:81415:10 (interactive_ui_tests+0xb24ba28) #13 chrome_sqlite3_step third_party/sqlite/amalgamation/sqlite3.c:81478 (interactive_ui_tests+0xb24ba28) #14 sql::Statement::StepInternal(bool) sql/statement.cc:71:13 (interactive_ui_tests+0xb233f6a) #15 sql::Statement::Step() sql/statement.cc:97:10 (interactive_ui_tests+0xb23452e) #16 history::URLDatabase::GetMostRecentKeywordSearchTerms(long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator<unsigned short> > const&, int, std::__1::vector<history::KeywordSearchTermVisit, std::__1::allocator<history::KeywordSearchTermVisit> >*) components/history/core/browser/url_database.cc:559:20 (interactive_ui_tests+0xc06f34c) #17 SearchProvider::DoHistoryQuery(bool) components/omnibox/browser/search_provider.cc:657:13 (interactive_ui_tests+0xd233632) #18 SearchProvider::Start(AutocompleteInput const&, bool) components/omnibox/browser/search_provider.cc:304:5 (interactive_ui_tests+0xd232cee) #19 AutocompleteController::Start(AutocompleteInput const&) components/omnibox/browser/autocomplete_controller.cc:320:11 (interactive_ui_tests+0xd1cf6e9) #20 OmniboxController::StartAutocomplete(AutocompleteInput const&) const components/omnibox/browser/omnibox_controller.cc:37:29 (interactive_ui_tests+0xd2200c0) #21 OmniboxEditModel::StartAutocomplete(bool, bool) components/omnibox/browser/omnibox_edit_model.cc:526:24 (interactive_ui_tests+0xd21b193) #22 OmniboxEditModel::UpdateInput(bool, bool) components/omnibox/browser/omnibox_edit_model.cc:459:3 (interactive_ui_tests+0xd21ab7b) #23 OmniboxViewViews::UpdatePopup() chrome/browser/ui/views/omnibox/omnibox_view_views.cc:631:12 (interactive_ui_tests+0xd7dbb93) #24 OmniboxViewViews::SetWindowTextAndCaretPos(std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator<unsigned short> > const&, unsigned long, bool, bool) chrome/browser/ui/views/omnibox/omnibox_view_views.cc:613:5 (interactive_ui_tests+0xd7db90e) #25 OmniboxEditModel::EnterKeywordModeForDefaultSearchProvider(metrics::OmniboxEventProto_KeywordModeEntryMethod) components/omnibox/browser/omnibox_edit_model.cc:660:10 (interactive_ui_tests+0xd21c194) #26 OmniboxViewViews::EnterKeywordModeForDefaultSearchProvider() chrome/browser/ui/views/omnibox/omnibox_view_views.cc:300:12 (interactive_ui_tests+0xd7da478) #27 LocationBarView::FocusSearch() chrome/browser/ui/views/location_bar/location_bar_view.cc:916:18 (interactive_ui_tests+0xd78ed1c) #28 chrome::FocusSearch(Browser*) chrome/browser/ui/browser_commands.cc:1084:40 (interactive_ui_tests+0xd474da0) #29 OmniboxViewTest_FocusSearchLongUrl_Test::RunTestOnMainThread() chrome/browser/ui/omnibox/omnibox_view_browsertest.cc:851:3 (interactive_ui_tests+0x3aa7c09) #30 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() content/public/test/browser_test_base.cc:447:5 (interactive_ui_tests+0x8ed5c44) #31 Invoke<void (content::BrowserTestBase::*)(), content::BrowserTestBase *> base/bind_internal.h:516:12 (interactive_ui_tests+0x8ed6d08) #32 MakeItSo<void (content::BrowserTestBase::*const &)(), content::BrowserTestBase *> base/bind_internal.h:616 (interactive_ui_tests+0x8ed6d08) #33 RunImpl<void (content::BrowserTestBase::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::BrowserTestBase> > &, 0> base/bind_internal.h:689 (interactive_ui_tests+0x8ed6d08) #34 base::internal::Invoker<base::internal::BindState<void (content::BrowserTestBase::*)(), base::internal::UnretainedWrapper<content::BrowserTestBase> >, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:671 (interactive_ui_tests+0x8ed6d08) #35 Run base/callback.h:129:12 (interactive_ui_tests+0x85f058a) #36 ChromeBrowserMainParts::PreMainMessageLoopRunImpl() chrome/browser/chrome_browser_main.cc:1787 (interactive_ui_tests+0x85f058a) #37 ChromeBrowserMainParts::PreMainMessageLoopRun() chrome/browser/chrome_browser_main.cc:1175:18 (interactive_ui_tests+0x85eecb7) #38 content::BrowserMainLoop::PreMainMessageLoopRun() content/browser/browser_main_loop.cc:983:13 (interactive_ui_tests+0x54ea6b5) #39 Invoke<int (content::BrowserMainLoop::*)(), content::BrowserMainLoop *> base/bind_internal.h:516:12 (interactive_ui_tests+0x54ed638) #40 MakeItSo<int (content::BrowserMainLoop::*const &)(), content::BrowserMainLoop *> base/bind_internal.h:616 (interactive_ui_tests+0x54ed638) #41 RunImpl<int (content::BrowserMainLoop::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::BrowserMainLoop> > &, 0> base/bind_internal.h:689 (interactive_ui_tests+0x54ed638) #42 base::internal::Invoker<base::internal::BindState<int (content::BrowserMainLoop::*)(), base::internal::UnretainedWrapper<content::BrowserMainLoop> >, int ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:671 (interactive_ui_tests+0x54ed638) #43 Run base/callback.h:129:12 (interactive_ui_tests+0x5d5f90a) #44 content::StartupTaskRunner::RunAllTasksNow() content/browser/startup_task_runner.cc:41 (interactive_ui_tests+0x5d5f90a) #45 content::BrowserMainLoop::CreateStartupTasks() content/browser/browser_main_loop.cc:917:25 (interactive_ui_tests+0x54e8df6) #46 content::BrowserMainRunnerImpl::Initialize(content::MainFunctionParams const&) content/browser/browser_main_runner_impl.cc:144:15 (interactive_ui_tests+0x54edfa8) #47 content::BrowserMain(content::MainFunctionParams const&) content/browser/browser_main.cc:43:32 (interactive_ui_tests+0x54e612c) #48 RunBrowserProcessMain content/app/content_main_runner_impl.cc:543:10 (interactive_ui_tests+0x81f3987) #49 content::ContentMainRunnerImpl::RunServiceManager(content::MainFunctionParams&, bool) content/app/content_main_runner_impl.cc:941 (interactive_ui_tests+0x81f3987) #50 content::ContentMainRunnerImpl::Run(bool) content/app/content_main_runner_impl.cc:866:12 (interactive_ui_tests+0x81f335f) #51 content::ContentServiceManagerMainDelegate::RunEmbedderProcess() content/app/content_service_manager_main_delegate.cc:53:32 (interactive_ui_tests+0x81f0abf) #52 service_manager::Main(service_manager::MainParams const&) services/service_manager/embedder/main.cc:470:29 (interactive_ui_tests+0xbcaf820) #53 content::ContentMain(content::ContentMainParams const&) content/app/content_main.cc:19:10 (interactive_ui_tests+0x81f119b) #54 content::BrowserTestBase::SetUp() content/public/test/browser_test_base.cc:349:3 (interactive_ui_tests+0x8ed56fa) #55 InProcessBrowserTest::SetUp() chrome/test/base/in_process_browser_test.cc:283:20 (interactive_ui_tests+0x85d5c04) #56 OmniboxViewTest::SetUp() chrome/browser/ui/omnibox/omnibox_view_browsertest.cc:170:27 (interactive_ui_tests+0x3abed66) #57 HandleExceptionsInMethodIfSupported<testing::Test, void> third_party/googletest/src/googletest/src/gtest.cc (interactive_ui_tests+0x3df4122) #58 testing::Test::Run() third_party/googletest/src/googletest/src/gtest.cc:2517 (interactive_ui_tests+0x3df4122) #59 testing::TestInfo::Run() third_party/googletest/src/googletest/src/gtest.cc:2703:11 (interactive_ui_tests+0x3df5568) #60 testing::TestCase::Run() third_party/googletest/src/googletest/src/gtest.cc:2825:28 (interactive_ui_tests+0x3df5ea6) #61 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5227:43 (interactive_ui_tests+0x3e06c46) #62 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/src/googletest/src/gtest.cc (interactive_ui_tests+0x3e06528) #63 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:4835 (interactive_ui_tests+0x3e06528) #64 RUN_ALL_TESTS third_party/googletest/src/googletest/include/gtest/gtest.h:2369:46 (interactive_ui_tests+0x9a1dce5) #65 base::TestSuite::Run() base/test/test_suite.cc:294 (interactive_ui_tests+0x9a1dce5) #66 InteractiveUITestSuiteRunner::RunTestSuite(int, char**) chrome/test/base/interactive_ui_tests_main.cc:135:47 (interactive_ui_tests+0x3af5e74) #67 ChromeTestLauncherDelegate::RunTestSuite(int, char**) chrome/test/base/chrome_test_launcher.cc:80:19 (interactive_ui_tests+0x85cca8f) #68 content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**) content/public/test/test_launcher.cc:647:31 (interactive_ui_tests+0x8ee68be) #69 LaunchChromeTests(unsigned long, content::TestLauncherDelegate*, int, char**) chrome/test/base/chrome_test_launcher.cc:182:10 (interactive_ui_tests+0x85cd011) #70 main chrome/test/base/interactive_ui_tests_main.cc:159:10 (interactive_ui_tests+0x3af5dc7) Location is global 'pcache1Create.dummyCurrentPage' of size 4 at 0x56051a90b610 (interactive_ui_tests+0x000012c1b610) Mutex M990364259539016336 is already destroyed. Mutex M17530 (0x7b0c0010dcd0) created at: #0 pthread_mutex_init /b/swarming/w/ir/kitchen-workdir/src/third_party/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1228:3 (interactive_ui_tests+0x38b8e63) #1 pthreadMutexAlloc third_party/sqlite/amalgamation/sqlite3.c:25487:9 (interactive_ui_tests+0xb3a8bdb) #2 sqlite3MutexAlloc third_party/sqlite/amalgamation/sqlite3.c:25019:10 (interactive_ui_tests+0xb25df5b) #3 openDatabase third_party/sqlite/amalgamation/sqlite3.c:155439 (interactive_ui_tests+0xb25df5b) #4 chrome_sqlite3_open_v2 third_party/sqlite/amalgamation/sqlite3.c:155754:10 (interactive_ui_tests+0xb25f800) #5 sql::Database::OpenInternal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, sql::Database::Retry) sql/database.cc:1597:13 (interactive_ui_tests+0xb2287e7) #6 sql::Database::OpenInMemory() sql/database.cc:358:10 (interactive_ui_tests+0xb228f89) #7 history::InMemoryDatabase::InitDB() components/history/core/browser/in_memory_database.cc:25:12 (interactive_ui_tests+0xc038d34) #8 history::InMemoryDatabase::InitFromDisk(base::FilePath const&) components/history/core/browser/in_memory_database.cc:65:8 (interactive_ui_tests+0xc038f67) #9 history::InMemoryHistoryBackend::Init(base::FilePath const&) components/history/core/browser/in_memory_history_backend.cc:28:15 (interactive_ui_tests+0xc0384f9) #10 history::HistoryBackend::InitImpl(history::HistoryDatabaseParams const&) components/history/core/browser/history_backend.cc:721:22 (interactive_ui_tests+0xc00d6f6) #11 history::HistoryBackend::Init(bool, history::HistoryDatabaseParams const&) components/history/core/browser/history_backend.cc:242:5 (interactive_ui_tests+0xc00d0fd) #12 Invoke<void (history::HistoryBackend::*)(bool, const history::HistoryDatabaseParams &), scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams> base/bind_internal.h:516:12 (interactive_ui_tests+0xc00a1a9) #13 MakeItSo<void (history::HistoryBackend::*)(bool, const history::HistoryDatabaseParams &), scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams> base/bind_internal.h:616 (interactive_ui_tests+0xc00a1a9) #14 RunImpl<void (history::HistoryBackend::*)(bool, const history::HistoryDatabaseParams &), std::__1::tuple<scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams>, 0, 1, 2> base/bind_internal.h:689 (interactive_ui_tests+0xc00a1a9) #15 base::internal::Invoker<base::internal::BindState<void (history::HistoryBackend::*)(bool, history::HistoryDatabaseParams const&), scoped_refptr<history::HistoryBackend>, bool, history::HistoryDatabaseParams>, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:658 (interactive_ui_tests+0xc00a1a9) #16 Run base/callback.h:99:12 (interactive_ui_tests+0x9a7795d) #17 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:99 (interactive_ui_tests+0x9a7795d) #18 base::MessageLoopImpl::RunTask(base::PendingTask*) base/message_loop/message_loop_impl.cc:374:46 (interactive_ui_tests+0x9a75ab5) #19 DeferOrRunPendingTask base/message_loop/message_loop_impl.cc:385:5 (interactive_ui_tests+0x9a7640b) #20 base::MessageLoopImpl::DoWork() base/message_loop/message_loop_impl.cc:473 (interactive_ui_tests+0x9a7640b) #21 non-virtual thunk to base::MessageLoopImpl::DoWork() base/message_loop/message_loop_impl.cc (interactive_ui_tests+0x9a7644d) #22 base::MessagePumpDefault::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_default.cc:39:31 (interactive_ui_tests+0x9a7a709) #23 base::MessageLoopImpl::Run(bool) base/message_loop/message_loop_impl.cc:326:12 (interactive_ui_tests+0x9a75368) #24 non-virtual thunk to base::MessageLoopImpl::Run(bool) base/message_loop/message_loop_impl.cc (interactive_ui_tests+0x9a75464) #25 base::RunLoop::Run() base/run_loop.cc:102:14 (interactive_ui_tests+0x9abda4f) #26 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:250:13 (interactive_ui_tests+0x9b4c22b) #27 base::Thread::ThreadMain() base/threading/thread.cc:332:3 (interactive_ui_tests+0x9b4c6cc) #28 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:81:13 (interactive_ui_tests+0x9bbba31) Thread T25 'Chrome_HistoryThread' (tid=19310, running) created by main thread at: #0 pthread_create /b/swarming/w/ir/kitchen-workdir/src/third_party/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:980:3 (interactive_ui_tests+0x38b78a5) #1 base::(anonymous namespace)::CreateThread(unsigned long, bool, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) base/threading/platform_thread_posix.cc:120:13 (interactive_ui_tests+0x9bbb316) #2 base::PlatformThread::CreateWithPriority(unsigned long, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) base/threading/platform_thread_posix.cc:238:10 (interactive_ui_tests+0x9bbb1d5) #3 base::Thread::StartWithOptions(base::Thread::Options const&) base/threading/thread.cc:112:15 (interactive_ui_tests+0x9b4b935) #4 history::HistoryService::Init(bool, history::HistoryDatabaseParams const&) components/history/core/browser/history_service.cc:943:21 (interactive_ui_tests+0xbff8ff0) #5 Init components/history/core/browser/history_service.h:101:12 (interactive_ui_tests+0x86c2892) #6 HistoryServiceFactory::BuildServiceInstanceFor(content::BrowserContext*) const chrome/browser/history/history_service_factory.cc:84 (interactive_ui_tests+0x86c2892) #7 BrowserContextKeyedServiceFactory::BuildServiceInstanceFor(base::SupportsUserData*) const components/keyed_service/content/browser_context_keyed_service_factory.cc:105:7 (interactive_ui_tests+0xc775fff) #8 KeyedServiceFactory::GetServiceForContext(base::SupportsUserData*, bool) components/keyed_service/core/keyed_service_factory.cc:89:15 (interactive_ui_tests+0xac810b7) #9 BrowserContextKeyedServiceFactory::GetServiceForBrowserContext(content::BrowserContext*, bool) components/keyed_service/content/browser_context_keyed_service_factory.cc:58:31 (interactive_ui_tests+0xc775e39) #10 HistoryServiceFactory::GetForProfile(Profile*, ServiceAccessType) chrome/browser/history/history_service_factory.cc:32:22 (interactive_ui_tests+0x86c240d) #11 BrowsingDataHistoryObserverService chrome/browser/browsing_data/browsing_data_history_observer_service.cc:22:27 (interactive_ui_tests+0x889514d) #12 BrowsingDataHistoryObserverService::Factory::BuildServiceInstanceFor(content::BrowserContext*) const chrome/browser/browsing_data/browsing_data_history_observer_service.cc:60 (interactive_ui_tests+0x889514d) #13 BrowserContextKeyedServiceFactory::BuildServiceInstanceFor(base::SupportsUserData*) const components/keyed_service/content/browser_context_keyed_service_factory.cc:105:7 (interactive_ui_tests+0xc775fff) #14 KeyedServiceFactory::GetServiceForContext(base::SupportsUserData*, bool) components/keyed_service/core/keyed_service_factory.cc:89:15 (interactive_ui_tests+0xac810b7) #15 KeyedServiceFactory::CreateServiceNow(base::SupportsUserData*) components/keyed_service/core/keyed_service_factory.cc:137:3 (interactive_ui_tests+0xac81a55) #16 DependencyManager::CreateContextServices(base::SupportsUserData*, bool) components/keyed_service/core/dependency_manager.cc:72:16 (interactive_ui_tests+0xac7cc78) #17 BrowserContextDependencyManager::DoCreateBrowserContextServices(content::BrowserContext*, bool) components/keyed_service/content/browser_context_dependency_manager.cc:47:22 (interactive_ui_tests+0xc774ff8) #18 BrowserContextDependencyManager::CreateBrowserContextServices(content::BrowserContext*) components/keyed_service/content/browser_context_dependency_manager.cc:32:3 (interactive_ui_tests+0xc774f82) #19 ProfileImpl::OnLocaleReady() chrome/browser/profiles/profile_impl.cc:912:11 (interactive_ui_tests+0x88b93d7) #20 ProfileImpl::OnPrefsLoaded(Profile::CreateMode, bool) chrome/browser/profiles/profile_impl.cc:946:3 (interactive_ui_tests+0x88b65a0) #21 ProfileImpl::ProfileImpl(base::FilePath const&, Profile::Delegate*, Profile::CreateMode, scoped_refptr<base::SequencedTaskRunner>) chrome/browser/profiles/profile_impl.cc:576:5 (interactive_ui_tests+0x88b60c4) #22 Profile::CreateProfile(base::FilePath const&, Profile::Delegate*, Profile::CreateMode) chrome/browser/profiles/profile_impl.cc:354:11 (interactive_ui_tests+0x88b3fd2) #23 ProfileManager::CreateProfileHelper(base::FilePath const&) chrome/browser/profiles/profile_manager.cc:1397:10 (interactive_ui_tests+0x88e4b41) #24 ProfileManager::CreateAndInitializeProfile(base::FilePath const&) chrome/browser/profiles/profile_manager.cc:1474:22 (interactive_ui_tests+0x88dc77e) #25 ProfileManager::GetProfile(base::FilePath const&) chrome/browser/profiles/profile_manager.cc:519:10 (interactive_ui_tests+0x88dc4f9) #26 GetStartupProfile(base::FilePath const&, base::CommandLine const&) chrome/browser/ui/startup/startup_browser_creator.cc:994:39 (interactive_ui_tests+0xd4c78b2) #27 CreatePrimaryProfile chrome/browser/chrome_browser_main.cc:411:13 (interactive_ui_tests+0x85ef74e) #28 ChromeBrowserMainParts::PreMainMessageLoopRunImpl() chrome/browser/chrome_browser_main.cc:1497 (interactive_ui_tests+0x85ef74e) #29 ChromeBrowserMainParts::PreMainMessageLoopRun() chrome/browser/chrome_browser_main.cc:1175:18 (interactive_ui_tests+0x85eecb7) #30 content::BrowserMainLoop::PreMainMessageLoopRun() content/browser/browser_main_loop.cc:983:13 (interactive_ui_tests+0x54ea6b5) #31 Invoke<int (content::BrowserMainLoop::*)(), content::BrowserMainLoop *> base/bind_internal.h:516:12 (interactive_ui_tests+0x54ed638) #32 MakeItSo<int (content::BrowserMainLoop::*const &)(), content::BrowserMainLoop *> base/bind_internal.h:616 (interactive_ui_tests+0x54ed638) #33 RunImpl<int (content::BrowserMainLoop::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::BrowserMainLoop> > &, 0> base/bind_internal.h:689 (interactive_ui_tests+0x54ed638) #34 base::internal::Invoker<base::internal::BindState<int (content::BrowserMainLoop::*)(), base::internal::UnretainedWrapper<content::BrowserMainLoop> >, int ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:671 (interactive_ui_tests+0x54ed638) #35 Run base/callback.h:129:12 (interactive_ui_tests+0x5d5f90a) #36 content::StartupTaskRunner::RunAllTasksNow() content/browser/startup_task_runner.cc:41 (interactive_ui_tests+0x5d5f90a) #37 content::BrowserMainLoop::CreateStartupTasks() content/browser/browser_main_loop.cc:917:25 (interactive_ui_tests+0x54e8df6) #38 content::BrowserMainRunnerImpl::Initialize(content::MainFunctionParams const&) content/browser/browser_main_runner_impl.cc:144:15 (interactive_ui_tests+0x54edfa8) #39 content::BrowserMain(content::MainFunctionParams const&) content/browser/browser_main.cc:43:32 (interactive_ui_tests+0x54e612c) #40 RunBrowserProcessMain content/app/content_main_runner_impl.cc:543:10 (interactive_ui_tests+0x81f3987) #41 content::ContentMainRunnerImpl::RunServiceManager(content::MainFunctionParams&, bool) content/app/content_main_runner_impl.cc:941 (interactive_ui_tests+0x81f3987) #42 content::ContentMainRunnerImpl::Run(bool) content/app/content_main_runner_impl.cc:866:12 (interactive_ui_tests+0x81f335f) #43 content::ContentServiceManagerMainDelegate::RunEmbedderProcess() content/app/content_service_manager_main_delegate.cc:53:32 (interactive_ui_tests+0x81f0abf) #44 service_manager::Main(service_manager::MainParams const&) services/service_manager/embedder/main.cc:470:29 (interactive_ui_tests+0xbcaf820) #45 content::ContentMain(content::ContentMainParams const&) content/app/content_main.cc:19:10 (interactive_ui_tests+0x81f119b) #46 content::BrowserTestBase::SetUp() content/public/test/browser_test_base.cc:349:3 (interactive_ui_tests+0x8ed56fa) #47 InProcessBrowserTest::SetUp() chrome/test/base/in_process_browser_test.cc:283:20 (interactive_ui_tests+0x85d5c04) #48 OmniboxViewTest::SetUp() chrome/browser/ui/omnibox/omnibox_view_browsertest.cc:170:27 (interactive_ui_tests+0x3abed66) #49 HandleExceptionsInMethodIfSupported<testing::Test, void> third_party/googletest/src/googletest/src/gtest.cc (interactive_ui_tests+0x3df4122) #50 testing::Test::Run() third_party/googletest/src/googletest/src/gtest.cc:2517 (interactive_ui_tests+0x3df4122) #51 testing::TestInfo::Run() third_party/googletest/src/googletest/src/gtest.cc:2703:11 (interactive_ui_tests+0x3df5568) #52 testing::TestCase::Run() third_party/googletest/src/googletest/src/gtest.cc:2825:28 (interactive_ui_tests+0x3df5ea6) #53 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5227:43 (interactive_ui_tests+0x3e06c46) #54 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> third_party/googletest/src/googletest/src/gtest.cc (interactive_ui_tests+0x3e06528) #55 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:4835 (interactive_ui_tests+0x3e06528) #56 RUN_ALL_TESTS third_party/googletest/src/googletest/include/gtest/gtest.h:2369:46 (interactive_ui_tests+0x9a1dce5) #57 base::TestSuite::Run() base/test/test_suite.cc:294 (interactive_ui_tests+0x9a1dce5) #58 InteractiveUITestSuiteRunner::RunTestSuite(int, char**) chrome/test/base/interactive_ui_tests_main.cc:135:47 (interactive_ui_tests+0x3af5e74) #59 ChromeTestLauncherDelegate::RunTestSuite(int, char**) chrome/test/base/chrome_test_launcher.cc:80:19 (interactive_ui_tests+0x85cca8f) #60 content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**) content/public/test/test_launcher.cc:647:31 (interactive_ui_tests+0x8ee68be) #61 LaunchChromeTests(unsigned long, content::TestLauncherDelegate*, int, char**) chrome/test/base/chrome_test_launcher.cc:182:10 (interactive_ui_tests+0x85cd011) #62 main chrome/test/base/interactive_ui_tests_main.cc:159:10 (interactive_ui_tests+0x3af5dc7) SUMMARY: ThreadSanitizer: data race third_party/sqlite/amalgamation/sqlite3.c:48979:25 in pcache1FreePage ================== [19276:19276:1210/135505.343009:WARNING:gaia_auth_fetcher.cc(891)] Could not reach Google Accounts servers: errno -11 [19276:19285:1210/135505.799020:WARNING:discardable_shared_memory_manager.cc(410)] Some MojoDiscardableSharedMemoryManagerImpls are still alive. They will be leaked. [19276:19276:1210/135505.824433:WARNING:pref_notifier_impl.cc(23)] Pref observer found at shutdown. [19276:19276:1210/135505.824558:WARNING:pref_notifier_impl.cc(23)] Pref observer found at shutdown. [ OK ] OmniboxViewTest.FocusSearchLongUrl (5555 ms) Since two of these tests also fail without the roll, I'm re-sending the try job. Maybe it's nothing, but it looks like it might be something. Happy with pinned clang: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20TSan%20Tests/30826
,
Dec 10
Maybe due to https://reviews.llvm.org/D54521 (r347383 "tsan: add pthread_tryjoin_np and pthread_timedjoin_np interceptors"). But the previous roll attempt https://chromium-review.googlesource.com/c/1356710 ("346388:347933") already had that and went through no problem
,
Dec 11
+pwnall as sql/OWNER to see if the stacks make sense at all.
,
Dec 11
It's not always happy with pinned clang. Here's a similar error: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20TSan%20Tests/30832 But it only happens once in a while. I suppose the race has to be exposed for tsan to catch it.
,
Dec 11
The tsan bot on https://chromium-review.googlesource.com/c/chromium/src/+/1369628 is failing pretty consistently. (And looks like mac/asan times out pretty consistently, hmm :-/ Probably a blocking bug too; will file when I get in.) I tried one tsan try run on a whitespace cl (https://chromium-review.googlesource.com/c/chromium/src/+/1296616) and that came back happy.
,
Dec 14
mac/asan also times out on a whitespace cl (https://chromium-review.googlesource.com/c/chromium/src/+/1296616), that trybot is probably just in bad shape atm. That's likely not a blocker. Our new tsan tot bot also has a bunch of failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ToTLinuxTSan/7 They're a superset of the https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_tsan_rel_ng/160944 failures though.
,
Dec 14
mpearson, does the stack in comment 0 look like a real threading issue around the history db, or does this look more like a tooling bug to you?
,
Dec 17
I tried to repro this today, but so far failed. To get the test to not time out, I used this combination of flags: $ testing/xvfb.py out/tsan/interactive_ui_tests --gtest_filter=OmniboxViewTest.CutTextToClipboard --no-sandbox --disable-gpu --single_process --no-first-run (It still takes 83s to run.) This is over ssh; maybe running it with X repros better. I had also run: export G_SLICE=always-malloc export NSS_DISABLE_ARENA_FREE_LIST=1 export NSS_DISABLE_UNLOAD=1 as recommended on http://dev.chromium.org/developers/testing/threadsanitizer-tsan-v2
,
Dec 17
Oh, and that was with https://chromium-review.googlesource.com/c/chromium/src/+/1366078 patched in.
,
Dec 17
With $ testing/run_with_dummy_home.py testing/xvfb.py out/tsan/interactive_ui_tests --gtest_filter=OmniboxViewTest.CutTextToClipboard --no-sandbox --disable-gpu --no-first-run --gtest_repeat=10 the first run times out, but the remaining runs all finish within 17s or so (and they all passed). Trying running more than one test next.
,
Dec 17
Hooray, repro. With: testing/run_with_dummy_home.py testing/xvfb.py out/tsan/interactive_ui_tests --gtest_filter=OmniboxViewTest.* --no-sandbox --disable-gpu --no-first-run --gtest_repeat=10 First test times out, next two pass, fourth shows the stack on this bug.
,
Dec 17
On second try, it took 10 tests ([10/31]) until the race appeared, and the first two tests timed out.
,
Dec 17
I'm now trying the same thing with Hans's roll (https://chromium-review.googlesource.com/c/1356710) instead of mine. With that, I'm at 4 test timeouts without a single successful run yet O_O
,
Dec 17
(The --user-data-dir flag above has no effect, ChromeTestLauncherDelegate::AdjustChildProcessCommandLine() filters it out)
,
Dec 17
With pinned clang, tests also always time out. So I guess I can at least repro that something has changed and maybe bisect that to a commit, but on my machine the change goes from "tests always time out" to "tests now run and sometimes report a race". Which is different from what the bots see.
,
Dec 17
"good" 347933 "bad" 348507
,
Dec 17
re: > mpearson, does the stack in comment 0 look like a real threading issue > around the history db, or does this look more like a tooling bug to you? I think it's (likely) not a real issue. The race, as near as I can tell, is the two following items: - when typing in the omnibox, SearchProvider calls GetMostRecentKeywordSearchTerms(). This looks at the keyword_search_terms table in the database. This happens synchronously at each keypress. - when typing the omnibox, HistoryURLProvider calls ExecuteWithDB. This happens asynchronously soon after each keypress. It looks at the urls table in the database. It's okay for these to happen in any order. Hope this helps; I'm not sure how else to contribute.
,
Dec 18
at 348250 tests still time out
,
Dec 18
at 348375 it also still times out. xvfb.py has a --tsan=1 flag that will set --disable-sandbox and the env vars (but not the other flags)
,
Dec 18
at 348440 also times out. will try 348507 with my locally-built clang/compiler-rt/lld to make sure that the issue isn't with my local llvm cmake config or something.
,
Dec 18
Also seeing timeouts with my locally-built clang at 348507, so I'm holding something wrong and my bisect results so far are invalid. Hans pointed me at the flags at https://www.chromium.org/developers/testing/browser-tests for increasing timeouts, and I finally found the code that adjusts timeouts for the sanitizers at https://cs.chromium.org/chromium/src/base/test/test_timeouts.cc?type=cs&sq=package:chromium&g=0&l=36 So trying to repro with locally-built clang again...
,
Dec 18
Passing --test-launcher-timeout=1000000 had no effect.
With this I'm able to repro with my locally-built clang (but see below):
diff --git a/base/test/test_timeouts.cc b/base/test/test_timeouts.cc
index 9328aaba0b6e..cb25ef6915a4 100644
--- a/base/test/test_timeouts.cc
+++ b/base/test/test_timeouts.cc
@@ -54,7 +54,7 @@ void InitializeTimeout(const char* switch_name, int min_value, int* value) {
// gets 3x.
constexpr int kTimeoutMultiplier = 3;
#elif defined(ADDRESS_SANITIZER) || defined(THREAD_SANITIZER)
- constexpr int kTimeoutMultiplier = 2;
+ constexpr int kTimeoutMultiplier = 5;
#else
constexpr int kTimeoutMultiplier = 1;
#endif
The first test takes a little under 90s to run, after that tests take 5-6s. My commandline is
time testing/run_with_dummy_home.py testing/xvfb.py --tsan=1 out/tsan/interactive_ui_tests --gtest_filter=OmniboxViewTest.* --no-first-run --gtest_repeat=10 --disable-gpu --test-launcher-timeout=1000000
(but that last flag has no effect.)
This runs 31 tests per run and does 10 runs. I ran it for 2 full 31 test cycles and 3 tests. On the first run, the race was detected after test 21/31. In the second cycle, it wasn't seen at all. In the third, it was seen on test 2/31.
So this is not the most reliable of repros. But maybe good enough for trying to bisect again.
,
Dec 18
Passing --test-launcher-bot-mode makes the output much less noisy. I've re-checked with pinned clang and with 347933 and couldn't repro a race with two full runs of the 31 OmniboxViewTest.* tests. Trying my bisect again.
,
Dec 18
2 runs and 10/31 tests in the third run without races at 348250
,
Dec 18
2 runs and 6/31 tests in the third run without races at 348375
,
Dec 18
4 runs and 16/31 tests in the fifth run without races at 348440
,
Dec 18
Reprod on first test in second run at 348500
,
Dec 19
4 runs without races at 348470
,
Dec 19
Repros at 348485 at test 21/31 of the 2nd run.
,
Dec 19
I tried bisecting over llvm 346388:348614 last night using this script: --- #!/bin/bash set -exo pipefail ninja -C /work/llvm.combined/build.release lld clang llvm-ar compiler-rt sancov fuzzer || exit 125 ninja -C /work/chromium/src/out/release -t clean || true ninja -C /work/chromium/src/out/release interactive_ui_tests || exit 125 cd /work/chromium/src/ ( G_SLICE=always-malloc NSS_DISABLE_ARENA_FREE_LIST=1 NSS_DISABLE_UNLOAD=1 testing/run_with_dummy_home.py testing/xvfb.py out/release/interactive_ui_tests --gtest_filter=OmniboxViewTest.DesiredTLD --no-sandbox --disable-gpu --no-first-run --gtest_repeat=50 --ui-test-action-max-timeout=5000 --test-launcher-timeout=90000 || true ) 2>&1 | tee /tmp/output # This is the "Bad" we're looking for. grep "ThreadSanitizer: data race sqlite3_shim.c:? in pcache1FetchStage2" /tmp/output && exit 1 # Otherwise check that the tests passed at least once (some runs time out) grep "SUCCESS: all tests passed" /tmp/output && exit 0 # Otherwise we don't know, so mark it skip. exit 125 --- It points to r348496: --- 158c6181483208a8814ab47f22fe25dbc8be0541 is the first bad commit commit 158c6181483208a8814ab47f22fe25dbc8be0541 Author: Alexandros Lamprineas <alexandros.lamprineas@arm.com> Date: Thu Dec 6 16:11:58 2018 +0000 [GVN] Don't perform scalar PRE on GEPs Partial Redundancy Elimination of GEPs prevents CodeGenPrepare from sinking the addressing mode computation of memory instructions back to its uses. The problem comes from the insertion of PHIs, which confuse CGP and make it bail. I've autogenerated the check lines of an existing test and added a store instruction to demonstrate the motivation behind this change. The store is now using the gep instead of a phi. Differential Revision: https://reviews.llvm.org/D55009 --- That would have the effect of moving some code around, so I suppose it could trigger this maybe. But Nico saw it reproduce w/ r348485 which is earlier :-( I'll do some more runs before and after the revision I found. But at this point maybe digging into the code and trying to figure out whether the tsan report is legit is the way to go?
,
Dec 19
From mpearson in #17: > I think it's (likely) not a real issue. The race, as near as I can tell, is the two following items: > - when typing in the omnibox, SearchProvider calls GetMostRecentKeywordSearchTerms(). This looks at the keyword_search_terms table in the database. This happens synchronously at each keypress. > - when typing the omnibox, HistoryURLProvider calls ExecuteWithDB. This happens asynchronously soon after each keypress. It looks at the urls table in the database. > > It's okay for these to happen in any order. Is there anything that ensures these happens one after the other, and that they can't overlap? Trying to decipher the tsan report. Thread "T25" accesses the database from history::URLDatabase::AutocompleteForPrefix called by HistoryURLProvider::DoAutocomplete scheduled on a background thread by HistoryService::ScheduleAutoComplete called by HistoryUrlProvider::Start called by AutocompleteController::Start At the same time, the main thread accesses the database from history::URLDatabase::GetMostRecentKeywordSearchTerms called by SearchProvider::DoHistoryQuery called by SearchProvider::Start called by AutocompleteController::Start So it seems that AutocompleteController::Start first kicks off a background task which accesses the URLDatabase, and then synchronously accesses the URLDatabase on its own thread. Unless I'm missing anything, it looks like a race to me. I don't see anything in history::URLDatabase that suggests it supports concurrent accesses. Should it be checking that all accesses are on the same thread or sequence?
,
Dec 19
> I'll do some more runs before and after the revision I found. I've done 100 test runs and am still unable to repro the same tsan stack with clang r348495. I do get another race instead, in unixRandomness() on a stack involving a WebDatabase and history::HistoryDatabase, but that causes the test to time out rather than fail as normal. Attaching that stack.
,
Dec 19
> So it seems that AutocompleteController::Start first kicks off a background task which accesses the URLDatabase, and then synchronously accesses the URLDatabase on its own thread. Unless I'm missing anything, it looks like a race to me. That's assuming sqlite can't handle it though. Looking in third_party/sqlite/BUILD.gn, it seems we build with -DSQLITE_THREADSAFE=1, which I think means sqlite is supposed to be able two threads using the same db concurrently. In which case it sounds like the problem is in sqlite and not our code?
,
Dec 19
No race in 4 full + 5/31 at 348481. I don't really see anything in 348482:348485 that could cause this, so I'm not sure I trust my results. I've sadly lost the scrollback with the race report at 85.
,
Dec 19
The "location is.." message in comment 0 points at https://cs.chromium.org/chromium/src/third_party/sqlite/src/src/pcache1.c?type=cs&q=pcache1FetchStage2&sq=package:chromium&g=0&l=783 , a static, which does look like the compiler might emit bad code for that somehow.
,
Dec 19
> a static, which does look like the compiler might emit bad code for that somehow. Maybe it's at least enough to confuse tsan's instrumentation? Seems worth a shot: https://chromium-review.googlesource.com/c/chromium/src/+/1384504
,
Dec 19
Looks like the tests still fail with my attempt.
,
Dec 19
re hans@ comment 31: Thanks for questioning me in more detail. I think you're right there's a real race here. > > It's okay for these to happen in any order. > Is there anything that ensures these happens one after the other, > and that they can't overlap? I would've thought so. Now I'm not so sure. AutocompleteController::Start calls each providers' Start() function synchronously HistoryUrlProvider::Start schedules the background task. For some reason, I simply assumed that the background task wouldn't run until the main thread's synchronous calls of all providers' Start() functions is complete. I guess that's not true. I don't think there would be a problem if they overlap. They were independent tables, and I thought databases supported simultaneous access, and I don't see anything worrying in the interface code to the database... That said, the simplest solution to test my theory is to push the HistoryURLProvider last in the list of providers to be called in AutocompleteController::Start(). The code to modify is here: https://cs.chromium.org/chromium/src/components/omnibox/browser/autocomplete_controller.cc?&l=198-271
,
Dec 19
No races in 5 full and 18/31 tests at 348484, so my bisection points to 348485. That's a libc++ change, which we don't pull from trunk. So my bisection went wrong somewhere :-( I'm making packages at and before the revision Hans found at https://chromium-review.googlesource.com/c/chromium/src/+/1384628 https://chromium-review.googlesource.com/c/chromium/src/+/1384629 to check if that has an effect on the bots.
,
Dec 19
I'd also suggest that, if we've found a pre-existing bug in chrome, it doesn't need to block the clang roll. Back when we had the memory fyi waterfall, a pretty standard response was to take a report, analyze it, file it against an OWNER, and find a way to suppress it. I started to see if this was still doable, and I found this existing sqlite3 suppression that looks very similar: https://cs.chromium.org/chromium/src/build/sanitizers/tsan_suppressions.cc?q=tsan+suppressions&sq=package:chromium&dr=CSs&l=34 https://bugs.chromium.org/p/chromium/issues/detail?id=84094 Maybe the LLVM change changed inliner behavior, and so this is an old pre-existing race that our suppressions no longer match?
,
Dec 19
mpearson: Thanks, trying that at https://chromium-review.googlesource.com/c/chromium/src/+/1384637/
,
Dec 19
mpearson's suggested fix seems to work, https://chromium-review.googlesource.com/c/chromium/src/+/1384637 has a green tsan run. I'll send a separate patch for that and then we might be able to land my original roll CL.
,
Dec 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/63fbcb6ce508234a711686683ac3291b22783966 commit 63fbcb6ce508234a711686683ac3291b22783966 Author: Nico Weber <thakis@chromium.org> Date: Thu Dec 20 01:13:31 2018 Add workaround to make concurrent HistoryDB access less likely. New versions of TSan find a new race in the history db code. Our historydb / sqilte code is known-racy ( https://crbug.com/84094 ), so reorder things a bit to make it less likely that the history db is accessed from two threads at once. See comment in the code for details. Fix suggested by mpearson. Bug: 913702 Change-Id: If88a3a4f981167e2d1ad8c929e7a1b6500a0f3ee Reviewed-on: https://chromium-review.googlesource.com/c/1384637 Commit-Queue: Nico Weber <thakis@chromium.org> Reviewed-by: Mark Pearson <mpearson@chromium.org> Cr-Commit-Position: refs/heads/master@{#618057} [modify] https://crrev.com/63fbcb6ce508234a711686683ac3291b22783966/components/omnibox/browser/autocomplete_controller.cc
,
Dec 20
The tryjob on https://chromium-review.googlesource.com/c/chromium/src/+/1384637/1 confirms it worked. I'll mark this fixed. |
||||
►
Sign in to add a comment |
||||
Comment 1 by thakis@chromium.org
, Dec 10