"TwoClientBookmarksSyncTest.Sanity" is flaky |
||||||
Issue description"TwoClientBookmarksSyncTest.Sanity" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label. We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLAsSBUZsYWtlIiFUd29DbGllbnRCb29rbWFya3NTeW5jVGVzdC5TYW5pdHkM. Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
,
Oct 31 2016
Yup, it's a SetUp() issue. Looks like it gets hung? I can't see any execution times for the tests but it appears they all time out in the same place. BrowserTestBase received signal: Terminated. Backtrace: #0 0x000001cbca7e base::debug::StackTrace::StackTrace() #1 0x00000579a1b9 content::(anonymous namespace)::DumpStackTraceSignalHandler() #2 0x7ff77c7100b0 <unknown> #3 0x7ff77c7c1993 __poll #4 0x7ff782057ff6 <unknown> #5 0x7ff782058124 g_main_context_iteration #6 0x000001cdfd36 base::MessagePumpGlib::Run() #7 0x000001cdd7e1 base::MessageLoop::RunHandler() #8 0x000001d00d50 base::RunLoop::Run() #9 0x000001bfb7fb StatusChangeChecker::StartBlockingWait() #10 0x000001bfb653 StatusChangeChecker::Wait() #11 0x000001bf3da0 ProfileSyncServiceHarness::AwaitQuiescence() #12 0x000001bf3c62 ProfileSyncServiceHarness::AwaitMutualSyncCycleCompletion() #13 0x0000004c7913 TwoClientBookmarksSyncTest_SC_DelFirstBMUnderBMFoldNonEmptyFoldAfterwards_Test::RunTestOnMainThread() #14 0x000001c0d0cf InProcessBrowserTest::RunTestOnMainThreadLoop() #15 0x000005799f35 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #16 0x00000220c844 ChromeBrowserMainParts::PreMainMessageLoopRunImpl() #17 0x00000220b701 ChromeBrowserMainParts::PreMainMessageLoopRun() #18 0x000001322e75 content::BrowserMainLoop::PreMainMessageLoopRun() #19 0x0000016f5f16 content::StartupTaskRunner::RunAllTasksNow() #20 0x000001321401 content::BrowserMainLoop::CreateStartupTasks() #21 0x000001325fa7 content::BrowserMainRunnerImpl::Initialize() #22 0x00000131e858 content::BrowserMain() #23 0x000001bc9a07 content::RunNamedProcessTypeMain() #24 0x000001bca47b content::ContentMainRunnerImpl::Run() #25 0x000001bc8df0 content::ContentMain() #26 0x000005799860 content::BrowserTestBase::SetUp() #27 0x000001c0c54e InProcessBrowserTest::SetUp() #28 0x000001c01ea9 SyncTest::SetUp() #29 0x000001c64f58 testing::Test::Run() #30 0x000001c65a23 testing::TestInfo::Run() #31 0x000001c65eb7 testing::TestCase::Run() #32 0x000001c6d147 testing::internal::UnitTestImpl::RunAllTests() #33 0x000001c6cdaa testing::UnitTest::Run() #34 0x000001ca72c1 base::TestSuite::Run() #35 0x000001c0b9a8 ChromeTestSuiteRunner::RunTestSuite() #36 0x00000579ba52 content::LaunchTests() #37 0x000001bdbb77 main #38 0x7ff77c6fb7ed __libc_start_main
,
Oct 31 2016
Flakes started on Oct 28 very shortly after the CL "https://codereview.chromium.org/2460703005 [Sync] IWYU cleanup" landed. I think it looks related, so let's try to revert that.
,
Oct 31 2016
Issue 660652 has been merged into this issue.
,
Oct 31 2016
Issue 660654 has been merged into this issue.
,
Oct 31 2016
,
Oct 31 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e8d55b3d552a5d23c8c71da157c2f4bbdab56130 commit e8d55b3d552a5d23c8c71da157c2f4bbdab56130 Author: phoglund <phoglund@chromium.org> Date: Mon Oct 31 11:13:51 2016 Revert of [Sync] IWYU cleanup. (patchset #1 id:1 of https://codereview.chromium.org/2460703005/ ) Reason for revert: Likely made sync tests flaky. See https://bugs.chromium.org/p/chromium/issues/detail?id=660655 for more details. BUG= 660655 Original issue's description: > [Sync] IWYU cleanup. > > BUG= > > Committed: https://crrev.com/ef2d347246bfc3cb768a92a5000883379bd51b88 > Cr-Commit-Position: refs/heads/master@{#428559} TBR=maxbogue@chromium.org,skym@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= Review-Url: https://codereview.chromium.org/2466463002 Cr-Commit-Position: refs/heads/master@{#428685} [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/chrome_sync_client.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/chrome_sync_client_unittest.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/profile_sync_service_factory_unittest.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/test/integration/migration_waiter.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/test/integration/multi_client_status_change_checker.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/test/integration/passwords_helper.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/chrome/browser/sync/test/integration/sync_integration_test_util.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/driver/generic_change_processor_unittest.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/driver/model_type_controller.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/driver/model_type_controller_unittest.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine/attachments/on_disk_attachment_store.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine/cycle/sync_cycle_snapshot_unittest.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/loopback_connection_manager.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/loopback_server.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/loopback_server_entity.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/persistent_bookmark_entity.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/persistent_permanent_entity.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/persistent_tombstone_entity.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/loopback_server/persistent_unique_client_entity.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/sync_manager_impl_unittest.cc [modify] https://crrev.com/e8d55b3d552a5d23c8c71da157c2f4bbdab56130/components/sync/engine_impl/test_entry_factory.cc
,
Oct 31 2016
There doesn't seem to be a consistent reason for these flakes. If you look at the failures, some are during client setup, some are during the actual test cases, and some are during cleanup. My CL that was reverted here shouldn't have changed anything, it was only removing duplicate includes in .cc files, which should always have been no-ops because their .h files already included those same things, unless header guards were broken. So the revert probably didn't fix anything. Running TwoClientBookmarksSyncTest.Sanity locally with Max' loopf script >> #!/bin/bash >> counter=0 >> "$@"; while [ $? -eq 0 ]; do >> let counter++ >> "$@"; >> done >> echo "Succeeded $counter times before failing." It eventually failed on the 25th run, during a AwaitMutualSyncCycleCompletion's AwaitQuiescence(). Running again with DVLOG on for some checker classes to see if we can get a little bit information with the following command > loopf out/Default/sync_integration_tests --gtest_filter=TwoClientBookmarksSyncTest.Sanity --enable-logging --vmodule=quiesce_status_change_checker=2,status_change_checker=2 --v=0 It failed on the 14th run at: [50897:50897:1031/101908:FATAL:keep_alive_registry.cc(83)] Check failed: !g_browser_process->IsShuttingDown(). #0 0x7f65a7a0953e base::debug::StackTrace::StackTrace() #1 0x7f65a7a78a6f logging::LogMessage::~LogMessage() #2 0x000001dbb3b1 KeepAliveRegistry::Register() #3 0x000001dc3d9b ScopedKeepAlive::ScopedKeepAlive() #4 0x000003e54e5d Browser::RegisterKeepAlive() #5 0x000003e74647 BrowserList::AddBrowser() #6 0x000003e51ea9 Browser::Browser() #7 0x0000016abf3e SyncTest::InitializeProfile() #8 0x0000016b6262 _ZN4base8internal13FunctorTraitsIM8SyncTestFviP7ProfileEvE6InvokeIPS2_JRKiS4_EEEvS6_OT_DpOT0_ #9 0x0000016b614b _ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIRKM8SyncTestFviP7ProfileEJPS4_RKiS6_EEEvOT_DpOT0_ #10 0x0000016b60c8 _ZN4base8internal7InvokerINS0_9BindStateIM8SyncTestFviP7ProfileEJNS0_17UnretainedWrapperIS3_EEiEEEFvS5_EE7RunImplIRKS7_RKSt5tupleIJS9_iEEJLm0ELm1EEEEvOT_OT0_NS_13IndexSequenceIJXspT1_EEEEOS5_ #11 0x0000016b5fcc _ZN4base8internal7InvokerINS0_9BindStateIM8SyncTestFviP7ProfileEJNS0_17UnretainedWrapperIS3_EEiEEEFvS5_EE3RunEPNS0_13BindStateBaseEOS5_ #12 0x000000768736 base::internal::RunMixin<>::Run() #13 0x0000016b1f6a (anonymous namespace)::SyncProfileDelegate::OnProfileCreated() #14 0x000001b174d4 ProfileImpl::DoFinalInit() #15 0x000001b192ff ProfileImpl::OnLocaleReady() #16 0x000001b1652f ProfileImpl::OnPrefsLoaded() #17 0x000001b1627a ProfileImpl::ProfileImpl() #18 0x000001b1497d Profile::CreateProfile() #19 0x0000016ac763 SyncTest::MakeTestProfile() #20 0x0000016abd66 SyncTest::CreateProfile() #21 0x0000016ad906 SyncTest::SetupClients() #22 0x0000016adf5c SyncTest::SetupSync() #23 0x000000654169 TwoClientBookmarksSyncTest_Sanity_Test::RunTestOnMainThread() ... This might be real, or because I'm trying to use my computer while it spawns new windows that take focus on creation and might be reacting to my accidental keystrokes. I'm worried this is going to take a while to investigate, and may not result in anything actionable. It seems like the approach these integ tests take is fundamentally flaky, see https://cs.chromium.org/chromium/src/chrome/browser/sync/test/integration/quiesce_status_change_checker.h and https://bugs.chromium.org/p/chromium/issues/detail?id=95742#c23 Assigning to current sync bug fixer to continue investigation.
,
Nov 1 2016
Ok. If you see any more flakes, please turn off the tests while investigating so you don't block the CQ for people.
,
Nov 10 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3b8f13960ccb28bf854a91f05878b95529c32d79 commit 3b8f13960ccb28bf854a91f05878b95529c32d79 Author: pnoland <pnoland@chromium.org> Date: Thu Nov 10 22:20:28 2016 [sync] "TwoClientBookmarksSyncTest.Sanity" is flaky Reduce flakiness in TwoClientBookmarksSyncTest by: 1) Reducing the nudge delay for sync to 1 ms 2) Switching to BookmarksMatchChecker.Wait(), which isn't prone to the "stale snapshot" race condition. Reducing the nudge delay applies to and should speed up all the sync integration tests, not just the TwoClientBookmarks ones. Only pairs of AwaitQuiescence|AwaitMutualSyncCycleCompletion AllModelsMatch(Verifier?)() Were converted to use the new pattern. BUG= 660655 R=zea@chromium.org, pavely@chromium.org Review-Url: https://codereview.chromium.org/2479313004 Cr-Commit-Position: refs/heads/master@{#431376} [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/chrome/browser/sync/test/integration/bookmarks_helper.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/chrome/browser/sync/test/integration/bookmarks_helper.h [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/chrome/browser/sync/test/integration/single_client_sessions_sync_test.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/chrome/browser/sync/test/integration/sync_test.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/chrome/browser/sync/test/integration/two_client_bookmarks_sync_test.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/components/sync/driver/glue/sync_backend_host_impl.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/components/sync/driver/sync_driver_switches.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/components/sync/driver/sync_driver_switches.h [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/components/sync/engine/engine_components_factory.h [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/components/sync/engine/engine_components_factory_impl.cc [modify] https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79/components/sync/engine_impl/sync_scheduler_impl.h
,
Nov 11 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe commit 09f9c8e02d8918a23099fc8d7402aa39f24ea1fe Author: hiroshige <hiroshige@chromium.org> Date: Fri Nov 11 08:29:09 2016 Revert of [sync] "TwoClientBookmarksSyncTest.Sanity" is flaky (patchset #2 id:40001 of https://codereview.chromium.org/2479313004/ ) Reason for revert: Suspected to cause TwoClientUssSyncTest.ConflictResolution flaky failures: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=TwoClientUssSyncTest.ConflictResolution&testType=sync_integration_tests Original issue's description: > [sync] "TwoClientBookmarksSyncTest.Sanity" is flaky > > Reduce flakiness in TwoClientBookmarksSyncTest by: > 1) Reducing the nudge delay for sync to 1 ms > 2) Switching to BookmarksMatchChecker.Wait(), which isn't prone to the > "stale snapshot" race condition. > > Reducing the nudge delay applies to and should speed up all the sync > integration tests, not just the TwoClientBookmarks ones. > > Only pairs of > AwaitQuiescence|AwaitMutualSyncCycleCompletion > AllModelsMatch(Verifier?)() > > Were converted to use the new pattern. > > BUG= 660655 > > R=zea@chromium.org, pavely@chromium.org > > Committed: https://crrev.com/3b8f13960ccb28bf854a91f05878b95529c32d79 > Cr-Commit-Position: refs/heads/master@{#431376} TBR=pavely@chromium.org,zea@chromium.org,pnoland@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 660655 Review-Url: https://codereview.chromium.org/2494633005 Cr-Commit-Position: refs/heads/master@{#431525} [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/chrome/browser/sync/test/integration/bookmarks_helper.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/chrome/browser/sync/test/integration/bookmarks_helper.h [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/chrome/browser/sync/test/integration/single_client_sessions_sync_test.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/chrome/browser/sync/test/integration/sync_test.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/chrome/browser/sync/test/integration/two_client_bookmarks_sync_test.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/components/sync/driver/glue/sync_backend_host_impl.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/components/sync/driver/sync_driver_switches.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/components/sync/driver/sync_driver_switches.h [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/components/sync/engine/engine_components_factory.h [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/components/sync/engine/engine_components_factory_impl.cc [modify] https://crrev.com/09f9c8e02d8918a23099fc8d7402aa39f24ea1fe/components/sync/engine_impl/sync_scheduler_impl.h
,
Nov 14 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dcdd97e380a2f5125c55774f486eab1569625edd commit dcdd97e380a2f5125c55774f486eab1569625edd Author: pnoland <pnoland@chromium.org> Date: Mon Nov 14 23:01:21 2016 [sync] Reland of "TwoClientBookmarksSyncTest.Sanity" is flaky Fix race condition in TwoClientUssSyncTest.ConflictResolution that was exposed by the speedup BUG= 660655 R=zea@chromium.org Review-Url: https://codereview.chromium.org/2497533003 Cr-Commit-Position: refs/heads/master@{#431944} [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/chrome/browser/sync/test/integration/bookmarks_helper.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/chrome/browser/sync/test/integration/bookmarks_helper.h [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/chrome/browser/sync/test/integration/single_client_sessions_sync_test.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/chrome/browser/sync/test/integration/sync_test.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/chrome/browser/sync/test/integration/two_client_bookmarks_sync_test.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/chrome/browser/sync/test/integration/two_client_uss_sync_test.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/components/sync/driver/glue/sync_backend_host_impl.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/components/sync/driver/sync_driver_switches.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/components/sync/driver/sync_driver_switches.h [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/components/sync/engine/engine_components_factory.h [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/components/sync/engine/engine_components_factory_impl.cc [modify] https://crrev.com/dcdd97e380a2f5125c55774f486eab1569625edd/components/sync/engine_impl/sync_scheduler_impl.h
,
Nov 17 2016
,
May 8 2017
Issue 523166 has been merged into this issue. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by phoglund@chromium.org
, Oct 31 2016