Issue metadata
Sign in to add a comment
|
ExistingUserControllerActiveDirectoryTest.PolicyChangeTriggersFileUpdate and UserKerberosFilesChangedSignalTriggersFileUpdate flaky on linux-chromeos-rel |
||||||||||||||||||||
Issue descriptionStarted flaking extensively recently. A large percentage of the tryjobs on linux-chromeos-rel are failing at least one browser_tests variant failing one of these tests. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel?limit=200 A few examples: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50278 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50522 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50547 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50538 Flakiness dashboard: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=ExistingUserControllerActiveDirectoryTest.PolicyChangeTriggersFileUpdate Example shard from https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50522 : https://chromium-swarm.appspot.com/task?id=3ec83174b0a6a010&refresh=10&show_raw=1 Stack: [2346:2346:0718/122626.380441:FATAL:barrier_closure.cc(32)] Check failed: !num_callbacks_left_.IsZero(). #0 0x000004b175fc base::debug::StackTrace::StackTrace() #1 0x000004a7f7cb logging::LogMessage::~LogMessage() #2 0x000004a69722 base::(anonymous namespace)::BarrierInfo::Run() #3 0x0000015df15e _ZN4base8internal7InvokerINS0_9BindStateIZN8chromeos12_GLOBAL__N_125KerberosFilesChangeWaiterC1EbEUlRKNS_17RepeatingCallbackIFvvEEERKNS_8FilePathEbE_JS8_EEEFvSD_bEE3RunEPNS0_13BindStateBaseESD_b #4 0x000004a7649d base::(anonymous namespace)::FilePathWatcherImpl::OnFilePathChangedOnOriginSequence() #5 0x000004a76661 _ZN4base8internal7InvokerINS0_9BindStateIMNS_12_GLOBAL__N_119FilePathWatcherImplEFviRKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEEbbbEJNS_7WeakPtrIS4_EEiSB_bbbEEEFvvEE7RunOnceEPNS0_13BindStateBaseE #6 0x000004b38995 base::debug::TaskAnnotator::RunTask() #7 0x000004a87dea base::MessageLoop::RunTask() #8 0x000004a8817a base::MessageLoop::DeferOrRunPendingTask() #9 0x000004a88477 base::MessageLoop::DoWork() #10 0x000004b33619 base::MessagePumpLibevent::Run() #11 0x000004a87814 base::MessageLoop::Run() #12 0x000004aae2b9 base::RunLoop::Run() #13 0x000004aae83a base::RunLoop::RunUntilIdle() #14 0x000009d657b7 policy::MockConfigurationPolicyProvider::UpdateChromePolicy() #15 0x0000015d80d6 chromeos::ExistingUserControllerActiveDirectoryTest::ApplyPolicyAndWaitFilesChanged() #16 0x0000015d7f38 chromeos::ExistingUserControllerActiveDirectoryTest_PolicyChangeTriggersFileUpdate_Test::RunTestOnMainThread() #17 0x0000050eec91 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #18 0x000004bf55ab ChromeBrowserMainParts::PreMainMessageLoopRunImpl() #19 0x000004bf439a ChromeBrowserMainParts::PreMainMessageLoopRun() #20 0x000001bdaf4a chromeos::ChromeBrowserMainPartsChromeos::PreMainMessageLoopRun() #21 0x000002da67b1 content::BrowserMainLoop::PreMainMessageLoopRun() #22 0x000003227845 content::StartupTaskRunner::RunAllTasksNow() #23 0x000002da50b8 content::BrowserMainLoop::CreateStartupTasks() #24 0x000002da8e23 content::BrowserMainRunnerImpl::Initialize() #25 0x000002da2c42 content::BrowserMain() #26 0x0000049ddb80 content::ContentMainRunnerImpl::Run() #27 0x000006aeacb5 service_manager::Main() #28 0x0000049dbe84 content::ContentMain() #29 0x0000050ee82c content::BrowserTestBase::SetUp() #30 0x000004b9605f InProcessBrowserTest::SetUp() #31 0x0000016c719c policy::DevicePolicyCrosBrowserTest::SetUp() This is causing a large number of retries on the CQ. Marking P0. Must be triaged immediately; possibly a code change caused this test to begin flaking.
,
Jul 18
,
Jul 19
If noone's done it yet, I'll go ahead and disable these tests to unblock the CQ.
,
Jul 19
CL to disable tests is in the CQ: https://chromium-review.googlesource.com/c/chromium/src/+/1142702
,
Jul 19
This is also failing very consistently on linux-chromeos-dbg since https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/linux-chromeos-dbg/6889. Don't see an obvious culprit in the blamelist though. rsorokin@: since you touched these tests last, would you mind triaging this further?
,
Jul 19
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c915f6568b03531c68550c88c631a0623dcd347a commit c915f6568b03531c68550c88c631a0623dcd347a Author: Alex Moshchuk <alexmos@chromium.org> Date: Thu Jul 19 00:37:13 2018 Disable two flaky ExistingUserControllerActiveDirectoryTests. Bug: 865206 Change-Id: Id94f5a610b31a5e63c494030a792fe168583d95d Tbr: rsorokin@chromium.org Reviewed-on: https://chromium-review.googlesource.com/1142702 Reviewed-by: Alex Moshchuk <alexmos@chromium.org> Commit-Queue: Alex Moshchuk <alexmos@chromium.org> Cr-Commit-Position: refs/heads/master@{#576277} [modify] https://crrev.com/c915f6568b03531c68550c88c631a0623dcd347a/chrome/browser/chromeos/login/existing_user_controller_browsertest.cc
,
Jul 19
alexmos@: Thanks for landing the CL that disables the tests! FWIW, I was also unable to find a culprit. Maybe it was an inherent flakiness that only now became apparent due to some unrelated change in scheduling (e.g., r576079)? Basically, the crashing code in KerberosFilesChangeWaiter doesn't expect that notifications about the watched files arrive more than once. So either this shouldn't happen indeed and the root cause is elsewhere, or the test code should be fixed to ignore subsequent notifications. Anyway, rsorokin@ seems to be the right assignee to continue the investigation there.
,
Jul 19
reducing priority as the test has been disabled
,
Jul 19
Thanks, I'll take a look
,
Jul 19
,
Jul 20
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7a20ffc4e1f5cbb57336f2c5254f61eba56c231e commit 7a20ffc4e1f5cbb57336f2c5254f61eba56c231e Author: Roman Sorokin <rsorokin@chromium.org> Date: Fri Jul 20 19:05:12 2018 Chromad: Fix kerberos files tests Modify KerberosFilesChangeWaiter to be more robust callbacks called several times BUG=chromium:865206 Change-Id: I0c5f22fb26871607257d3b29ad3bd88d5b505c7c Reviewed-on: https://chromium-review.googlesource.com/1145074 Commit-Queue: Alexander Alekseev <alemate@chromium.org> Reviewed-by: Alexander Alekseev <alemate@chromium.org> Cr-Commit-Position: refs/heads/master@{#576941} [modify] https://crrev.com/7a20ffc4e1f5cbb57336f2c5254f61eba56c231e/chrome/browser/chromeos/login/existing_user_controller_browsertest.cc
,
Jul 23
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0acc75bf23ecbf90e4ddb6860d155b26241d2128 commit 0acc75bf23ecbf90e4ddb6860d155b26241d2128 Author: Yoichi Osato <yoichio@chromium.org> Date: Thu Jul 26 04:46:05 2018 Revert "Chromad: Fix kerberos files tests" This reverts commit 7a20ffc4e1f5cbb57336f2c5254f61eba56c231e. Reason for revert: Still flaky https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyUwsSBUZsYWtlIkhFeGlzdGluZ1VzZXJDb250cm9sbGVyQWN0aXZlRGlyZWN0b3J5VGVzdC5Qb2xpY3lDaGFuZ2VUcmlnZ2Vyc0ZpbGVVcGRhdGUM Original change's description: > Chromad: Fix kerberos files tests > > Modify KerberosFilesChangeWaiter to be more robust callbacks called > several times > > BUG=chromium:865206 > > Change-Id: I0c5f22fb26871607257d3b29ad3bd88d5b505c7c > Reviewed-on: https://chromium-review.googlesource.com/1145074 > Commit-Queue: Alexander Alekseev <alemate@chromium.org> > Reviewed-by: Alexander Alekseev <alemate@chromium.org> > Cr-Commit-Position: refs/heads/master@{#576941} TBR=alemate@chromium.org,rsorokin@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 865206, 867386 Change-Id: I97bc01ed73dcd0dcc7c0fa3735f40548edc92a81 Reviewed-on: https://chromium-review.googlesource.com/1150061 Commit-Queue: Yoichi Osato <yoichio@chromium.org> Reviewed-by: Yoichi Osato <yoichio@chromium.org> Cr-Commit-Position: refs/heads/master@{#578207} [modify] https://crrev.com/0acc75bf23ecbf90e4ddb6860d155b26241d2128/chrome/browser/chromeos/login/existing_user_controller_browsertest.cc
,
Aug 20
,
Aug 20
Issue 867386 has been merged into this issue.
,
Aug 20
FYI in case it helps: ExistingUserControllerActiveDirectoryTest.PolicyChangeTriggersFileUpdate was already flaky when it was added https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVy1gELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKfAWNocm9taXVtLm1lbW9yeS9MaW51eCBDaHJvbWl1bU9TIE1TYW4gVGVzdHMvNzA3MS9icm93c2VyX3Rlc3RzL1JYaHBjM1JwYm1kVmMyVnlRMjl1ZEhKdmJHeGxja0ZqZEdsMlpVUnBjbVZqZEc5eWVWUmxjM1F1VUc5c2FXTjVRMmhoYm1kbFZISnBaMmRsY25OR2FXeGxWWEJrWVhSbAwLEhNNYXN0ZXJGbGFrZUFuYWx5c2lzGAEM
,
Dec 12
|
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by sky@chromium.org
, Jul 18Components: Enterprise