New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 865206 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug-Regression



Sign in to add a comment

ExistingUserControllerActiveDirectoryTest.PolicyChangeTriggersFileUpdate and UserKerberosFilesChangedSignalTriggersFileUpdate flaky on linux-chromeos-rel

Project Member Reported by kbr@chromium.org, Jul 18

Issue description

Started flaking extensively recently. A large percentage of the tryjobs on linux-chromeos-rel are failing at least one browser_tests variant failing one of these tests.

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel?limit=200

A few examples:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50278
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50522
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50547
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50538

Flakiness dashboard:

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=ExistingUserControllerActiveDirectoryTest.PolicyChangeTriggersFileUpdate

Example shard from https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/50522 :

https://chromium-swarm.appspot.com/task?id=3ec83174b0a6a010&refresh=10&show_raw=1

Stack:

[2346:2346:0718/122626.380441:FATAL:barrier_closure.cc(32)] Check failed: !num_callbacks_left_.IsZero(). 
#0 0x000004b175fc base::debug::StackTrace::StackTrace()
#1 0x000004a7f7cb logging::LogMessage::~LogMessage()
#2 0x000004a69722 base::(anonymous namespace)::BarrierInfo::Run()
#3 0x0000015df15e _ZN4base8internal7InvokerINS0_9BindStateIZN8chromeos12_GLOBAL__N_125KerberosFilesChangeWaiterC1EbEUlRKNS_17RepeatingCallbackIFvvEEERKNS_8FilePathEbE_JS8_EEEFvSD_bEE3RunEPNS0_13BindStateBaseESD_b
#4 0x000004a7649d base::(anonymous namespace)::FilePathWatcherImpl::OnFilePathChangedOnOriginSequence()
#5 0x000004a76661 _ZN4base8internal7InvokerINS0_9BindStateIMNS_12_GLOBAL__N_119FilePathWatcherImplEFviRKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEEbbbEJNS_7WeakPtrIS4_EEiSB_bbbEEEFvvEE7RunOnceEPNS0_13BindStateBaseE
#6 0x000004b38995 base::debug::TaskAnnotator::RunTask()
#7 0x000004a87dea base::MessageLoop::RunTask()
#8 0x000004a8817a base::MessageLoop::DeferOrRunPendingTask()
#9 0x000004a88477 base::MessageLoop::DoWork()
#10 0x000004b33619 base::MessagePumpLibevent::Run()
#11 0x000004a87814 base::MessageLoop::Run()
#12 0x000004aae2b9 base::RunLoop::Run()
#13 0x000004aae83a base::RunLoop::RunUntilIdle()
#14 0x000009d657b7 policy::MockConfigurationPolicyProvider::UpdateChromePolicy()
#15 0x0000015d80d6 chromeos::ExistingUserControllerActiveDirectoryTest::ApplyPolicyAndWaitFilesChanged()
#16 0x0000015d7f38 chromeos::ExistingUserControllerActiveDirectoryTest_PolicyChangeTriggersFileUpdate_Test::RunTestOnMainThread()
#17 0x0000050eec91 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop()
#18 0x000004bf55ab ChromeBrowserMainParts::PreMainMessageLoopRunImpl()
#19 0x000004bf439a ChromeBrowserMainParts::PreMainMessageLoopRun()
#20 0x000001bdaf4a chromeos::ChromeBrowserMainPartsChromeos::PreMainMessageLoopRun()
#21 0x000002da67b1 content::BrowserMainLoop::PreMainMessageLoopRun()
#22 0x000003227845 content::StartupTaskRunner::RunAllTasksNow()
#23 0x000002da50b8 content::BrowserMainLoop::CreateStartupTasks()
#24 0x000002da8e23 content::BrowserMainRunnerImpl::Initialize()
#25 0x000002da2c42 content::BrowserMain()
#26 0x0000049ddb80 content::ContentMainRunnerImpl::Run()
#27 0x000006aeacb5 service_manager::Main()
#28 0x0000049dbe84 content::ContentMain()
#29 0x0000050ee82c content::BrowserTestBase::SetUp()
#30 0x000004b9605f InProcessBrowserTest::SetUp()
#31 0x0000016c719c policy::DevicePolicyCrosBrowserTest::SetUp()


This is causing a large number of retries on the CQ. Marking P0. Must be triaged immediately; possibly a code change caused this test to begin flaking.

 
Cc: atwilson@chromium.org emaxx@chromium.org
Components: Enterprise
+some policy folks.
Cc: rsorokin@chromium.org ljusten@chromium.org
If noone's done it yet, I'll go ahead and disable these tests to unblock the CQ.
Cc: -rsorokin@chromium.org
Owner: rsorokin@chromium.org
Status: Assigned (was: Untriaged)
This is also failing very consistently on linux-chromeos-dbg since https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/linux-chromeos-dbg/6889.  Don't see an obvious culprit in the blamelist though.  rsorokin@: since you touched these tests last, would you mind triaging this further?
Project Member

Comment 6 by bugdroid1@chromium.org, Jul 19

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c915f6568b03531c68550c88c631a0623dcd347a

commit c915f6568b03531c68550c88c631a0623dcd347a
Author: Alex Moshchuk <alexmos@chromium.org>
Date: Thu Jul 19 00:37:13 2018

Disable two flaky ExistingUserControllerActiveDirectoryTests.

Bug: 865206
Change-Id: Id94f5a610b31a5e63c494030a792fe168583d95d
Tbr: rsorokin@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/1142702
Reviewed-by: Alex Moshchuk <alexmos@chromium.org>
Commit-Queue: Alex Moshchuk <alexmos@chromium.org>
Cr-Commit-Position: refs/heads/master@{#576277}
[modify] https://crrev.com/c915f6568b03531c68550c88c631a0623dcd347a/chrome/browser/chromeos/login/existing_user_controller_browsertest.cc

alexmos@: Thanks for landing the CL that disables the tests!

FWIW, I was also unable to find a culprit. Maybe it was an inherent flakiness that only now became apparent due to some unrelated change in scheduling (e.g., r576079)? Basically, the crashing code in KerberosFilesChangeWaiter doesn't expect that notifications about the watched files arrive more than once. So either this shouldn't happen indeed and the root cause is elsewhere, or the test code should be fixed to ignore subsequent notifications.

Anyway, rsorokin@ seems to be the right assignee to continue the investigation there.
Labels: -Pri-0 Pri-2
reducing priority as the test has been disabled
Thanks, I'll take a look
Labels: -Sheriff-Chromium
Status: Started (was: Assigned)
Project Member

Comment 12 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7a20ffc4e1f5cbb57336f2c5254f61eba56c231e

commit 7a20ffc4e1f5cbb57336f2c5254f61eba56c231e
Author: Roman Sorokin <rsorokin@chromium.org>
Date: Fri Jul 20 19:05:12 2018

Chromad: Fix kerberos files tests

Modify KerberosFilesChangeWaiter to be more robust callbacks called
several times

BUG=chromium:865206

Change-Id: I0c5f22fb26871607257d3b29ad3bd88d5b505c7c
Reviewed-on: https://chromium-review.googlesource.com/1145074
Commit-Queue: Alexander Alekseev <alemate@chromium.org>
Reviewed-by: Alexander Alekseev <alemate@chromium.org>
Cr-Commit-Position: refs/heads/master@{#576941}
[modify] https://crrev.com/7a20ffc4e1f5cbb57336f2c5254f61eba56c231e/chrome/browser/chromeos/login/existing_user_controller_browsertest.cc

Status: Fixed (was: Started)
Project Member

Comment 14 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0acc75bf23ecbf90e4ddb6860d155b26241d2128

commit 0acc75bf23ecbf90e4ddb6860d155b26241d2128
Author: Yoichi Osato <yoichio@chromium.org>
Date: Thu Jul 26 04:46:05 2018

Revert "Chromad: Fix kerberos files tests"

This reverts commit 7a20ffc4e1f5cbb57336f2c5254f61eba56c231e.

Reason for revert: Still flaky
https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyUwsSBUZsYWtlIkhFeGlzdGluZ1VzZXJDb250cm9sbGVyQWN0aXZlRGlyZWN0b3J5VGVzdC5Qb2xpY3lDaGFuZ2VUcmlnZ2Vyc0ZpbGVVcGRhdGUM

Original change's description:
> Chromad: Fix kerberos files tests
>
> Modify KerberosFilesChangeWaiter to be more robust callbacks called
> several times
>
> BUG=chromium:865206
>
> Change-Id: I0c5f22fb26871607257d3b29ad3bd88d5b505c7c
> Reviewed-on: https://chromium-review.googlesource.com/1145074
> Commit-Queue: Alexander Alekseev <alemate@chromium.org>
> Reviewed-by: Alexander Alekseev <alemate@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#576941}

TBR=alemate@chromium.org,rsorokin@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 865206,  867386 
Change-Id: I97bc01ed73dcd0dcc7c0fa3735f40548edc92a81
Reviewed-on: https://chromium-review.googlesource.com/1150061
Commit-Queue: Yoichi Osato <yoichio@chromium.org>
Reviewed-by: Yoichi Osato <yoichio@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578207}
[modify] https://crrev.com/0acc75bf23ecbf90e4ddb6860d155b26241d2128/chrome/browser/chromeos/login/existing_user_controller_browsertest.cc

Status: Assigned (was: Fixed)
 Issue 867386  has been merged into this issue.
Components: -UI>Shell>StartScreen

Sign in to add a comment