New issue
Advanced search Search tips

Issue 801700 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: 2018-01-12
OS: Chrome
Pri: 1
Type: Bug
Team-Accessibility



Sign in to add a comment

TrayAccessibilityTestInstance/TrayAccessibilityTest.ShowNotification/0 has flaky crashes

Project Member Reported by jamescook@chromium.org, Jan 12 2018

Issue description

It's only flaky on the bots, I can't repro locally. It looks like something is leaving open a window at the end of the test which causes a crash during ash::Shell teardown. It might be a notification, or it might be the ChromeVoxPanel widget.

It hits both browser_tests and mus_browser_tests.
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=TrayAccessibilityTestInstance%2FTrayAccessibilityTest.ShowNotification%2F0

I'm not sure when it started because it usually passes after retries.

The only thing I can think of that changed recently is the AshShellInit changes.

I'm tempted to try a hacky fix (make sure notification tray is empty and that ChromeVoxPanel is closed at the end of the test) instead of disabling.

BrowserTestBase received signal: Segmentation fault. Backtrace:
#0 0x0000033ddadc base::debug::StackTrace::StackTrace()
#1 0x0000038c2bb2 content::(anonymous namespace)::DumpStackTraceSignalHandler()
#2 0x7f9f84bdccb0 <unknown>
#3 0x000003fadd9e views::Widget::OnNativeWidgetDestroying()
#4 0x000003fbb6b0 views::NativeWidgetAura::OnWindowDestroying()
#5 0x000004f4f504 aura::Window::~Window()
#6 0x000004f4fd9e aura::Window::~Window()
#7 0x0000059e7548 ash::RootWindowController::CloseChildWindows()
#8 0x000005a082c8 ash::Shell::CloseAllRootWindowChildWindows()
#9 0x000005a06d07 ash::Shell::~Shell()
#10 0x000005a0836e ash::Shell::~Shell()
#11 0x0000079fd6ed ash::WindowManager::~WindowManager()
#12 0x0000079fd8ae ash::WindowManager::~WindowManager()
#13 0x000005cdad7d ChromeBrowserMainExtraPartsAsh::PostMainMessageLoopRun()
#14 0x0000034d833a ChromeBrowserMainParts::PostMainMessageLoopRun()
#15 0x000001884e61 chromeos::ChromeBrowserMainPartsChromeos::PostMainMessageLoopRun()
#16 0x0000022cf9e8 content::BrowserMainLoop::ShutdownThreadsAndCleanUp()
#17 0x0000022d22ef content::BrowserMainRunnerImpl::Shutdown()
#18 0x0000022cba28 content::BrowserMain()
#19 0x0000033ba6b5 content::ContentMainRunnerImpl::Run()
#20 0x0000050964b6 service_manager::Main()
#21 0x0000033b9214 content::ContentMain()
#22 0x0000038c25dd content::BrowserTestBase::SetUp()
#23 0x00000349859f InProcessBrowserTest::SetUp()
#24 0x000001b64ef1 testing::Test::Run()
#25 0x000001b65a20 testing::TestInfo::Run()
#26 0x000001b65f07 testing::TestCase::Run()
#27 0x000001b6c527 testing::internal::UnitTestImpl::RunAllTests()
#28 0x000001b6c177 testing::UnitTest::Run()
#29 0x0000034ac6d2 base::TestSuite::Run()
#30 0x0000033d37d9 ChromeTestSuiteRunner::RunTestSuite()
#31 0x0000033d374d (anonymous namespace)::MusTestLauncherDelegate::RunTestSuite()
#32 0x0000038f2ee7 content::LaunchTests()
#33 0x0000033d3ca4 LaunchChromeTests()
#34 0x0000033d34f6 RunMashBrowserTests()
#35 0x0000033d33a7 main

(Similar stack on non-mash/non-mus):

BrowserTestBase received signal: Segmentation fault. Backtrace:
#0 0x000003e96c5c base::debug::StackTrace::StackTrace()
#1 0x0000044d9b12 content::(anonymous namespace)::DumpStackTraceSignalHandler()
#2 0x7fd5c2bbacb0 <unknown>
#3 0x000004df21b2 views::Widget::OnNativeWidgetDestroying()
#4 0x000004e00940 views::NativeWidgetAura::OnWindowDestroying()
#5 0x000005f4cbfc aura::Window::~Window()
#6 0x000005f4d2ce aura::Window::~Window()
#7 0x000006c08936 ash::RootWindowController::CloseChildWindows()
#8 0x000006c2dc08 ash::Shell::CloseAllRootWindowChildWindows()
#9 0x000006c2c781 ash::Shell::~Shell()
#10 0x000006c2dcae ash::Shell::~Shell()
#11 0x00000717a508 AshShellInit::~AshShellInit()
#12 0x000006f7684d ChromeBrowserMainExtraPartsAsh::PostMainMessageLoopRun()
#13 0x000003fcb88a ChromeBrowserMainParts::PostMainMessageLoopRun()
#14 0x000001908961 chromeos::ChromeBrowserMainPartsChromeos::PostMainMessageLoopRun()
#15 0x0000026d189d content::BrowserMainLoop::ShutdownThreadsAndCleanUp()
#16 0x0000026d4873 content::BrowserMainRunnerImpl::Shutdown()
#17 0x0000026cd526 content::BrowserMain()

 

Comment 1 by dtseng@chromium.org, Jan 12 2018

NextAction: 2018-01-12
I fixed something similar in
https://chromium-review.googlesource.com/c/chromium/src/+/838180
Project Member

Comment 2 by bugdroid1@chromium.org, Jan 12 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/00cf7f8cd182b9479615b2edcebf3ce3e4514561

commit 00cf7f8cd182b9479615b2edcebf3ce3e4514561
Author: James Cook <jamescook@chromium.org>
Date: Fri Jan 12 22:36:55 2018

cros: Disable TrayAccessibilityTest.ShowNotification

It has been causing flaky crashes on the trybots for at least 3 days and
possibly much longer.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=TrayAccessibilityTestInstance%2FTrayAccessibilityTest.ShowNotification%2F0

https://chromium-swarm.appspot.com/task?id=3b04a39265d53210&refresh=10&show_raw=1

TBR=yiyix@chromium.org

Bug:  801700 
Change-Id: I35a2db4510e7b9eb977ba3e2dbe6d8bb6385d79d
Reviewed-on: https://chromium-review.googlesource.com/865635
Reviewed-by: James Cook <jamescook@chromium.org>
Reviewed-by: Yi Xu <yiyix@chromium.org>
Commit-Queue: James Cook <jamescook@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529090}
[modify] https://crrev.com/00cf7f8cd182b9479615b2edcebf3ce3e4514561/chrome/browser/chromeos/system/tray_accessibility_browsertest.cc

The ChromeVoxPanel is sometimes being opened twice, the second time during test shutdown. This causes the single ChromeVoxPanelWidgetObserver to be deleted, so OnNativeWidgetDestroying calls into a deleted observer.

I think the cause is a race between test shutdown with NOTIFICATION_APP_TERMINATING and the component extension loader. In particular, if the load completes after NOTIFICATION_APP_TERMINATING then it sees AccessibilityManager::chromevox_panel_ null and opens a second ChromeVoxPanel. This behavior was added in https://chromium-review.googlesource.com/c/chromium/src/+/838180 in late December, so it's probably been flaky since then.

Fix out for review at https://chromium-review.googlesource.com/865615

Project Member

Comment 4 by bugdroid1@chromium.org, Jan 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0c4f37ac0d5713ca046b7eedc99c68592137078f

commit 0c4f37ac0d5713ca046b7eedc99c68592137078f
Author: James Cook <jamescook@chromium.org>
Date: Wed Jan 17 02:17:09 2018

cros: Fix flaky crashes in TrayAccessibilityTest.ShowNotification

The ChromeVoxPanel is sometimes being opened twice, the second time
during test shutdown. This causes the ChromeVoxPanelWidgetObserver
to be deleted, so OnNativeWidgetDestroying for the second panel calls
into a deleted observer.

The root cause is a race between test shutdown and the component
extension loader. In particular, if the load completes after
NOTIFICATION_APP_TERMINATING then it sees that
AccessibilityManager::chromevox_panel_ is null and opens a second
ChromeVoxPanel. Nulling out chromevox_panel_ during shutdown was
added in http://crrev.com/c/838180 in late December to fix a shutdown
crash, so the test has probably been flaky since then.

Instead, explicitly avoid creating new ChromeVox panels during
shutdown, both due to screen unlock/shutdown and due to late loading
of the component extension.

Bug:  801700 
Test: browser_tests --gtest_filter=*TrayAccessibilityTest.ShowNotication* --gtest_repeat=10 --gtest_shuffle
Change-Id: Ic313b440669a3097ad100e8f4b77404f01112fa0
Reviewed-on: https://chromium-review.googlesource.com/865615
Reviewed-by: David Tseng <dtseng@chromium.org>
Commit-Queue: James Cook <jamescook@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529548}
[modify] https://crrev.com/0c4f37ac0d5713ca046b7eedc99c68592137078f/chrome/browser/chromeos/accessibility/accessibility_manager.cc
[modify] https://crrev.com/0c4f37ac0d5713ca046b7eedc99c68592137078f/chrome/browser/chromeos/accessibility/accessibility_manager.h
[modify] https://crrev.com/0c4f37ac0d5713ca046b7eedc99c68592137078f/chrome/browser/chromeos/accessibility/chromevox_panel.cc
[modify] https://crrev.com/0c4f37ac0d5713ca046b7eedc99c68592137078f/chrome/browser/chromeos/system/tray_accessibility_browsertest.cc

Status: Fixed (was: Started)
Flakiness dashboard shows the crashes have gone away.

Sign in to add a comment