New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 719030 link

Starred by 9 users

Issue metadata

Status: WontFix
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows , Mac
Pri: 2
Type: Bug
Team-Accessibility

Blocking:
issue 622387



Sign in to add a comment

DumpAccessibilityEventsTest tests disabled or flaky

Project Member Reported by dmazz...@chromium.org, May 5 2017

Issue description

This bug consolidates issues with the DumpAccessibilityEventsTest suite of accessibility tests.

Currently there are 30 tests, and only 22 run on Windows. Some of those tests are still experiencing some flakiness.

Looking at the flakiness and failures, it looks like there are a few root causes that can be addressed to make these tests robust. Let's try to re-enable them all and make it possible to add new ones.

 
 Issue 652706  has been merged into this issue.
Cc: lpalmaro@chromium.org nek...@chromium.org ellyjo...@chromium.org dmazz...@chromium.org
 Issue 615411  has been merged into this issue.
 Issue 598527  has been merged into this issue.
Cc: robliao@chromium.org kpschoedel@chromium.org moh...@chromium.org
 Issue 486861  has been merged into this issue.
Cc: jam@chromium.org dcheng@chromium.org dtseng@chromium.org
 Issue 588271  has been merged into this issue.
Project Member

Comment 6 by bugdroid1@chromium.org, May 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d252421685e84c1f262f200ef54f79d159be47a7

commit d252421685e84c1f262f200ef54f79d159be47a7
Author: dmazzoni <dmazzoni@chromium.org>
Date: Wed May 10 15:56:15 2017

Fix primary cause of flakiness of DumpAccessibilityEvents tests on Win

Normally we suppress focus events when the native window isn't itself
focused. This was causing test flakiness for tests that specifically
listen for those focus events; add a way to suppress them to make the
tests more robust.

BUG= 719030 

Review-Url: https://codereview.chromium.org/2863833003
Cr-Commit-Position: refs/heads/master@{#470589}

[modify] https://crrev.com/d252421685e84c1f262f200ef54f79d159be47a7/content/browser/accessibility/browser_accessibility_manager.cc
[modify] https://crrev.com/d252421685e84c1f262f200ef54f79d159be47a7/content/browser/accessibility/browser_accessibility_manager.h
[modify] https://crrev.com/d252421685e84c1f262f200ef54f79d159be47a7/content/browser/accessibility/dump_accessibility_browsertest_base.cc

Project Member

Comment 7 by bugdroid1@chromium.org, May 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19776ea6b20f56507e080f09a8b508c3ec6ba5be

commit 19776ea6b20f56507e080f09a8b508c3ec6ba5be
Author: dmazzoni <dmazzoni@chromium.org>
Date: Fri May 12 00:22:39 2017

Re-enable 3 DumpAccessibilityEvents tests that were disabled on windows.

They all had the same underlying cause: we were firing accessibility events
based on changes to the accessibility tree, but before we had finished
making changes to BrowserAccessibilityWin. To fix this, queue up events
caused by tree changes and fire them after the tree is loaded.

We should probably use this technique more in the future. For now this
just re-enables some good tests.

BUG= 719030 

Review-Url: https://codereview.chromium.org/2857743006
Cr-Commit-Position: refs/heads/master@{#471140}

[modify] https://crrev.com/19776ea6b20f56507e080f09a8b508c3ec6ba5be/content/browser/accessibility/browser_accessibility_manager.cc
[modify] https://crrev.com/19776ea6b20f56507e080f09a8b508c3ec6ba5be/content/browser/accessibility/browser_accessibility_manager.h
[modify] https://crrev.com/19776ea6b20f56507e080f09a8b508c3ec6ba5be/content/browser/accessibility/dump_accessibility_events_browsertest.cc
[modify] https://crrev.com/19776ea6b20f56507e080f09a8b508c3ec6ba5be/content/test/data/accessibility/event/aria-combo-box-expand-expected-win.txt
[modify] https://crrev.com/19776ea6b20f56507e080f09a8b508c3ec6ba5be/content/test/data/accessibility/event/aria-combo-box-next-expected-win.txt

Project Member

Comment 8 by bugdroid1@chromium.org, May 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2751e50876e2591e8268c124001b58dabfab6828

commit 2751e50876e2591e8268c124001b58dabfab6828
Author: dmazzoni <dmazzoni@chromium.org>
Date: Fri May 26 09:03:10 2017

Fix DumpAccessibilityEvents tests on Mac and improve test coverage

Fix some minor issues that caused DumpAccessibilityEvents tests to fail
or behave flakily. Re-enable several tests and add many more with a
goal of covering all possible Mac notifications with tests.

This is part of a larger effort to fire fewer events from Blink
and fire them in platform-specific code instead. Increasing
test coverage will help with this.

BUG= 719030 ,699438

Review-Url: https://codereview.chromium.org/2897943003
Cr-Commit-Position: refs/heads/master@{#474971}

[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/browser/accessibility/accessibility_event_recorder_mac.mm
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/browser/accessibility/browser_accessibility_manager.cc
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/browser/accessibility/browser_accessibility_manager.h
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/browser/accessibility/browser_accessibility_manager_mac.mm
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/browser/accessibility/dump_accessibility_browsertest_base.cc
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/browser/accessibility/dump_accessibility_events_browsertest.cc
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-combo-box-collapse.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-combo-box-expand.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-combo-box-focus-expected-win.txt
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-combo-box-focus.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-combo-box-next.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-tree-collapse-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-tree-collapse.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-tree-expand-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-tree-expand.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-treeitem-focus-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/aria-treeitem-focus.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/expanded-change.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/input-type-text-value-changed.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/invalid-status-change-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/invalid-status-change.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/listbox-focus.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/listbox-next.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-add-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-add.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-change-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-change.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-create-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-create.html
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-remove-expected-mac.txt
[add] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/live-region-remove.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/menulist-collapse.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/menulist-expand.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/menulist-focus.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/menulist-next.html
[modify] https://crrev.com/2751e50876e2591e8268c124001b58dabfab6828/content/test/data/accessibility/event/menulist-popup.html

Status: Fixed (was: Started)
Fixed for now

Labels: Sheriff-Chromium
Status: Assigned (was: Fixed)
Hmm, DumpAccessibilityEventsTest seems still flaky.
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests&tests=DumpAccessibilityEventsTest

Could you investigate it?
Labels: -Sheriff-Chromium
Removing Sheriff-Chromium now that the bug is assigned and being investigated.
Labels: a11y-secondary
Components: Tests>Disabled
Labels: Test-Disabled
Cc: linds...@chromium.org
Labels: Hotlist-DisableReview
@dmazzoni can you ptal at these flaky tests?
Status: WontFix (was: Assigned)
I actually just did some work to make the tests more robust but updated a different bug:

https://chromium-review.googlesource.com/c/chromium/src/+/947039

According to the flakiness dashboard things are looking good!

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests%20(with%20patch)&tests=DumpAccessibilityEventsTest*

Closing as WontFix

Thanks!

Sign in to add a comment