Sample failure: https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/47829

LanguageOptionsWebUITest.testOpenLanguageOptions (run #1):
[ RUN      ] LanguageOptionsWebUITest.testOpenLanguageOptions
Xlib:  extension "RANDR" missing on display ":9".
Xlib:  extension "RANDR" missing on display ":9".
[22569:22569:1019/092606:WARNING:audio_manager.cc(317)] Multiple instances of AudioManager detected
[22569:22569:1019/092606:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
[22569:22569:1019/092606:WARNING:password_store_factory.cc(248)] Using basic (unencrypted) store for password storage. See https://chromium.googlesource.com/chromium/src/+/master/docs/linux_password_storage.md for more information about password storage options.
[22569:22569:1019/092607:WARNING:CONSOLE(0)] "/deep/ combinator is deprecated. See https://www.chromestatus.com/features/6750456638341120 for more details.", source:  (0)
[22569:22569:1019/092607:INFO:CONSOLE(1206)] "Running TestCase LanguageOptionsWebUITest.testOpenLanguageOptions", source: test_api.js (1206)
[22569:22569:1019/092608:ERROR:web_ui_test_handler.cc(76)] Failed: RUN_TEST_F("LanguageOptionsWebUITest","testOpenLanguageOptions")
Error:
Accessibility issues found on chrome://settings-frame/languages
*** Begin accessibility audit results ***
An accessibility audit found
Errors:
Error: AX_ARIA_08 (Elements with ARIA roles must ensure required owned elements are present) failed on the following elements (1 - 2 of 2):
#default-search-engine-list
#other-search-engine-list
See https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules#ax_aria_08 for more information.


*** End accessibility audit results ***
    at LanguageOptionsWebUITest.runAccessibilityAudit (test_api.js:371:17)
    at TestCase.runAccessibilityAudit (test_api.js:501:22)
    at test_api.js:1005:20
    at testDone (test_api.js:783:31)
    at runTest (test_api.js:1052:7)
    at <anonymous>:1:1
gen/chrome/browser/ui/webui/options/language_options_browsertest-gen.cc:23: Failure
Value of: RunJavascriptTestF( false, "LanguageOptionsWebUITest", "testOpenLanguageOptions")
  Actual: false
Expected: true
[  FAILED  ] LanguageOptionsWebUITest.testOpenLanguageOptions, where TypeParam =  and GetParam() =  (1486 ms)

Many other tests are suffering the same error:

AdvancedSettingsWebUITest.testAdvancedSettingsShown
CertificateManagerSettingsWebUITest.testViewAndDeleteCert
EditDictionaryWebUITest.testAddNotification
EditDictionaryWebUITest.testAddRemoveWords
FontSettingsWebUITest.testOpenFontSettings
LanguagesOptionsDictionaryDownloadWebUITest.testdictionaryDownloadRetry
MultilanguagePreferenceWebUIBrowserTest.SelectFromBlank
SettingsFormatWebUITest.RadioCheckboxStyleCheck

Moreover it seems a bit flaky.  Assigning to dbeam@.

I'm raising this issue to P1 because these test flakes are causing retries on the commit queue, which is very expensive. Please handle these flaky tests. A couple of instances of failures of SettingsFormatWebUITest.RadioCheckboxStyleCheck:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/319821
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/319771

Here are all of the instances of this test flaking on the try bots:

http://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyOgsSBUZsYWtlIi9TZXR0aW5nc0Zvcm1hdFdlYlVJVGVzdC5SYWRpb0NoZWNrYm94U3R5bGVDaGVjaww

Ran test locally and don't see error, but here are some theories from tracing.

I'm think these are real bugs, and were previously masked because test infra was buggy? Perhaps somehow infra got fixed and now bugs are all over?

Reason why I think this: Here's the test scrip axs_testing.js, on line 1706:
https://cs.chromium.org/chromium/src/third_party/accessibility-audit/axs_testing.js?q=axs_testing.js&sq=package:chromium&dr&l=1706

    this.test_(f, a.config) && this.addElement(d, f);

So if |test.test_()| returns *true*, then then effectively |d.push(f)| gets called to accumulate error.  Later |d.length != 0| results in axs.constants.AuditResult.FAIL.

|this.test_()| is dynamic and can e.g., point to line 2143. However, it seems that these "test()" functions return *false* == !1 on failure, and true == !0 on success?

So I think line 1706 should be

    !this.test_(f, a.config) && this.addElement(d, f);

Anyway, the bugs seem real and should be fixed one-by-one. Meanwhile it would be helpful to see why the bots and local runs don't agree.

+aboxhall@

@huangs, can you just disable the tests temporarily until this is figured out?

Another example:

https://build.chromium.org/p/chromium.fyi/builders/Browser%20Side%20Navigation%20Linux/builds/22773/steps/browser_tests%20on%20Ubuntu-12.04/logs/AdvancedSettingsWebUITest.testAdvancedSettingsShown

Yup, holte@ is onto that.

I think this is happening in a bunch of different tests. Is there any recent change to the accessibility code which coincided with this flakiness that we can roll back?

re comment 4: Not sure what you mean by test infra was buggy? you mean browser tests harness? there hasn't been any changes there afaik.

I think this may be the same issue as crbug/617066?

Note that this is an accessibility *audit* that's failing. That code hasn't changed in months (see third_party/accessibility-audit), and either way it's just JavaScript that runs at the end of some WebUI tests, and not related to the rest of Chrome's accessibility code.

I wouldn't be surprised if this change is related:
https://codereview.chromium.org/2346393004

However, that wouldn't explain why these tests suddenly started failing frequently and flakily.

Isn't FindIt supposed to be good for this type of thing?

Yeah, talked with aboxhall@, turns out returning true => problem, so I was mistaken.

And maybe crbug/631940 too?

Is there any chance of a race condition causing page content not to be fully loaded before the test runs? 

And/or might the profile used on the bots have, for example, no default search engine? A list with 0 items would cause a result like this.

Also crbug/616791

So it seems like many of these accessibility audits have been flaky on Windows at least in the past.  I'm moving forward with disabling many of the tests on linux/chromeos, but this seems likely to be a bit of a whack-a-mole approach.

Are these all the same audit rule failing?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/486e2998a18f28fa0a0afe8e1e485805fdc00be7

commit 486e2998a18f28fa0a0afe8e1e485805fdc00be7
Author: Steven Holte <holte@google.com>
Date: Wed Oct 19 23:13:48 2016

Disable many tests with flaky accessibility audits.

BUG= 657514 
TBR=stevenjb

Review URL: https://codereview.chromium.org/2434473004 .

Cr-Commit-Position: refs/heads/master@{#426325}

[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/browser_options_browsertest.js
[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/certificate_manager_browsertest.js
[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/edit_dictionary_browsertest.js
[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/font_settings_browsertest.js
[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/language_options_dictionary_download_browsertest.js
[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/multilanguage_options_webui_browsertest.js
[modify] https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7/chrome/browser/ui/webui/options/settings_format_browsertest.js

Has anyone investigated whether these are all the same rule flaking? If so, you could disable that rule in the test runner instead of disabling tests on a per-test basis (and re-enable the tests so that they can continue to provide coverage).

See https://cs.chromium.org/chromium/src/chrome/test/data/webui/test_api.js?q=test_api.js&sq=package:chromium&dr&l=195

The only ones I see are:

Error: AX_ARIA_08 (Elements with ARIA roles must ensure required owned elements are present) failed on the following elements (1 - 2 of 2):
#default-search-engine-list
#other-search-engine-list
See https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules#ax_aria_08 for more information.

another failure: BrowserOptionsFrameWebUITest.testAdvancedSettingsHiddenByDefault https://build.chromium.org/p/chromium.fyi/builders/Browser%20Side%20Navigation%20Linux/builds/22807/steps/browser_tests%20on%20Ubuntu-12.04/logs/BrowserOptionsFrameWebUITest.testAdvancedSettingsHiddenByDefault

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6e92a0ab099744efc025c76ace00614b9cb5ad2

commit c6e92a0ab099744efc025c76ace00614b9cb5ad2
Author: mathp <mathp@chromium.org>
Date: Thu Oct 20 17:16:20 2016

[Aria tests] Disable a flaky rule (AX_ARIA_08)

See bug for many failing tests.
Example: https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/47887/steps/browser_tests%20on%20Ubuntu-12.04/logs/EditDictionaryWebUITest.testNoCloseOnSearchEnter

BUG= 657514 
TBR=aboxhall
NOTRY=true

Review-Url: https://chromiumcodereview.appspot.com/2436063002
Cr-Commit-Position: refs/heads/master@{#426514}

[modify] https://crrev.com/c6e92a0ab099744efc025c76ace00614b9cb5ad2/chrome/test/data/webui/test_api.js

more:
https://build.chromium.org/p/chromium.fyi/builders/Browser%20Side%20Navigation%20Linux/builds/22810/steps/browser_tests%20on%20Ubuntu-12.04/

AutofillEditAddressWebUITest.testFieldValuesSaved
AutofillEditAddressWebUITest.testLoadAddressComponents
CertificateManagerWebUIUnpopulatedTest.testUnpopulatedCertificateManager
ContentSettingsExceptionsAreaAsyncWebUITest.testAddRemoveLocationExceptions
ProfileSettingsResetWebUITest.testOpenProfileSettingsReset

should this accessibility checker be disabled until the root cause is tracked down instead of playing whack-a-mole with different tests?

Yeah, should follow aboxhall@'s advise. Also, these are all AX_ARIA_08, so here's a CL to disable these:
https://chromiumcodereview.appspot.com/2441643003/

Oh wait, mathp@ already took care of that: https://chromiumcodereview.appspot.com/2436063002

Were the tests referenced in https://codereview.chromium.org/2434473004 also failing due to AX_ARIA_08?  If so, that CL should probably be reverted now that mathp's change has landed.

 Issue 657567  has been merged into this issue.

Created a revert: https://codereview.chromium.org/2436063002

Also, please note in future that you can disable the audit instead of disabling the test: just set runAccessibilityChecks to false in your config, or call this.disableAccessibilityChecks() at the start of a test method.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8c598600615e8c53ea85b628ab770271d859d390

commit 8c598600615e8c53ea85b628ab770271d859d390
Author: aboxhall <aboxhall@chromium.org>
Date: Fri Oct 21 22:25:32 2016

Revert of Disable many tests with flaky accessibility audits. (patchset #2 id:20001 of https://codereview.chromium.org/2434473004/ )

Reason for revert:
This is done more efficiently by https://codereview.chromium.org/2436063002

Original issue's description:
> Disable many tests with flaky accessibility audits.
>
> BUG= 657514 
> TBR=stevenjb
>
> Committed: https://crrev.com/486e2998a18f28fa0a0afe8e1e485805fdc00be7
> Cr-Commit-Position: refs/heads/master@{#426325}

TBR=holte@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 657514 

Review-Url: https://chromiumcodereview.appspot.com/2440973003
Cr-Commit-Position: refs/heads/master@{#426916}

[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/browser_options_browsertest.js
[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/certificate_manager_browsertest.js
[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/edit_dictionary_browsertest.js
[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/font_settings_browsertest.js
[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/language_options_dictionary_download_browsertest.js
[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/multilanguage_options_webui_browsertest.js
[modify] https://crrev.com/8c598600615e8c53ea85b628ab770271d859d390/chrome/browser/ui/webui/options/settings_format_browsertest.js

> However, that wouldn't explain why these tests suddenly started failing frequently and flakily.

yeah, this doesn't really make much sense to me either

Closing during triage since there's lack of activity.