These tests are marked with disabled and run with --gtest_also_run_disabled_tests because they have hardware requirements that prevent them from running on most Windows bots. We only pass the --gtest_also_run_disabled_tests with a specific test filter on devices that we know meet these requirements.

However, given how many times this has been reported as an issue, combined with the fact that this hides the results on the flakiness dashboard, I think it's about time I got around to implementing another solution to this.

I'll look into the flakes as well.

Yes, it would be best to factor these tests out into their own binary. Sunny: we run these tests on the GPU bots as a courtesy to the WebXR team.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/502162f1f79563e4f522d4b2d38d5772f801aa4a

commit 502162f1f79563e4f522d4b2d38d5772f801aa4a
Author: bsheedy <bsheedy@chromium.org>
Date: Thu Dec 13 18:47:41 2018

Add XR JavaScript session logging

Adds logging to the WebXR JavaScript boilerplate used for WebXR browser
and instrumentation tests whenever a session request is attempted,
succeeds, or fails. Failure to get sessions is one of the more common
flake causes, so this will help give an initial idea of whether the
failures are due to the promise being rejected or something in the
implementation hanging.

Bug: 914589
Change-Id: Ibbd0aa7890638d6da8ac9037f4740516c74ab271
Reviewed-on: https://chromium-review.googlesource.com/c/1375394
Reviewed-by: Bill Orr <billorr@chromium.org>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616371}
[modify] https://crrev.com/502162f1f79563e4f522d4b2d38d5772f801aa4a/chrome/test/data/xr/e2e_test_files/resources/webxr_boilerplate.js

The combination of logging and splitting out the tests into their own target has let us get some data on how flaky these tests actually are. They're mostly green, but a few will get a timed out test every now and then, but the retries seem to be preventing these from actually showing as failures.

When the tests time out, it looks like we're requesting a session but the returned promise is neither resolved nor rejected within the 10 second timeout, whereas it's normally resolved almost instantly during passing tests.

What's strange though is that once we get past that part (apparently EXPECT_TRUE failing doesn't actually end the test immediately?) the promise resolves almost immediately after we get a TypeError due to trying to add an event listener to the non-existent session. For example, we get the following log output from one of the failed tests:

[3740:2812:1218/090945.366:INFO:CONSOLE(83)] "Requesting immersive session", source: file:///C:/b/s/w/ir/chrome/test/data/xr/e2e_test_files/resources/webxr_boilerplate.js (83)
[3740:2812:1218/090945.366:ERROR:xr_browser_test.cc(127)] Run JavaScript: sessionInfos[sessionTypes.IMMERSIVE].currentSession != null

...

[3740:2812:1218/090955.339:ERROR:xr_browser_test.cc(127)] Run JavaScript: sessionInfos[sessionTypes.IMMERSIVE].currentSession != null
../../chrome/browser/vr/test/xr_browser_test.cc(167): error: Value of: PollJavaScriptBoolean(bool_expression, timeout, web_contents)

  Actual: false

Expected: true

Timed out polling JavaScript boolean expression: sessionInfos[sessionTypes.IMMERSIVE].currentSession != null
[3740:2812:1218/090955.606:INFO:CONSOLE(59)] "Uncaught TypeError: Cannot read property 'addEventListener' of null", source: file:///C:/b/s/w/ir/chrome/test/data/xr/e2e_test_files/html/test_webxr_input.html (59)
[3740:2812:1218/090955.693:INFO:CONSOLE(85)] "Immersive session request succeeded", source: file:///C:/b/s/w/ir/chrome/test/data/xr/e2e_test_files/resources/webxr_boilerplate.js (85)


Based on that, it sounds like something on Blink's side is getting stuck?