Issue 916744  and  Issue 916544  indicate that for a period of time, webgl_conformance_tests was the top source of false rejections from the CQ. It seems that Findit's queries aren't discovering this and the other GPU test suites.

From an email I sent to the team:

This failure happened late yesterday and earlier today:
 http://crbug.com/916544 

It was caught only on these two categories of bots:
 - The Mac10.13 Tests (dbg) bot on the main waterfall:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests%20%28dbg%29

between roughly these builds:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests%20%28dbg%29/7465
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests%20%28dbg%29/7471

and on the GPU team's release Mac bots, because we run with dcheck_always_on=true. Example ranges:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29/9276
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29/9289

and:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28AMD%29/10674
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28AMD%29/10691

We know for a fact that this tryserver was impacted:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_optional_gpu_tests_rel

A lot of jobs turned red while this failure was on the tree.

Searching FindIt, I don't see any flakes for test_type::webgl_conformance_tests , which would be great if true, but I don't believe our tests are that reliable. :) Could the team please see whether flakes are being detected for this and the other GPU test suites? (See the test group "gpu_telemetry_tests" in src/testing/buildbot/test_suites.pyl)

If this is really blocked on Issue 894254 then that has to be prioritized. Support for FindIt's command line arguments in the GPU test harnesses was added in  Issue 894258 .

Could the GPU tests be whitelisted (perhaps by step name) in FindIt until all of the Telemetry-based test harnesses are upgraded? The step names can be found here:

https://cs.chromium.org/chromium/src/testing/buildbot/test_suites.pyl

by searching for the test suites ending in "_telemetry_tests". They include:

webgl_conformance_gles_passthrough
info_collection
webgl_conformance_d3d11_validating
webgl_conformance_d3d9_passthrough
webgl_conformance_vulkan_passthrough
webgl_conformance_gl_passthrough
webgl_conformance_d3d9_tests
webgl2_conformance_d3d11_validating_tests
webgl2_conformance_gl_passthrough_tests
webgl2_conformance_tests
context_lost
depth_capture
gpu_process_launch_tests
hardware_accelerated_feature
maps_pixel_test
pixel_test
screenshot_sync
trace_test
webgl_conformance

For all of these, if the name doesn't end in "_test" or "_tests", then append "_tests" to get the step name on the bot.

Thanks.

stgao@ told me yesterday that this bug is confirmed to have been caused by the disabling of the "retry with patch" step for the GPU integration tests in  Issue 906166 .

Unlinking from Issue 894254.

Also, the indication was that by updating the query FindIt uses against its BigTable database, the data would be surfaced again. However, as of this morning Pacific time, there's still no data in FindIt for webgl_conformance_tests or webgl2_conformance_tests.

Yes, the root cause is "retry with patch" got disabled for those test steps.

The query is almost in shape. Since things get so complex now, I need to spend more time to test it to ensure it works as we expect.

ETA is to patch in the change in Findit prod this afternoon, assumed that the query is verified to work as I want it to.

After that, I will provide more details here.

Thank you stgao@.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ed27e765683bf0a7da1507eb490971b19fbf232a

commit ed27e765683bf0a7da1507eb490971b19fbf232a
Author: Shuotao Gao <stgao@chromium.org>
Date: Sat Jan 05 01:43:01 2019

[Findit] Fix the breakage caused by disabling "retry with patch".

Recently, a support was added to disable "retry with patch", and GPU tests opted
out to not run "retry with patch" ( https://crbug.com/906166 ). This caused a
2-week breakage in flake detection for GPU tests.

This CL is to make the queries support test steps with "retry with patch"
enabled or disabled.

Bug:  916762 
Change-Id: I5015d166aa04a45c21a4f9b6c1424dae8336010b
Reviewed-on: https://chromium-review.googlesource.com/c/1388674
Commit-Queue: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19793}
[modify] https://crrev.com/ed27e765683bf0a7da1507eb490971b19fbf232a/appengine/findit/services/flake_detection/flaky_tests.retry_with_patch.sql
[modify] https://crrev.com/ed27e765683bf0a7da1507eb490971b19fbf232a/appengine/findit/services/flake_detection/flaky_tests.hidden_flakes.sql
[modify] https://crrev.com/ed27e765683bf0a7da1507eb490971b19fbf232a/appengine/findit/services/flake_detection/flaky_tests.cq_false_rejection.sql