Good idea. Yes, feel free to duplicate any of the GPU tests' entries in generate_buildbot_json.py and send out for review. screenshot_sync_tests sounds like a good suggestion because it draws multiple frames and tries to ensure that they all reached the screen at the expected time.

I think we still need this and I'm still intending to add the tests. Running screenshot_sync_tests with --enable-features=VizDisplayCompositor fails though. I think it's because Page.captureScreenshot doesn't work, but hopefully that's the last blocker.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6f5d66e15cd45f92ce61734f9750eb1af480f3e5

commit 6f5d66e15cd45f92ce61734f9750eb1af480f3e5
Author: kylechar <kylechar@chromium.org>
Date: Fri Apr 27 22:14:47 2018

Add viz_screenshot_sync_tests.

Add a new GPU test target that runs screenshot_sync a second time with
--enable-features=VizDisplayCompositor. This test will verify that GPU
compositing is working for OOP-D. The test only runs on Windows and
Linux bots.

Bug:  812385 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I56de0d01922d1c08521f3fd5aed4f64c148374e3
Reviewed-on: https://chromium-review.googlesource.com/1028613
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: kylechar <kylechar@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554529}
[modify] https://crrev.com/6f5d66e15cd45f92ce61734f9750eb1af480f3e5/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/6f5d66e15cd45f92ce61734f9750eb1af480f3e5/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/6f5d66e15cd45f92ce61734f9750eb1af480f3e5/testing/buildbot/chromium.gpu.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/682897dbd0c91ba33a25b264f248e289a1da0bd1

commit 682897dbd0c91ba33a25b264f248e289a1da0bd1
Author: kylechar <kylechar@chromium.org>
Date: Mon Apr 30 00:15:27 2018

Revert "Add viz_screenshot_sync_tests."

This reverts commit 6f5d66e15cd45f92ce61734f9750eb1af480f3e5.

Reason for revert: Test is failing on Windows 7 GPU bots.

Original change's description:
> Add viz_screenshot_sync_tests.
> 
> Add a new GPU test target that runs screenshot_sync a second time with
> --enable-features=VizDisplayCompositor. This test will verify that GPU
> compositing is working for OOP-D. The test only runs on Windows and
> Linux bots.
> 
> Bug:  812385 
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
> Change-Id: I56de0d01922d1c08521f3fd5aed4f64c148374e3
> Reviewed-on: https://chromium-review.googlesource.com/1028613
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: kylechar <kylechar@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#554529}

TBR=kbr@chromium.org,kylechar@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  812385 
Change-Id: Icf9c2d398312bebfd7ca5b7d47e2b0876716dd70
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/1034392
Reviewed-by: kylechar <kylechar@chromium.org>
Commit-Queue: kylechar <kylechar@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554682}
[modify] https://crrev.com/682897dbd0c91ba33a25b264f248e289a1da0bd1/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/682897dbd0c91ba33a25b264f248e289a1da0bd1/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/682897dbd0c91ba33a25b264f248e289a1da0bd1/testing/buildbot/chromium.gpu.json

It looks like all chromium.gpu.fyi Windows 7 bots were failing but only those bots. The Windows 10 bots were fine.

This seems weird to me because it ran win7_chromium_rel_ng in CQ and passed. Looking closer win7_chromium_rel_ng runs GPU tests on Windows 10. I'm not sure if that's intentional, do you know kbr?

> viz_screenshot_sync_tests on NVIDIA GPU on Windows (with patch) on Windows-10 viz_screenshot_sync_tests on NVIDIA GPU on Windows (with patch) on Windows-10

https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/155999

The test runs on Windows 7 but there is a color mismatch. Lots of errors that look like this:

> AssertionError: Color mismatch at (10, 0): expected (142, 176, 150), got (217, 11, 223)

https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8947864913263888528%2F%2B%2Fsteps%2Fviz_screenshot_sync_tests_on_NVIDIA_GPU_on_Windows_on_Windows-2008ServerR2-SP1%2F0%2Fstdout

We have a Windows 7 lab machine but I can't reproduce the failures. The one difference I noticed is that the lab machine doesn't have numpy/cv2 installed and the test takes much longer. If this is a race where we're getting a screenshot too early/late then that would might be the reason I can't reproduce. The lab machine doesn't have gsutil credentials so it can't run the checkout_telemetry_binary_dependencies hook.

Please always link to failing builds when reverting CLs. Here's one of the failures:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20FYI%20Release%20%28NVIDIA%29/819

Yes, it is expected that win7_chromium_rel_ng triggers all of its GPU tests on Windows 10. Sorry for the confusion but we had to update the GPU bots to Windows 10 to increase testing coverage and didn't have another good tryserver on which to run those jobs.

If you invoke the test with vpython (provided by depot_tools) then you will pick up the numpy and cv2 dependencies automatically. No checkout_telemetry_binary_dependencies needed any more.

It seems plausible that you would need numpy/cv2 installed to provoke this failure. Please try again on your lab machine with vpython. Chrome's rendering paths on Win7 vs. Win10 are probably different enough that the viz version of these tests could be broken on Win7 but not Win10.

Note also that we could set up tryservers for the Win 7 bots which could be manually triggered. Your help would be appreciated doing that if you think it would be valuable; I would be happy to guide you through the process.

Thanks for all the info kbr! I tried running with vpython and it picked up numpy and cv2 as you said. The test runs much faster, but unfortunately still passes consistently.

I'm happy to do the setup for manually triggerable Windows 7 GPU trybots, since that is probably the best way for me to reproduce this and will be useful going forward. Is there something you can point me at or is a quick GVC conversation the best path forward here?

kylechar: I'll be glad to work with you on this. We need to update our public documentation https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/gpu_testing_bot_details.md to describe more maintenance scenarios. I'm filing a bug to request the needed Windows VMs and then you and I can work together to deploy them and simultaneously update the documentation.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8d6d5225272cd89e7c61ce07f6725c1af5ec7ae2

commit 8d6d5225272cd89e7c61ce07f6725c1af5ec7ae2
Author: kylechar <kylechar@chromium.org>
Date: Mon May 14 20:27:47 2018

RELAND: Add viz_screenshot_sync_tests.

Add a new GPU test target that runs screenshot_sync a second time with
--enable-features=VizDisplayCompositor. This test will verify that GPU
compositing is working for OOP-D. The test only runs on Windows and
Linux bots.

Originally landed in https://crrev.com/c/1028613 but was reverted
because it failed on Windows 7 waterfall bots. There is some bug in the
interaction between the passthrough command decoder, OOP-D and frame
capture. Disable viz_screenshot_sync on Windows 7 FYI bots for now.

Bug:  812385 ,  825906 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I96ad1b49313bf5bbcdca0f464da4b185ee74722a
Reviewed-on: https://chromium-review.googlesource.com/1047146
Commit-Queue: kylechar <kylechar@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558443}
[modify] https://crrev.com/8d6d5225272cd89e7c61ce07f6725c1af5ec7ae2/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/8d6d5225272cd89e7c61ce07f6725c1af5ec7ae2/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/8d6d5225272cd89e7c61ce07f6725c1af5ec7ae2/testing/buildbot/chromium.gpu.json