This bot configuration has the long-standing ReadPixels flaky failure and cannot upgrade its drivers. The current pixel wrangler should mark these tests as flaky if possible. enne can you handle this?

While at it,
GpuRasterization.BlueBox failed in https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28AMD%20R5%20230%29/builds/216
(even though it should already be marked as flaky)
and
ScreenshotSync.SWRasterWithCanvas failed in https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28AMD%20R5%20230%29/builds/225

I notice that all these tests fail because they read (0, 0, 0) color. I wonder if this can be because this machine is slow and doesn't render in time?
I also see that it often excepts with "Connection to the other side was lost in a non-clean fashion" message:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28AMD%20R5%20230%29/builds/213
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28AMD%20R5%20230%29/builds/182

I see same behaviour on (New Intel) box, for which we try to see if newer hardware will solve the problem. I was wondering if we should ask for a new AMD R5 230 machine as well?

I am not sure Yuly, but this AMD card has had a problem of persistent flakiness on almost any test. If the new Intel ones fixes that we might look at it, but I think we can assume the AMD one will always be flaky. We should mark the tests as flaky on this card, fix the test harness as necessary, and just be super happy we don't have to deal with supporting this config fully. :)

Jamie, could you help enne out by drafting at least one of the needed flakiness suppression entries? (i.e., vendor and pci id)

I'll be out of town until Wednesday Dec 7. I think enne will have rotated out by then. I'll certainly be available to help the current GPU wrangler (or enne) at that point.

maps_pixel_test also flaky on Win10 Release (NVidia). Failing approx 6% of runs.


e.g., https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%29/builds/1607

AssertionError: Expected pixel at [40, 250] (actual pixel (40, 250))  to be [190, 214, 190] but got [230, 226, 217]

Stephen, could you dig a little deeper? That log contains this link:

http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?bd27d5fb3a918c21d8231b32fa0ec929ec66afae_Win10_Release_NVIDIA__telemetry

which shows the output of the test run.

It doesn't look totally wrong, so are we displaying the wrong last frame? It sort of looks like it. The output from my Mac laptop is attached.

Maybe DirectComposition related at this point? CC'ing jbauman.

Could you add a flaky suppression to src/content/test/gpu/gpu_tests/maps_expectations.py ?

  self.Flaky('Maps_maps_004', ['win10', nvidia'], bug= 670098 )

Deleted: Screen Shot 2017-04-26 at 4.53.03 PM.png 877 KB

	Screen Shot 2017-04-26 at 4.53.03 PM.png 877 KB View Download

Done: https://codereview.chromium.org/2852453002/

Yeah, it does look like we're snapshotting the image a frame or two early. I don't have a Win10 machine; over to jbauman@ for the DirectComposition question.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4a8be6cb76f35f025a3969bf074118114e607ce5

commit 4a8be6cb76f35f025a3969bf074118114e607ce5
Author: senorblanco <senorblanco@chromium.org>
Date: Fri Apr 28 11:38:22 2017

maps_pixel_test (gpu_tests): mark Maps_maps_004 as flaky on Win10 NVidia.

R=kbr@chromium.org
BUG= 670098 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2852453002
Cr-Commit-Position: refs/heads/master@{#467960}

[modify] https://crrev.com/4a8be6cb76f35f025a3969bf074118114e607ce5/content/test/gpu/gpu_tests/maps_expectations.py

I'm mystified but even after Stephen's suppression this test continues to be flaky on the Win10 NVIDIA bot.

https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20(NVIDIA)

Specifically:

https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%29/builds/1663
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%29/builds/1658
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%29/builds/1656
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%29/builds/1647
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%29/builds/1646

The comment in content/test/gpu/gpu_tests/maps_expectations.py says that the name should be "Maps.maps_004" and not "Maps_maps_004"?

Though, reading the log, it does seem to treat the test as Flaky, it just fails 3 retires.

It should be "Maps_maps_004" after the naming convention changes due to switching to the new gpu_integration_test harness.

Looking at some of the failures:

http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?39fa8ef8aa19b2ce2724cc682fc923b14581111c_Win10_Release_NVIDIA__telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?c362658d2401554189fb6cb79f8e3c4f028fcbee_Win10_Release_NVIDIA__telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?a7bdae0c8fc2af8615d676c546cef735a131a8f1_Win10_Release_NVIDIA__telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?934e8926ee432fe5f5e6b2b9a1f196455ede77df_Win10_Release_NVIDIA__telemetry

it looks like the last couple of animation frames are being intermittently dropped.

I'm not sure whether this test is using requestAnimationFrame like it should be, or is using something like setTimeout in order to emulate the behavior of setImmediate. I'll instrument the test harness to try to gather that information. John said he'd try to reproduce this on his Windows 10 workstation.

Ran this about a dozen times on my Win10 workstation with AMD GPU:
python content\test\gpu\run_gpu_integration_test.py maps --browser=canary

didn't see any failures. Would be good to know if any reproduce with an NVIDIA GPU.



Graphics Feature Status
Canvas: Hardware accelerated
Flash: Hardware accelerated
Flash Stage3D: Hardware accelerated
Flash Stage3D Baseline profile: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Enabled
Native GpuMemoryBuffers: Software only. Hardware acceleration disabled
Rasterization: Hardware accelerated
Video Decode: Hardware accelerated
Video Encode: Hardware accelerated
VPx Video Decode: Software only, hardware acceleration unavailable
WebGL: Hardware accelerated
WebGL2: Hardware accelerated
Driver Bug Workarounds
clear_uniforms_before_first_program_use
decode_encode_srgb_for_generatemipmap
disable_discard_framebuffer
disable_dxgi_zero_copy_video
disable_framebuffer_cmaa
exit_on_context_lost
force_cube_complete
scalarize_vec_and_mat_constructor_args
texsubimage_faster_than_teximage
Problems Detected
VPx decoding isn't supported before Windows 10 anniversary update.: 616318
Disabled Features: accelerated_vpx_decode
Some drivers are unable to reset the D3D device in the GPU process sandbox
Applied Workarounds: exit_on_context_lost
TexSubImage is faster for full uploads on ANGLE
Applied Workarounds: texsubimage_faster_than_teximage
Clear uniforms before first program use on all platforms: 124764, 349137
Applied Workarounds: clear_uniforms_before_first_program_use
Always rewrite vec/mat constructors to be consistent: 398694
Applied Workarounds: scalarize_vec_and_mat_constructor_args
ANGLE crash on glReadPixels from incomplete cube map texture: 518889
Applied Workarounds: force_cube_complete
Framebuffer discarding can hurt performance on non-tilers: 570897
Applied Workarounds: disable_discard_framebuffer
Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198
Applied Workarounds: disable_framebuffer_cmaa
Disable KHR_blend_equation_advanced until cc shaders are updated: 661715
Decode and Encode before generateMipmap for srgb format textures on Windows: 634519
Applied Workarounds: decode_encode_srgb_for_generatemipmap
Zero-copy DXGI video hangs or displays incorrect colors on AMD drivers: 623029
Applied Workarounds: disable_dxgi_zero_copy_video
Native GpuMemoryBuffers have been disabled, either via about:flags or command line.
Disabled Features: native_gpu_memory_buffers
Version Information
Data exported	5/1/2017, 3:25:51 PM
Chrome version	Chrome/60.0.3086.0
Operating system	Windows NT 10.0.10586
Software rendering list version	13.3
Driver bug list version	10.4
ANGLE commit id	f58417747f86
2D graphics backend	Skia/60 0c5cf5d7a26001e774116e470b8df76d01be77ff-
Command Line	"C:\Users\kbr\AppData\Local\Google\Chrome SxS\Application\chrome.exe" --flag-switches-begin --flag-switches-end
Driver Information
Initialization time	96
In-process GPU	false
Passthrough Command Decoder	false
Supports overlays	false
Sandboxed	false
GPU0	VENDOR = 0x1002, DEVICE= 0x683d
Optimus	false
Optimus	false
AMD switchable	false
Desktop compositing	Aero Glass
Diagonal Monitor Size of \\.\DISPLAY1	29.7"
Driver vendor	Advanced Micro Devices, Inc.
Driver version	21.19.519.2
Driver date	2-10-2017
Pixel shader version	5.0
Vertex shader version	5.0
Max. MSAA samples	8
Machine model name	
Machine model version	
GL_VENDOR	Google Inc.
GL_RENDERER	ANGLE (AMD Radeon R7 200 Series Direct3D11 vs_5_0 ps_5_0)
GL_VERSION	OpenGL ES 3.0 (ANGLE 2.1.0.f58417747f86)
GL_EXTENSIONS	GL_ANGLE_client_arrays GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_robust_resource_initialization GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc_srgb GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
Disabled Extensions	GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent
Window system binding vendor	Google Inc. (adapter LUID: 0000000000014ea2)
Window system binding version	1.4 (ANGLE 2.1.0.f58417747f86)
Window system binding extensions	EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_CHROMIUM_sync_control EGL_EXT_pixel_format_float EGL_ANGLE_display_texture_share_group EGL_ANGLE_create_context_client_arrays EGL_ANGLE_create_context_robust_resource_initialization
Direct rendering	Yes
Reset notification strategy	0x8252
GPU process crash count	0
Compositor Information
Tile Update Mode	One-copy
Partial Raster	Enabled
GpuMemoryBuffers Status
ATC	Software only
ATCIA	Software only
DXT1	Software only
DXT5	Software only
ETC1	Software only
R_8	Software only
RG_88	Software only
BGR_565	Software only
RGBA_4444	Software only
RGBX_8888	Software only
RGBA_8888	Software only
BGRX_8888	Software only
BGRA_8888	Software only
RGBA_F16	Software only
YVU_420	Software only
YUV_420_BIPLANAR	Software only
UYVY_422	Software only
Diagnostics
0
b3DAccelerationEnabled	true
b3DAccelerationExists	true
bAGPEnabled	true
bAGPExistenceValid	true
bAGPExists	true
bCanRenderWindow	true
bDDAccelerationEnabled	true
bDriverBeta	false
bDriverDebug	false
bDriverSigned	false
bDriverSignedValid	false
bNoHardware	false
dwBpp	32
dwDDIVersion	12
dwHeight	1600
dwRefreshRate	60
dwWHQLLevel	0
dwWidth	2560
iAdapter	0
lDriverSize	1589880
lMiniVddSize	0
szAGPStatusEnglish	Enabled
szAGPStatusLocalized	Enabled
szChipType	AMD Radeon Graphics Processor (0x683D)
szD3DStatusEnglish	Enabled
szD3DStatusLocalized	Enabled
szDACType	Internal DAC(400MHz)
szDDIVersionEnglish	12
szDDIVersionLocalized	12
szDDStatusEnglish	Enabled
szDDStatusLocalized	Enabled
szDXVAHDEnglish	Not Supported
szDXVAModes	ModeMPEG2_A ModeMPEG2_C ModeVC1_C ModeWMV9_C
szDescription	AMD Radeon R7 200 Series
szDeviceId	0x683D
szDeviceIdentifier	{D7B71EE2-2B7D-11CF-EA77-8400BCC2D835}
szDeviceName	\\.\DISPLAY1
szDisplayMemoryEnglish	66498 MB
szDisplayMemoryLocalized	66498 MB
szDisplayModeEnglish	2560 x 1600 (32 bit) (60Hz)
szDisplayModeLocalized	2560 x 1600 (32 bit) (60Hz)
szDriverAssemblyVersion	21.19.519.2
szDriverAttributes	Final Retail
szDriverDateEnglish	2/10/2017 12:00:00 AM
szDriverDateLocalized	2/10/2017 00:00:00
szDriverLanguageEnglish	English
szDriverLanguageLocalized	English
szDriverModelEnglish	WDDM 2.0
szDriverModelLocalized	WDDM 2.0
szDriverName	aticfx64.dll,aticfx64.dll,aticfx64.dll,amdxc64.dll,aticfx32,aticfx32,aticfx32,amdxc32,atiumd64.dll,atidxx64.dll,atidxx64.dll,atiumdag,atidxx32,atidxx32,atiumdva,atiumd6a.cap,atitmm64.dll
szDriverNodeStrongName	oem50.inf:cb0ae414ac142df8:ati2mtag_R575A:21.19.519.2:pci\ven_1002&dev_683d&subsys_22831458
szDriverSignDate	
szDriverVersion	21.19.0519.0002
szKeyDeviceID	Enum\PCI\VEN_1002&DEV_683D&SUBSYS_22831458&REV_00
szKeyDeviceKey	\Registry\Machine\System\CurrentControlSet\Control\Video\{FF459084-6D31-4DEE-A7C3-B061E366698E}\0000
szManufacturer	Advanced Micro Devices, Inc.
szMiniVdd	n/a
szMiniVddDateEnglish	n/a
szMiniVddDateLocalized	n/a
szMonitorMaxRes	
szMonitorName	HP Z30i IPS Display
szNotesEnglish	No problems found.
szNotesLocalized	No problems found.
szOverlayEnglish	Not Supported
szRankOfInstalledDriver	00D10001
szRegHelpText	
szRevision	
szRevisionId	0x0000
szSubSysId	0x22831458
szTestResultD3D7English	Not run
szTestResultD3D7Localized	Not run
szTestResultD3D8English	Not run
szTestResultD3D8Localized	Not run
szTestResultD3D9English	Not run
szTestResultD3D9Localized	Not run
szTestResultDDEnglish	Not run
szTestResultDDLocalized	Not run
szVdd	n/a
szVendorId	0x1002
Log Messages
GpuProcessHostUIShim: The GPU process exited normally. Everything is okay.

Note: with the attached patch and running:

./content/test/gpu/run_gpu_integration_test.py maps --browser=canary

and then opening the DevTools window while the test's paused (unfortunately I couldn't find a combination of arguments like -vv and --passthrough which would print the console.log logs), it looks strongly like this test's using setTimeout to perform its animation in its current form.

I wonder whether the Blink scheduler or something similar could be coalescing the setTimeout calls.

Deleted: logging.patch 2.2 KB

logging.patch 2.2 KB Download

Yeah, looks like the test is using setTimeout, but it waits for the last settimeout to run before marking it as done. We would never throw away a setTimeout callback, and it also waits for several RAFs afterwards.

I can't seem to reproduce this on my NVIDIA Win10 machine.

This test is unworkably flaky on the bots. Do you think that something could be wrong in our Win10 GPU bots' configuration (like Aero being disabled, or similar) that's specifically affecting them?

This is a major problem. Without diagnosing and fixing this, upgrading our bots to Win10 will leave a huge gap in testing.

I'm looking into using GetDC(nullptr) to grab the window contents on Win10. Unfortunately that one requires that the window be contained fully within the screen and be on top of everything, so it could cause flakiness if the bots aren't setup correctly.

We could also try IDXGIOutputDuplication or the Magnification API, both of which are used by WebRTC.

For reference - commands for running a local change on that swarming bot:

tools/mb/mb.py isolate //out/Default telemetry_gpu_integration_test

tools/swarming_client/isolate.py archive -I https://isolateserver.appspot.com -i out/Default/telemetry_gpu_integration_test.isolate -s out/Default/telemetry_gpu_integration_test.isolated --verbose

tools/swarming_client/swarming.py trigger -S https://chromium-swarm.appspot.com -I https://isolateserver.appspot.com --dimension cpu x86-64 --dimension gpu 10de:104a --dimension os Windows-10  --dimension pool Chrome --verbose out/Default/telemetry_gpu_integration_test.isolated -- maps --show-stdout '--browser=default' --passthrough -v '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc' --os-type win  '--isolated-script-test-output=${ISOLATED_OUTDIR}/output.json'

It looks like the browser window may have another window overlapping it, but I'm not sure of the dimensions yet - the overlap may just be of some part that doesn't matter.

The overlap is from the taskbar, because the monitor is 1024x768 while the chrome window is larger (default size is 1280x1024). I don't think we attempt to read that part of the window, though, so that's not why my new test mechanism is failing. It seems like the GetDC(0) is reading even older contents than PrintWindow does.

I may have to log into the IP KVM to see what's going on with the real monitor.

Any luck figuring out what's going on on these bots? Are the failures of GetDC / PrintWindow reproducible locally? We should be able to assume that there are no windows overlapping the browser window. This has been an assumption for a long time on these bots.

I can't reproduce the failure locally. Using GetDC(null) doesn't seem to help, but increasing the time delay between drawing and readback may make it work better.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5cfb0199ae5af577ca239feccd2d3c3f976ddac6

commit 5cfb0199ae5af577ca239feccd2d3c3f976ddac6
Author: jbauman <jbauman@chromium.org>
Date: Thu May 18 09:13:40 2017

Do glFinish before sending snapshot latency info to browser

On some GPU bots the swap was completing long before the contents got on
screen, so PrintWindow didn't get the correct window contents and some
tests were flaky. Triggering a glFinish before sending the latency info
for a snapshot request to the browser should be enough to ensure DWM
executes the Present before the snapshot is executed.

BUG= 670098 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2888993002
Cr-Commit-Position: refs/heads/master@{#472749}

[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/gpu/ipc/service/child_window_surface_win.cc
[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/gpu/ipc/service/child_window_surface_win.h
[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/gpu/ipc/service/direct_composition_surface_win.cc
[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/gpu/ipc/service/direct_composition_surface_win.h
[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/gpu/ipc/service/pass_through_image_transport_surface.cc
[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/ui/gl/gl_surface.cc
[modify] https://crrev.com/5cfb0199ae5af577ca239feccd2d3c3f976ddac6/ui/gl/gl_surface.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/95b208fa12493944a843e67c48bf6b21499c3f6d

commit 95b208fa12493944a843e67c48bf6b21499c3f6d
Author: jbauman <jbauman@chromium.org>
Date: Fri May 19 03:21:14 2017

Re-enable maps_pixel_test on Win10 NVIDIA

The flakiness should have been fixed by r472749.

BUG= 670098 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2891083003
Cr-Commit-Position: refs/heads/master@{#473058}

[modify] https://crrev.com/95b208fa12493944a843e67c48bf6b21499c3f6d/content/test/gpu/gpu_tests/maps_expectations.py

Thanks John for tenaciously tracking this down and re-enabling this critical test! Let's keep an eye on these bots for the next few days.