FWIW the actual ANGLE regression range only has one very innocent CL:

https://chromium.googlesource.com/angle/angle.git/+log/1de29ab..95644f92dd1

No idea why AMD might be failing with hubbe's change or skia.

OK let's intersect these 3 first failed builds:

1) 500373-474
2) 500429-498
3) 500461-491

So the failure will be in 500429-500474

https://chromium.googlesource.com/chromium/src/+log/e7d66efe5dd9396bc97569640704f33202edef6e..d7718847e9938b80e58148d91555c2bb6eff9b14

Off by one, https://chromium.googlesource.com/chromium/src/+log/e8e49e184dbaf840d63be93f2163fa1dafa1e208..d7718847e9938b80e58148d91555c2bb6eff9b14

Any chance it is https://chromium.googlesource.com/chromium/src/+/ceb8727d80e9ddd671577494ddeb1c182645833e ?

An example failure:

[6/179] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance2_glsl3_shader_with_invalid_characters passed 0.1720s
[1728:2508:0907/212241.022:INFO:CONSOLE(118)] "should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.023:INFO:CONSOLE(118)] "FAIL should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.054:INFO:CONSOLE(118)] "should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.054:INFO:CONSOLE(118)] "FAIL should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.070:INFO:CONSOLE(118)] "should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.070:INFO:CONSOLE(118)] "FAIL should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.089:INFO:CONSOLE(118)] "should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)
[1728:2508:0907/212241.089:INFO:CONSOLE(118)] "FAIL should be green at (0, 0) expected: 0,255,0,255 was 255,0,0,255", source:  (118)

> 1) 500373-474
> 2) 500429-498
> 3) 500461-491
> 
> So the failure will be in 500429-500474

OOPS. Intersect means it's 500461-500474.

https://chromium.googlesource.com/chromium/src/+log/0e7dd5e393d7d838f7ffac633c727dd4e7a036fa..d7718847e9938b80e58148d91555c2bb6eff9b14

hubbe's patch is the only remaining suspect.

Great, so my patch fixes things on intel and break things on AMD. :(
Can we just blacklist Windows?

What do you mean by blacklisting Windows? If you mean blacklisting a certain feature on certain GPUs, yes, that's done all the time -- but that doesn't sound like what's going on here.

It's not workable to mark these tests as expected failures. That will leave a huge gap in testing.

We need to get to the bottom of these failures and fix them.

If some of the GPU bots seem misconfigured -- for example, if their display color profiles vary -- we can either work with the Labs team to fix this, or pass some command line flag which will force the use of a particular color profile.

Please own this issue and tell us what we can do to help you.

Sorry, I was making a bad joke about blacklisting Windows.

The problem isn't that the bots are badly configured (I think), the problem is that video APIs on windows are a minefield of bugs which I don't know if I can navigate. I'm going to take another look at the machine I was using yesterday and see if I made a mistake, or if that machine just behaves differently from the bot. I might also try a few alternate ways to patch the problem, but trial-and-error with FYI bots is pretty slow process.

You can log onto these machines directly, take them offline briefly and run your own binaries on them. I would strongly advocate that you try this. It's not that bad a process and I personally do it for the really tough bugs that only reproduce on one particular machine. The instructions for logging on to the bots are linked from https://www.chromium.org/developers/testing/gpu-testing#TOC-Running-Locally-Built-Binaries-on-the-GPU-Bots . If the instructions are incorrect in any way please tell me.
 

If you want Geoff or me to beta-test a patch we can do that too. We have a machine that can run Intel/AMD/NVIDIA gpus on Windows 7 and 10.

The patch that was causing this was reverted.
The new fix has been deployed, without breaking the AMD bots.

Fantastic. Thank you Fredrik for persisting with this fix.