CC'ing thakis and maruel who have more experience with Chromium's build setup, which is probably where the bug is, rather than on the Infra side.

Thanks Ken, I wasn't sure who to ask.

Another build failing here:

https://build.chromium.org/p/tryserver.chromium.angle/builders/win_angle_rel_ng/builds/978

You supplied angle_deqp_gles2_tests_run as a potential target but this target doesn't exist in this configuration. FWIU, it's a recipe bug.

Did I forget to put the build_angle_deqp_tests GYP_DEFINE on these bots? It's very possible.

In the tools/build workspace:

scripts/slave/recipe_modules/chromium_tests/trybots.py says that win_angle_rel_ng mirrors:

          {
            'mastername': 'chromium.gpu.fyi',
            'buildername': 'GPU Win Builder',
            'tester': 'Win7 Release (NVIDIA)',
          },
          {
            'mastername': 'chromium.gpu.fyi',
            'buildername': 'GPU Win Builder',
            'tester': 'Win7 Release (ATI)',
          },

and scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py states:

    'GPU Win Builder': {
      'chromium_config': 'chromium',
      'chromium_apply_config': ['ninja_confirm_noop', 'archive_gpu_tests',
                                'chrome_with_codecs',
                                'internal_gles2_conform_tests',
                                'build_angle_deqp_tests'],

so theoretically that gyp_define should be being set, but the 'gclient runhooks (with patch)' step from the tryjob above says:

 GYP_DEFINES: archive_gpu_tests=1 chromium_win_pch=0 component=static_library dcheck_always_on=1 fastbuild=1 ffmpeg_branding=Chrome gomadir='E:\b\build\goma' proprietary_codecs=1 target_arch=ia32 test_isolation_mode=prepare use_goma=1

which doesn't include the associated flag.

Not sure what's going on here. The bot's running the chromium_trybot recipe and a waterfall restart should not have been necessary to pick up that gyp_define change.

Ken, in a successful build build_angle_deqp_tests does show up:

https://build.chromium.org/p/tryserver.chromium.angle/builders/win_angle_rel_ng/builds/981/steps/gclient%20runhooks%20%28with%20patch%29/logs/stdio

Something fishy is definitely going on.

It's that the update_scripts step is failing, so the bot is running an old version of the recipe. This is something the troopers need to fix.

https://build.chromium.org/p/tryserver.chromium.angle/buildslaves/vm1012-m4

Similar issue: Issue 570808.

Log extract:

------
________ running 'git -c core.deltaBaseCacheLimit=512m fetch origin --verbose' in 'E:\b\build'
[0:00:04] From https://chromium.googlesource.com/a/chromium/tools/build
[0:00:04]  = [up to date]      infra/config -> origin/infra/config
[0:00:04]  = [up to date]      master     -> origin/master
[0:00:04]  = [up to date]      recipe-package-canary -> origin/recipe-package-canary

________ running 'git reset --hard HEAD' in 'E:\b\build'
[0:00:04] fatal: Unable to create 'E:/b/build/.git/index.lock': File exists.
[0:00:04] 
[0:00:04] If no other git process is currently running, this probably means a
[0:00:04] git process crashed in this repository earlier. Make sure no other git
[0:00:04] process is running and remove the file manually to continue.

I deleted the .git/index.lock file, I'll keep an eye on the builder.

Looks like it's green now.

Thanks all for figuring this out!