Set priority to 1 for end of Q3 completion of public client migration

Builder blocked on master bug 868204

Builder is blocked on master bug 868207

Builder mirrored to LUCI and latest builds are failing due to MSVC issue. See builds at: https://ci.chromium.org/p/nacl/g/main/console

Recommended action: Fix and verify. 

Assigned to hinoka@ to try out a recipe change. Add label for tracking. CC'ed dpranke@ since he's looking into similar issues for Crashpad builders. 

Friendly ping to hinoka@. Have you had the chance to try the recipe change to fix these? 

Assigning directly to iannucci to see if using MSVC SDK from CIPD will fix this issue. Thanks Robbie!

So it appears that:
  * SCons scans for msvc and fails if it can't find it
  * SCons is then configured to NOT USE MSVC AT ALL (but use clang instead).
  * Since it never actually ran any msvc processes, mspdbsrv is not running
  * The recipe currently counts it as a fatal flaw if it can't kill mspdbsrv (no one has run into this because everything that uses msvc actually uses the binaries it installs).

I'll change the windows_sdk module to not treat the taskkill as fatal, but it would be great if someone close to pnacl could remove this bogus dependency on msvc.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/77900be4e71e5ccda7d9efeca7bbff6dc5f3c597

commit 77900be4e71e5ccda7d9efeca7bbff6dc5f3c597
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Sep 19 01:33:43 2018

[windows_sdk] Allow taskkill mspdbsrv to fail.

It looks like PNaCl actually depends on MSVC's presance, but doesn't
actually run any compile steps with it (instead using clang). That means
that mspdbsrv never actually runs during their build, and then the
taskkill fails.

R=hinoka@chromium.org, tandrii@chromium.org

Bug:  861512 
Change-Id: I004d28f198224adaf16b1d3f14401b0d2a7d700b
Reviewed-on: https://chromium-review.googlesource.com/1232885
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/77900be4e71e5ccda7d9efeca7bbff6dc5f3c597/recipes/README.recipes.md
[modify] https://crrev.com/77900be4e71e5ccda7d9efeca7bbff6dc5f3c597/recipes/recipe_modules/windows_sdk/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc9f174223aefa0eb0ab45701c005327274f555a

commit fc9f174223aefa0eb0ab45701c005327274f555a
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Wed Sep 19 04:30:40 2018

Roll src/third_party/depot_tools 79c651330b94..77900be4e71e (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/79c651330b94..77900be4e71e


git log 79c651330b94..77900be4e71e --date=short --no-merges --format='%ad %ae %s'
2018-09-19 iannucci@chromium.org [windows_sdk] Allow taskkill mspdbsrv to fail.


Created with:
  gclient setdep -r src/third_party/depot_tools@77900be4e71e

The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:861512 
TBR=agable@chromium.org

Change-Id: Ic6c3aadd5d77ae4b4df7be7265c0109dc30bee65
Reviewed-on: https://chromium-review.googlesource.com/1232625
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#592317}
[modify] https://crrev.com/fc9f174223aefa0eb0ab45701c005327274f555a/DEPS

@iannucci I'm a little confused about how you found out what you said in #10. 
Edit: nevermind, You're looking at the toolchain bot.

I'm looking at https://ci.chromium.org/p/nacl/builders/luci.nacl.ci/win7-64-glibc-opt/21 , specifically https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8935101935740774512/+/steps/scons_compile/0/stdout where it appears that SCons is using the system-installed MSVC, but then eventually failing to kill mspdbserv. Presumably that's what efoo@ meant when he said "using the MSVC SDK from CIPD". I am interested in fixing that btw (i.e. making the NaCl build use the CIPD SDK) but not sure how I would do it without converting all of the builds to recipes.

Also that builder has a GN build https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8935101935740774512/+/steps/gn_compile_64/0/stdout but since it doesn't print its command lines I have no idea what it's actually doing to build.

Ok, so I took another look at some stuff. For the recipe based bots, I added 
nacl-toolchain-builder@chops-service-accounts.iam.gserviceaccount.com to https://chrome-infra-auth.appspot.com/auth/groups/winsdk-cipd-readers, which should let them get the CIPD package.

For the bots not using recipes, if their service account is added to the winsdk-cipd-readers group, they should be able to basically do what the recipe module does (https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/master/recipes/recipe_modules/windows_sdk/api.py#24). Basically:
  * Pull and unpack the SDK from cipd
  * read sdk_dir/win_sdk/bin/SetEnv.x86.json (or x64) to get all the envvar modifications
  * Modify the env accordingly
  * profit?

After #11 we have a puzzling situation in https://ci.chromium.org/p/nacl/builders/luci.nacl.ci/win7-64-glibc-opt/22 wherein all the steps are green, but the result is still red. Any ideas?

Though it looks like all of these builders already run recipes, they just segue into the nacl shell scripts. Here's one that's running under the context of windows_sdk: https://ci.chromium.org/p/nacl/builders/luci.nacl.toolchain/win-pnacl-x86_32/6

Is it possible that buildbot_selector.py exits non-zero?

ah, it's because the task leaks processes. See the bottom of https://chromium-swarm.appspot.com/task?id=4010fa438791ee10&refresh=10&show_raw=1&wide_logs=true

Failed to delete e:\b\s\w\ir (39 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 2 seconds.
Failed to delete e:\b\s\w\ir (31 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 4 seconds.
Failed to delete e:\b\s\w\ir. The following files remain:
- \\?\e:\b\s\w\ir
- \\?\e:\b\s\w\ir\kitchen-checkout
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-file-l1-2-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-file-l2-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-localization-l1-2-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-processthreads-l1-1-1.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-synch-l1-2-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-timezone-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-convert-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-environment-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-filesystem-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-heap-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-locale-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-math-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-multibyte-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-runtime-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-stdio-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-string-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-time-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-utility-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\msvcp140.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\ucrtbase.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\vcruntime140.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\vctip.exe
Enumerating processes:
- pid 4676; Handles: 119; Exe: e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\win_sdk\bin\..\..\VC\bin\amd64\VCTIP.EXE; Cmd:  
Terminating 1 processes:
- 4676 killed
*** Swarming tried multiple times to delete the run directory and failed ***
*** Hard failing the task ***

Swarming detected that your testing script ran an executable, which may have
started a child executable, and the main script returned early, leaving the
children executables playing around unguided.

You don't want to leave children processes outliving the task on the Swarming
bot, do you? The Swarming bot doesn't.

How to fix?
- For any process that starts children processes, make sure all children
  processes terminated properly before each parent process exits. This is
  especially important in very deep process trees.
  - This must be done properly both in normal successful task and in case of
    task failure. Cleanup is very important.
- The Swarming bot sends a SIGTERM in case of timeout.
  - You have 30.0 seconds to comply after the signal was sent to the process
    before the process is forcibly killed.
- To achieve not leaking children processes in case of signals on timeout, you
  MUST handle signals in each executable / python script and propagate them to
  children processes.
  - When your test script (python or binary) receives a signal like SIGTERM or
    CTRL_BREAK_EVENT on Windows), send it to all children processes and wait for
    them to terminate before quitting.

See
https://chromium.googlesource.com/infra/luci/luci-py.git/+/master/appengine/swarming/doc/Bot.md#Graceful-termination_aka-the-SIGTERM-and-SIGKILL-dance
for more information.

*** May the SIGKILL force be with you ***

Also looks like this is trying to upload stuff to a bucket (but doesn't annotate the step correctly as red): https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8934828408077614336/+/steps/libdl__build_/0/stdout

I'm not sure which cloud project owns gs://nativeclient-once, so I can't add permissions to it :/

Also it seems to be using chromium.archive@gmail.com somehow?

nativeclient-bot is the project that owns nativeclient-once.

Assigning this to dschuff to look into the weird download/upload using chromium.archive@gmail.com permissions. 

Derek, the logs show the following: AccessDeniedException: 403 chromium.archive@gmail.com does not have storage.objects.create access to nativeclient-once

What's happening here? Are there special permissions needed for these builders? 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7b15b3a088901e62e76a74c4a2303d070f8c5f17

commit 7b15b3a088901e62e76a74c4a2303d070f8c5f17
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Sep 21 00:00:49 2018

Roll src/native_client b64cb7b68093..9f85491d8830 (1 commits)

https://chromium.googlesource.com/native_client/src/native_client.git/+log/b64cb7b68093..9f85491d8830


git log b64cb7b68093..9f85491d8830 --date=short --no-merges --format='%ad %ae %s'
2018-09-20 iannucci@chromium.org Whitespace to trigger additional builds.


Created with:
  gclient setdep -r src/native_client@9f85491d8830

The AutoRoll server is located here: https://autoroll.skia.org/r/nacl-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:861512 
TBR=mseaborn@chromium.org

Change-Id: Ib74c016615152c02dea0c08364c6e443524349f8
Reviewed-on: https://chromium-review.googlesource.com/1237353
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#593011}
[modify] https://crrev.com/7b15b3a088901e62e76a74c4a2303d070f8c5f17/DEPS

@efoo These builders need to upload their build artifacts to a storage bucket owned by nativeclient-bot. I'm not sure how they were set up or authenticated before. But let's back up and say, what is the right way to give them the right credentials in the LUCI world?

I believe you need to add permissions to the service account nacl-toolchain-builder@chops-service-accounts.iam.gserviceaccount.com. This is defined in https://chromium.googlesource.com/native_client/src/native_client/+/infra/config/cr-buildbucket.cfg

Builders migrated to LUCI. Builders removed from Buildbot