New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Task

Blocked on:
issue 868207



Sign in to add a comment
link

Issue 861512: Migrate "win-pnacl-x86_32" to LUCI

Reported by luci-mig...@appspot.gserviceaccount.com, Jul 7 2018 Project Member

Issue description

Migrate builder client.nacl.toolchain:win-pnacl-x86_32 to LUCI.

Buildbot: https://ci.chromium.org/buildbot/client.nacl.toolchain/win-pnacl-x86_32
LUCI: https://ci.chromium.org/buildbucket/luci.nacl.toolchain/win-pnacl-x86_32

Migration app will be posting updates on changes of the migration status.
For the latest status, see
https://luci-migration.appspot.com/masters/client.nacl.toolchain/builders/win-pnacl-x86_32

Migration app will close this bug when the builder is entirely migrated from Buildbot to LUCI.
 

Comment 1 by efoo@chromium.org, Jul 10 2018

Labels: LUCI-Clients-Mass LUCI-Clients LUCI-Mig-Client

Comment 2 by efoo@chromium.org, Jul 10 2018

Labels: -Restrict-View-Google

Comment 3 by efoo@chromium.org, Jul 10 2018

Labels: Pri-1
Set priority to 1 for end of Q3 completion of public client migration

Comment 4 by efoo@chromium.org, Jul 27 2018

Blockedon: 868204
Builder blocked on master bug 868204

Comment 5 by efoo@chromium.org, Jul 27 2018

Blockedon:

Comment 6 by efoo@chromium.org, Jul 27 2018

Blockedon: 868207
Builder is blocked on master bug 868207

Comment 7 by efoo@chromium.org, Aug 11

Cc: dpranke@chromium.org dschuff@chromium.org
Labels: LUCI-Mirror-Red
Owner: hinoka@chromium.org
Status: Assigned (was: Available)
Builder mirrored to LUCI and latest builds are failing due to MSVC issue. See builds at: https://ci.chromium.org/p/nacl/g/main/console

Recommended action: Fix and verify. 

Assigned to hinoka@ to try out a recipe change. Add label for tracking. CC'ed dpranke@ since he's looking into similar issues for Crashpad builders.

Comment 8 by efoo@chromium.org, Sep 8

Friendly ping to hinoka@. Have you had the chance to try the recipe change to fix these?

Comment 9 by efoo@chromium.org, Sep 13

Owner: iannucci@chromium.org
Assigning directly to iannucci to see if using MSVC SDK from CIPD will fix this issue. Thanks Robbie!

Comment 10 by iannucci@chromium.org, Sep 19

So it appears that:
  * SCons scans for msvc and fails if it can't find it
  * SCons is then configured to NOT USE MSVC AT ALL (but use clang instead).
  * Since it never actually ran any msvc processes, mspdbsrv is not running
  * The recipe currently counts it as a fatal flaw if it can't kill mspdbsrv (no one has run into this because everything that uses msvc actually uses the binaries it installs).

I'll change the windows_sdk module to not treat the taskkill as fatal, but it would be great if someone close to pnacl could remove this bogus dependency on msvc.

Comment 11 by bugdroid1@chromium.org, Sep 19

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/77900be4e71e5ccda7d9efeca7bbff6dc5f3c597

commit 77900be4e71e5ccda7d9efeca7bbff6dc5f3c597
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Sep 19 01:33:43 2018

[windows_sdk] Allow taskkill mspdbsrv to fail.

It looks like PNaCl actually depends on MSVC's presance, but doesn't
actually run any compile steps with it (instead using clang). That means
that mspdbsrv never actually runs during their build, and then the
taskkill fails.

R=hinoka@chromium.org, tandrii@chromium.org

Bug:  861512 
Change-Id: I004d28f198224adaf16b1d3f14401b0d2a7d700b
Reviewed-on: https://chromium-review.googlesource.com/1232885
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/77900be4e71e5ccda7d9efeca7bbff6dc5f3c597/recipes/README.recipes.md
[modify] https://crrev.com/77900be4e71e5ccda7d9efeca7bbff6dc5f3c597/recipes/recipe_modules/windows_sdk/api.py

Comment 12 by bugdroid1@chromium.org, Sep 19

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc9f174223aefa0eb0ab45701c005327274f555a

commit fc9f174223aefa0eb0ab45701c005327274f555a
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Wed Sep 19 04:30:40 2018

Roll src/third_party/depot_tools 79c651330b94..77900be4e71e (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/79c651330b94..77900be4e71e


git log 79c651330b94..77900be4e71e --date=short --no-merges --format='%ad %ae %s'
2018-09-19 iannucci@chromium.org [windows_sdk] Allow taskkill mspdbsrv to fail.


Created with:
  gclient setdep -r src/third_party/depot_tools@77900be4e71e

The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:861512 
TBR=agable@chromium.org

Change-Id: Ic6c3aadd5d77ae4b4df7be7265c0109dc30bee65
Reviewed-on: https://chromium-review.googlesource.com/1232625
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#592317}
[modify] https://crrev.com/fc9f174223aefa0eb0ab45701c005327274f555a/DEPS

Comment 13 by dschuff@chromium.org, Sep 19

@iannucci I'm a little confused about how you found out what you said in #10. 
Edit: nevermind, You're looking at the toolchain bot.

I'm looking at https://ci.chromium.org/p/nacl/builders/luci.nacl.ci/win7-64-glibc-opt/21 , specifically https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8935101935740774512/+/steps/scons_compile/0/stdout where it appears that SCons is using the system-installed MSVC, but then eventually failing to kill mspdbserv. Presumably that's what efoo@ meant when he said "using the MSVC SDK from CIPD". I am interested in fixing that btw (i.e. making the NaCl build use the CIPD SDK) but not sure how I would do it without converting all of the builds to recipes.

Also that builder has a GN build https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8935101935740774512/+/steps/gn_compile_64/0/stdout but since it doesn't print its command lines I have no idea what it's actually doing to build.

Comment 14 by iannucci@chromium.org, Sep 20

Ok, so I took another look at some stuff. For the recipe based bots, I added 
nacl-toolchain-builder@chops-service-accounts.iam.gserviceaccount.com to https://chrome-infra-auth.appspot.com/auth/groups/winsdk-cipd-readers, which should let them get the CIPD package.

For the bots not using recipes, if their service account is added to the winsdk-cipd-readers group, they should be able to basically do what the recipe module does (https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/master/recipes/recipe_modules/windows_sdk/api.py#24). Basically:
  * Pull and unpack the SDK from cipd
  * read sdk_dir/win_sdk/bin/SetEnv.x86.json (or x64) to get all the envvar modifications
  * Modify the env accordingly
  * profit?

Comment 15 by dschuff@chromium.org, Sep 20

After #11 we have a puzzling situation in https://ci.chromium.org/p/nacl/builders/luci.nacl.ci/win7-64-glibc-opt/22 wherein all the steps are green, but the result is still red. Any ideas?

Comment 16 by iannucci@chromium.org, Sep 20

Though it looks like all of these builders already run recipes, they just segue into the nacl shell scripts. Here's one that's running under the context of windows_sdk: https://ci.chromium.org/p/nacl/builders/luci.nacl.toolchain/win-pnacl-x86_32/6

Comment 17 by iannucci@chromium.org, Sep 20

Is it possible that buildbot_selector.py exits non-zero?

Comment 18 by iannucci@chromium.org, Sep 20

ah, it's because the task leaks processes. See the bottom of https://chromium-swarm.appspot.com/task?id=4010fa438791ee10&refresh=10&show_raw=1&wide_logs=true

Failed to delete e:\b\s\w\ir (39 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 2 seconds.
Failed to delete e:\b\s\w\ir (31 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 4 seconds.
Failed to delete e:\b\s\w\ir. The following files remain:
- \\?\e:\b\s\w\ir
- \\?\e:\b\s\w\ir\kitchen-checkout
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-file-l1-2-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-file-l2-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-localization-l1-2-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-processthreads-l1-1-1.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-synch-l1-2-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-core-timezone-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-convert-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-environment-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-filesystem-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-heap-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-locale-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-math-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-multibyte-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-runtime-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-stdio-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-string-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-time-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\api-ms-win-crt-utility-l1-1-0.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\msvcp140.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\ucrtbase.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\vcruntime140.dll
- \\?\e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\VC\bin\amd64\vctip.exe
Enumerating processes:
- pid 4676; Handles: 119; Exe: e:\b\s\w\ir\kitchen-checkout\depot_tools\win_toolchain\vs_files\d3cb0e37bdd120ad0ac4650b674b09e81be45616\win_sdk\bin\..\..\VC\bin\amd64\VCTIP.EXE; Cmd:  
Terminating 1 processes:
- 4676 killed
*** Swarming tried multiple times to delete the run directory and failed ***
*** Hard failing the task ***

Swarming detected that your testing script ran an executable, which may have
started a child executable, and the main script returned early, leaving the
children executables playing around unguided.

You don't want to leave children processes outliving the task on the Swarming
bot, do you? The Swarming bot doesn't.

How to fix?
- For any process that starts children processes, make sure all children
  processes terminated properly before each parent process exits. This is
  especially important in very deep process trees.
  - This must be done properly both in normal successful task and in case of
    task failure. Cleanup is very important.
- The Swarming bot sends a SIGTERM in case of timeout.
  - You have 30.0 seconds to comply after the signal was sent to the process
    before the process is forcibly killed.
- To achieve not leaking children processes in case of signals on timeout, you
  MUST handle signals in each executable / python script and propagate them to
  children processes.
  - When your test script (python or binary) receives a signal like SIGTERM or
    CTRL_BREAK_EVENT on Windows), send it to all children processes and wait for
    them to terminate before quitting.

See
https://chromium.googlesource.com/infra/luci/luci-py.git/+/master/appengine/swarming/doc/Bot.md#Graceful-termination_aka-the-SIGTERM-and-SIGKILL-dance
for more information.

*** May the SIGKILL force be with you ***

Comment 19 by iannucci@chromium.org, Sep 20

Also looks like this is trying to upload stuff to a bucket (but doesn't annotate the step correctly as red): https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8934828408077614336/+/steps/libdl__build_/0/stdout

Comment 20 by iannucci@chromium.org, Sep 20

I'm not sure which cloud project owns gs://nativeclient-once, so I can't add permissions to it :/

Comment 21 by iannucci@chromium.org, Sep 20

Also it seems to be using chromium.archive@gmail.com somehow?

Comment 22 by dschuff@chromium.org, Sep 20

nativeclient-bot is the project that owns nativeclient-once.

Comment 23 by efoo@chromium.org, Sep 21

Owner: dschuff@chromium.org
Assigning this to dschuff to look into the weird download/upload using chromium.archive@gmail.com permissions. 

Derek, the logs show the following: AccessDeniedException: 403 chromium.archive@gmail.com does not have storage.objects.create access to nativeclient-once

What's happening here? Are there special permissions needed for these builders?

Comment 24 by bugdroid1@chromium.org, Sep 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7b15b3a088901e62e76a74c4a2303d070f8c5f17

commit 7b15b3a088901e62e76a74c4a2303d070f8c5f17
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Sep 21 00:00:49 2018

Roll src/native_client b64cb7b68093..9f85491d8830 (1 commits)

https://chromium.googlesource.com/native_client/src/native_client.git/+log/b64cb7b68093..9f85491d8830


git log b64cb7b68093..9f85491d8830 --date=short --no-merges --format='%ad %ae %s'
2018-09-20 iannucci@chromium.org Whitespace to trigger additional builds.


Created with:
  gclient setdep -r src/native_client@9f85491d8830

The AutoRoll server is located here: https://autoroll.skia.org/r/nacl-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:861512 
TBR=mseaborn@chromium.org

Change-Id: Ib74c016615152c02dea0c08364c6e443524349f8
Reviewed-on: https://chromium-review.googlesource.com/1237353
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#593011}
[modify] https://crrev.com/7b15b3a088901e62e76a74c4a2303d070f8c5f17/DEPS

Comment 25 by dschuff@chromium.org, Sep 21

@efoo These builders need to upload their build artifacts to a storage bucket owned by nativeclient-bot. I'm not sure how they were set up or authenticated before. But let's back up and say, what is the right way to give them the right credentials in the LUCI world?

Comment 27 by efoo@chromium.org, Dec 4

Owner: efoo@chromium.org
Status: Fixed (was: Assigned)
Builders migrated to LUCI. Builders removed from Buildbot

Comment 28 by luci-mig...@appspot.gserviceaccount.com, Dec 17

Project Member
Description: Show this description

Sign in to add a comment