Looking at the two "llvm_src (source)" steps on the two ci.chromium.org builds linked to in comment 0.

https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8927954871956171664/+/steps/llvm_src__source_/0/stdout (win) has

command:
'https://chromium.googlesource.com/native_client/pnacl-llvm.git'
'https://chromium.googlesource.com/native_client/pnacl-llvm.git'
'%(output)s'
[('http://git.chromium.org/native_client/', 'https://chromium.googlesource.com/native_client/')]
[('http://git.chromium.org/native_client/', 'https://chromium.googlesource.com/native_client/'), ('https://chromium.googlesource.com/a/native_client/', 'https://chromium.googlesource.com/native_client/'), ('https://chromium.googlesource.com/native_client/', 'https://chromium.googlesource.com/native_client/'), ('ssh://gerrit.chromium.org/native_client/', 'https://chromium.googlesource.com/native_client/')]
6e4d678f9a721c07ee2bafdc1f27d48fbf917167

...

Populating Cache: C:\b\s\w\ir\kitchen-workdir\native_client\toolchain_build\git_cache [https://chromium.googlesource.com/native_client/pnacl-llvm.git]

...

From https://chromium.googlesource.com/a/native_client/pnacl-llvm
 * [new branch]      bradnelson/pnacl-in-pnacl  -> bradnelson/pnacl-in-pnacl






while https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8926740418554785952/+/steps/llvm_src__source_/0/stdout (linux) has 

command:
'https://chromium.googlesource.com/native_client/pnacl-llvm.git'
'https://chromium.googlesource.com/native_client/pnacl-llvm.git'
'%(output)s'
[('http://git.chromium.org/native_client/', 'https://chromium.googlesource.com/native_client/')]
[('http://git.chromium.org/native_client/', 'https://chromium.googlesource.com/native_client/'), ('https://chromium.googlesource.com/a/native_client/', 'https://chromium.googlesource.com/native_client/'), ('https://chromium.googlesource.com/native_client/', 'https://chromium.googlesource.com/native_client/'), ('ssh://gerrit.chromium.org/native_client/', 'https://chromium.googlesource.com/native_client/')]
6e4d678f9a721c07ee2bafdc1f27d48fbf917167

...

Populating Cache: /b/s/w/ir/kitchen-workdir/native_client/toolchain_build/git_cache [https://chromium.googlesource.com/native_client/pnacl-llvm.git]

...

From https://chromium.googlesource.com/a/native_client/pnacl-llvm
 * [new branch]            bradnelson/pnacl-in-pnacl  -> bradnelson/pnacl-in-pnacl



Hmmm that looks very similar.

The clang_src step is probably more relevant, but that too looks similar.

Linux: https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8926740418554785952/+/steps/clang_src__source_/0/stdout

Win: https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8927954871956171664/+/steps/clang_src__source_/0/stdout

The following revision refers to this bug:
  https://chromium.googlesource.com/native_client/src/native_client.git/+/84d6fbfa44f1de95e21107766fc35fa1f6ac0897

commit 84d6fbfa44f1de95e21107766fc35fa1f6ac0897
Author: Nico Weber <thakis@chromium.org>
Date: Fri Jan 18 00:40:40 2019

Add some debug logging.

Bug: chromium:923062
Change-Id: I5e1a59bbda0095a72642f56a55550aea1cc2bf9f
Reviewed-on: https://chromium-review.googlesource.com/c/1418292
Reviewed-by: Derek Schuff <dschuff@chromium.org>

[modify] https://crrev.com/84d6fbfa44f1de95e21107766fc35fa1f6ac0897/toolchain_build/command.py

The following revision refers to this bug:
  https://chromium.googlesource.com/native_client/src/native_client.git/+/db511ff7952658efc9bf67afa8dfdac9faba2f93

commit db511ff7952658efc9bf67afa8dfdac9faba2f93
Author: Nico Weber <thakis@chromium.org>
Date: Fri Jan 18 13:18:55 2019

Add some more logging.

TBR=dschuff

Bug: chromium:923062
Change-Id: Ice65b2cee689b42e2bc8f26218d2e73adee6282f
Reviewed-on: https://chromium-review.googlesource.com/c/1421477
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/db511ff7952658efc9bf67afa8dfdac9faba2f93/toolchain_build/command.py

The logging added in comment 4 and 5 doesn't show anything interesting. That repo has been cleaned previously, so there's no .git dir, so the code in https://cs.chromium.org/chromium/src/native_client/toolchain_build/command.py?type=cs&sq=package:chromium&g=0&l=389 (update_valid_mirrors) never runs. That's the only thing reading SyncGitRepoCmds()'s push_mirrors arg, which I think is the only thing referencing ALT_GIT_BASE_URL (which is the git repo with the /a/ in it). Weird.


However! One difference in the two outputs listed in comment 3 is that the "Populating Cache:" step looks very different. Linux:

Populating Cache: /b/s/w/ir/kitchen-workdir/native_client/toolchain_build/git_cache [https://chromium.googlesource.com/native_client/pnacl-clang.git]
Running: subprocess.check_call(['git', 'cache', 'populate', '-c', '.', 'https://chromium.googlesource.com/native_client/pnacl-clang.git'], cwd='/b/s/w/ir/kitchen-workdir/native_client/toolchain_build/git_cache')
Downloading gs://chromium-git-cache/chromium.googlesource.com-native_client-pnacl--clang/56667.zip
Copying gs://chromium-git-cache/chromium.googlesource.com-native_client-pnacl--clang/56667.zip...
/ [0 files][    0.0 B/ 86.2 MiB]                                                
-
- [1 files][ 86.2 MiB/ 86.2 MiB]
Operation completed over 1 objects/86.2 MiB.
download took 0.0 minutes
Archive:  ./_cache_tmprMNxvs/56667.zip
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/config
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/info/
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/info/exclude
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/info/refs
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/pre-push.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/prepare-commit-msg.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/pre-commit.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/fsmonitor-watchman.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/commit-msg.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/applypatch-msg.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/pre-receive.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/post-update.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/pre-applypatch.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/pre-rebase.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/hooks/update.sample
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/number.idx
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/objects/
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/objects/info/
 extracting: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/objects/info/packs
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/objects/pack/
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/objects/pack/pack-bb03803c50826de80a2ee94d838d78ea4c9d30df.pack
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/objects/pack/pack-bb03803c50826de80a2ee94d838d78ea4c9d30df.idx
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/description
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/packed-refs
 extracting: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/HEAD
  inflating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/FETCH_HEAD
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/refs/
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/refs/number/
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/refs/tags/
   creating: ./_cache_tmpT2bt1Gchromium.googlesource.com-native_client-pnacl--clang/refs/heads/
unzip took 0.0 minutes



Windows:
Populating Cache: C:\b\s\w\ir\kitchen-workdir\native_client\toolchain_build\git_cache [https://chromium.googlesource.com/native_client/pnacl-clang.git]
Running: subprocess.check_call(['cmd.exe', '/c', 'git.bat', 'cache', 'populate', '-c', '.', 'https://chromium.googlesource.com/native_client/pnacl-clang.git'], cwd='C:\\b\\s\\w\\ir\\kitchen-workdir\\native_client\\toolchain_build\\git_cache')
bash: warning: setlocale: LC_ALL: cannot change locale (English_United States.UTF-8): No such file or directory
No bootstrap file for .\chromium.googlesource.com-native_client-pnacl--clang found in chromium-git-cache, stderr:
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] goroutine 1:
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #0 go.chromium.org/luci/vpython/python/find.go:87 - python.Find()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]   annotation #0:
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]     reason: no Python found
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]   annotation #1:
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]     reason: could not find executable for: "python"
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]   annotation #2:
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]     reason: could not find target in system
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]     internal reason: target(python)/dirs([C:\b\s\w\ir\cipd_bin_packages\mingw64\libexec\git-core C:\b\s\w\ir\cipd_bin_packages\mingw64\bin C:\b\s\w\ir\cipd_bin_packages\usr\bin C:\b\s\w\ir\tmp\git_home_task\bin C:\b\s\w\ir\cache\windows_sdk\win_sdk\bin\10.0.17134.0\x64 C:\b\s\w\ir\cache\windows_sdk\VC\Tools\MSVC\14.14.26428\bin\HostX64\x64 C:\b\s\w\ir\cipd_bin_packages\ C:\b\s\w\ir\cipd_bin_packages\usr\bin C:\b\s\cipd_cache\bin C:\Windows\system32 C:\Windows C:\Windows\System32\Wbem C:\Windows\System32\WindowsPowerShell\v1.0 C:\Program Files\Puppet Labs\Puppet\bin C:\b\s\w\ir\kitchen-checkout\depot_tools C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32 C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32 C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32 C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32])
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/config.go:272 - venv.(*Config).resolvePythonInterpreter()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]   reason: could not find Python for:
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #2 go.chromium.org/luci/vpython/venv/config.go:187 - venv.(*Config).makeEnv()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]   reason: failed to resolve system Python interpreter
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241]
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:163 - venv.With()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #4 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #5 go.chromium.org/luci/vpython/application/application.go:318 - application.(*application).mainImpl()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #6 go.chromium.org/luci/vpython/application/application.go:402 - application.(*Config).Main.func1()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #7 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #8 go.chromium.org/luci/vpython/application/application.go:401 - application.(*Config).Main()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #9 vpython/main.go:93 - main.mainImpl()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #10 vpython/main.go:99 - main.main()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #11 runtime/proc.go:201 - runtime.main()
  [E2018-12-05T11:01:15.299315-08:00 5496 0 annotate.go:241] #12 runtime/asm_amd64.s:1333 - runtime.goexit()

running "git init --bare" in ".\_cache_tmpqp_frcchromium.googlesource.com-native_client-pnacl--clang"
Initialized empty Git repository in C:/b/s/w/ir/kitchen-workdir/native_client/toolchain_build/git_cache/_cache_tmpqp_frcchromium.googlesource.com-native_client-pnacl--clang/




So I'm guessing Linux (and Mac) gets the /a/ from that cache, and on Windows something can't find python, the cache isn't used, and the regular checkout code (which I've been looking at) doesn't have that /a/.

iannucci, the stack in comment 6 looks pretty similar to issue 812693 -- do the nacl win bots need whatever fix you did over there?

Likely broken by https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/849663/ which made git-cache use vpython instead of python. (That change is from early 2018, but we haven't built pnacl packages in a while.)

Here's the swarming output for a win-pnacl-x86_32 run: https://chromium-swarm.appspot.com/task?id=4277317037cc6d10&refresh=10&show_raw=1&wide_logs=true

For comparison, here's an arbitrary swarming task of a main waterfall win bot: https://chromium-swarm.appspot.com/task?id=427a34dc8312b010&refresh=10&show_raw=1&wide_logs=true

The latter has

caches: builder_046e3ec71eedfeafdeda4d1273ca8730cf81163c1e0b4fe8ea36f810f7466458_v2 git goma_v2 vpython


at the top, the former doesn't. Maybe that "vpython" is where python is supposed to come from?

+vadimsh who also has touched vpython things a bit. iannucci, vadimsh, git-clone calls vpython which then can't find python on this bot. Later build steps on the same bot run it fine  (see "Command" at e.g. https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8924037594781264592/+/steps/driver_tests_x86-32/0/stdout), but git-clone can't find it. The recipe for these bots is scripts/slave/recipes/nacl.py ; it's probably missing the magic "vpython, here's where you find python" line (maybe something via cipd?), but there are enough layers of things (recipes cipd vpython swarming luci) that I don't know what to add or even how to debug. Help?

(s/iannucci@chromium/iannucci@google)

Hm... That first task you link (4277317037cc6d10) starts with vpython & cpython both in $PATH. I'm not sure why it would have difficulty finding python either. On first inspection, this looks identical to ~all other windows tasks we've run for the past year or so.

I'm going to start from the top.

(the /a/ bit means "force authentication, even for otherwise public repos" I'm not sure why that differs, but it's probably not the root cause).

(the caches are also entirely optional, i.e. not the source of truth for anything)

Oh, I see; something is picking up /a/ in the version string. Is it an option to remove the /a/ from the repo URL everywhere instead? The /a/ thing is a gerrit artifact and is not the canonical repo name.

That's fine too, but for that we also need to understand where it comes from, right?

Currently the linux bot gets it but the windows bot doesn't, and I don't understand why.

As for "why isn't vpython finding python", I'm not sure, but the paths that it searches:

C:\b\s\w\ir\cipd_bin_packages\mingw64\libexec\git-core 
C:\b\s\w\ir\cipd_bin_packages\mingw64\bin 
C:\b\s\w\ir\cipd_bin_packages\usr\bin 
C:\b\s\w\ir\tmp\git_home_task\bin 
C:\b\s\w\ir\cache\windows_sdk\win_sdk\bin\10.0.17134.0\x64 C:\b\s\w\ir\cache\windows_sdk\VC\Tools\MSVC\14.14.26428\bin\HostX64\x64 
C:\b\s\w\ir\cipd_bin_packages\ 
C:\b\s\w\ir\cipd_bin_packages\usr\bin 
C:\b\s\cipd_cache\bin 
C:\Windows\system32 
C:\Windows 
C:\Windows\System32\Wbem 
C:\Windows\System32\WindowsPowerShell\v1.0 
C:\Program Files\Puppet Labs\Puppet\bin 
C:\b\s\w\ir\kitchen-checkout\depot_tools 
C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32 
C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32 
C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32 
C:\b\s\w\ir\cache\vpython\d2cd22\lib\site-packages\pywin32_system32

Do not contain the actual cpython location, which is in 

C:\b\s\w\ir\cipd_bin_packages\bin

Is it possible that there's some mingw/cygwin $PATH muckery that's happening? In particular, 
C:\b\s\w\ir\cipd_bin_packages\usr\bin isn't a directory that actually exists.

(So, actually, I AM sure why it doesn't find it, I'm NOT sure why $PATH looks so weird)

Unfortunately, it looks like this builder is using annotated steps, so I have no idea what script/command line arguments correspond to "clang_src (source)" or what environment variables it's run with :(.

https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8926740418554785952/+/steps/annotated_steps/0/stdout -> buildbot/buildbot_pnacl_toolchain.py --buildbot --tests-arch x86-64 (that stdout also has the env vars)

That script is at https://cs.chromium.org/chromium/src/native_client/buildbot/buildbot_pnacl_toolchain.py?q=buildbot_pnacl_toolchain.py&sq=package:chromium&dr

Right, that tells me "the build now defers to this totally custom script for the remainder of the build". The plus side of this is that you have "more control". The downside of this is that no one outside of the script maintainers are familiar with it.

"clang_src" doesn't seem to appear in this directory: https://cs.chromium.org/search/?q=clang_src+file:%5Esrc/native_client/buildbot/+package:%5Echromium$&type=cs

(i.e. I can't tell if the envvars change at all before the step runs, what script that particular step is running, what args it's being passed, etc)

My guess at this point is that msys is goobering the path, but I don't know why it's only affecting your builder.

I found this super confusing too. (Me being unfamiliar with both nacl and infra code meant that the recipe/cipd/luci/etc stuff confused me equally though.)

Here's a short summary of what I discovered yesterday. That script calls https://cs.chromium.org/chromium/src/native_client/buildbot/buildbot_pnacl_toolchain.py?type=cs&q=ToolchainBuildCmd&g=0&l=86 which shells out to toolchain_build/toolchain_build_pnacl.py (https://cs.chromium.org/chromium/src/native_client/toolchain_build/toolchain_build_pnacl.py?q=toolchain_build_pnacl&dr=C&l=1379).

That script has clang_src here: https://cs.chromium.org/chromium/src/native_client/toolchain_build/toolchain_build_pnacl.py?q=toolchain_build_pnacl&dr=C&l=605

It has a mechanism for building up arbitrary "command" objects which through once.py cache their results on cloud storage -- so no code runs immediately, instead a list of commands is produced and that then runs later.

GetGitSyncCmds (here: https://cs.chromium.org/chromium/src/native_client/toolchain_build/toolchain_build_pnacl.py?type=cs&q=GetGitSyncCmds&g=0&l=569) produces the commands for pulling a repo.

The list at the top of https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8926740418554785952/+/steps/clang_src__source_/0/stdout ("command:" etc) is the arguments passed to the command object, followed by a hash of the command (6e4d678f9a721c07ee2bafdc1f27d48fbf917167) for once (...which for git operations doesn't make that much sense; the main idea is to get build outputs off cloud storage if the code hasn't changed from what I understand).

Then below "======================================================================" is the output from executing all the commands. (There's also a should_run filter, but I think that's always true here and all commands run.)


"Cleaning Git Working Directory: is https://cs.chromium.org/chromium/src/native_client/toolchain_build/command.py?type=cs&g=0&l=509

"Populating Cache:" is https://cs.chromium.org/chromium/src/native_client/toolchain_build/command.py?type=cs&g=0&l=436

(This mostly calls `Running: subprocess.check_call(['git', 'cache', 'populate', '-c', '.', 'https://chromium.googlesource.com/native_client/pnacl-clang.git'], cwd='/b/s/w/ir/kitchen-workdir/native_client/toolchain_build/git_cache')` from depot_tools.)


"Validating Repo:" and "Syncing Git Repo:" are also in command.py.

"Validating Repo:" just early-returns, and "Syncing Git Repo:" again calls depot_tools.

This is weird too; the binutils_x86_src step DOESN'T blow up (https://logs.chromium.org/logs/nacl/buildbucket/cr-buildbucket.appspot.com/8927954871956171664/+/steps/binutils_x86_src__source_/0/stdout).

Sidenote: I didn't realize this extensive use of `git cache` existed outside of bot_update... it's definitely not the most robust code :(

Actually, on that note, bot_update directly invokes git_cache.py, not via cmd/git.bat. I don't have a windows machine to test, but is it possible that this is actually just doesn't work? https://cs.chromium.org/chromium/tools/depot_tools/recipes/recipe_modules/bot_update/resources/bot_update.py?q=bot_update.py&sq=package:chromium&g=0&l=84

(gclient, which is the only other use of git cache on windows that I know of, directly imports the `git_cache` module).

What's cmd/git.bat? I can't find it on cs.chromium.org.

From the Populating Cache call:

subprocess.check_call(['cmd.exe', '/c', 'git.bat', 'cache', 'populate', '-c', '.', 'https://chromium.googlesource.com/native_client/pnacl-clang.git'], cwd='C:\\b\\s\\w\\ir\\kitchen-workdir\\native_client\\toolchain_build\\git_cache')

This does (roughly):
  cmd.exe
    git.bat 
      git.exe
        bash.exe git-cache
          bash python_runner.sh
            python.exe git_cache.py
         

And what I'm wondering is if the 'git.exe' or 'bash.exe' in the middle are messing with PATH (if '..../bin' is special-cased in msys)

The following revision refers to this bug:
  https://chromium.googlesource.com/native_client/src/native_client.git/+/321c47ddeedcf7e87ffd2c127500e4ef9888fb69

commit 321c47ddeedcf7e87ffd2c127500e4ef9888fb69
Author: Nico Weber <thakis@chromium.org>
Date: Tue Jan 22 16:11:14 2019

Add even more debug logging.

TR=dschuff
Bug: chromium:923062
Change-Id: I691153c007a7571fe71094be88a0bab17a1a2808
Reviewed-on: https://chromium-review.googlesource.com/c/1426819
Reviewed-by: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/321c47ddeedcf7e87ffd2c127500e4ef9888fb69/pynacl/repo_tools.py

I added logging that prints `git remote -v`. On both Windows and Linux, it prints

origin	https://chromium.googlesource.com/a/native_client/pnacl-clang.git (fetch)
origin	https://chromium.googlesource.com/a/native_client/pnacl-clang.git (push)

in toolchain_build/src/clang.

So the /a/ is there on Windows, but somehow it gets lost by the time llvm's build gets there? Or the Windows build somehow has the old URL in its once.py cache for some reason and hasn't had a reason to rebuild that entry and the cache doesn't have a dep on the remote?

dschuff, is there a way to blow away the once.py cache and let the bot build a fresh package for Windows? Maybe that's all we need?