Oh, E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap\tools\clang\test\Modules\Output\dependency-dump.m.tmp is a directory, and it has a long subfile in its vfs\ subdir. Deleting that fails, and that path is probably longer than 260 chars.

http://gnuwin32.sourceforge.net/packages/coreutils.htm contains rm's source (.../src/remove.c is the library bit, the thin binary wrapper is src/rm.c)

If I ssh into the bot and run that rm command, it runs fine and seems to do the deed.

If I manually run the test a few times, that works fine too:

chrome-bot@VM166-M4 ~
$ cmd
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\chrome-bot>cd E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap
cd E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap

C:\Users\chrome-bot>e:
e:

E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
'python' is not recognized as an internal or external command,
operable program or batch file.
 ' is not recognized as an internal or external command,
operable program or batch file.ng\build\src\third_party\llvm-bootstrap>set PATH=%PATH%;e:\b\depot_tools


E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
-- Testing: 1 tests, 1 threads --
PASS: Clang :: Modules/dependency-dump.m (1 of 1)
Testing Time: 0.75s
  Expected Passes    : 1
llvm-lit.py: lit.cfg:195: note: using clang: 'E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-bootstrap/./bin/clang.EXE'

E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
-- Testing: 1 tests, 1 threads --
PASS: Clang :: Modules/dependency-dump.m (1 of 1)
Testing Time: 0.11s
  Expected Passes    : 1
llvm-lit.py: lit.cfg:195: note: using clang: 'E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-bootstrap/./bin/clang.EXE'




...oh, but that's with cygwin's rm, if I tell it to use the gnuwin one it fails:

E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>where rm
where rm
C:\cygwin\bin\rm.exe

E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>set PATH=e:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build-tools\gnuwin;%PATH%
set PATH=e:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build-tools\gnuwin;%PATH%

E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>where rm
where rm
e:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build-tools\gnuwin\rm.exe
C:\cygwin\bin\rm.exe
 ' is not recognized as an internal or external command,
operable program or batch file.ng\build\src\third_party\llvm-bootstrap>

E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-bootstrap>python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
python bin\llvm-lit.py ..\llvm\tools\clang\test\Modules\dependency-dump.m
-- Testing: 1 tests, 1 threads --
FAIL: Clang :: Modules/dependency-dump.m (1 of 1)
Testing Time: 0.08s
********************
Failing Tests (1):
    Clang :: Modules/dependency-dump.m

  Unexpected Failures: 1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/051cfcb6697da79bd57d47e923505d2ec806cc3a

commit 051cfcb6697da79bd57d47e923505d2ec806cc3a
Author: thakis <thakis@chromium.org>
Date: Thu Apr 14 22:41:13 2016

clang/win update.py: Put gnuwin last in path.

gnuwin's rm.exe doesn't seem to work with long path names on the bots for some
reason.  The bots have cygwin installed and have a better working rm.exe
earlier in the path, so try to use this.  (Local devs don't have long paths.)

BUG= 603364 
NOTRY=true

Review URL: https://codereview.chromium.org/1887163003

Cr-Commit-Position: refs/heads/master@{#387463}

[modify] https://crrev.com/051cfcb6697da79bd57d47e923505d2ec806cc3a/tools/clang/scripts/update.py

Giving it a try here: https://build.chromium.org/p/tryserver.chromium.win/builders/win_upload_clang/builds/19

that's off revision 861e99cb57a0e240c836986efcb500c9da6f25c9 , is that new enough?

grr, i keep forgetting it's not using ToT
trying again: https://build.chromium.org/p/tryserver.chromium.win/builders/win_upload_clang/builds/21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6678c06e28f094cf1d32d2b4cd723b8bb4cdc48c

commit 6678c06e28f094cf1d32d2b4cd723b8bb4cdc48c
Author: thakis <thakis@chromium.org>
Date: Fri Apr 15 14:15:50 2016

Revert of clang/win update.py: Put gnuwin last in path. (patchset #1 id:1 of https://codereview.chromium.org/1887163003/ )

Reason for revert:
Didn't help: Cygwin is in the PATH when ssh'ing in, but not for normal build steps on the bot.

Original issue's description:
> clang/win update.py: Put gnuwin last in path.
>
> gnuwin's rm.exe doesn't seem to work with long path names on the bots for some
> reason.  The bots have cygwin installed and have a better working rm.exe
> earlier in the path, so try to use this.  (Local devs don't have long paths.)
>
> BUG= 603364 
> NOTRY=true
>
> Committed: https://crrev.com/051cfcb6697da79bd57d47e923505d2ec806cc3a
> Cr-Commit-Position: refs/heads/master@{#387463}

TBR=hans@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 603364 

Review URL: https://codereview.chromium.org/1888353003

Cr-Commit-Position: refs/heads/master@{#387595}

[modify] https://crrev.com/6678c06e28f094cf1d32d2b4cd723b8bb4cdc48c/tools/clang/scripts/update.py

C:\src>type makelongdir.py
import os

n = '\\\\?\\c:\\src\\'
for i in range(30):
  n += 'a' * 30 + '\\'
  try:
    os.mkdir(n)
  except WindowsError: pass

open(n + 'file', 'w').write('hi\n')

C:\src>python makelongdir.py

C:\src>rm -rf c:\src\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

C:\src>echo %ERRORLEVEL%
-1073741819

C:\src>rm -rf \\?\c:\src\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

C:\src>echo %ERRORLEVEL%
0


I wonder if we should let lit prepend paths with \\?\ on Windows. At least when calling rm.

Hans asks what \\?\ means: It's the "here comes a path longer than 260 chars" marker in Windows. https://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx#maxpath

I tried this:

C:\src\llvm-rw>svn diff
Index: utils/lit/lit/TestRunner.py
===================================================================
--- utils/lit/lit/TestRunner.py (revision 266986)
+++ utils/lit/lit/TestRunner.py (working copy)
@@ -548,6 +548,9 @@
         tmpDir = tmpDir.replace('\\', '/')
         tmpBase = tmpBase.replace('\\', '/')

+    if kIsWindows:
+      tmpBase = '\\\\?\\' + tmpBase
+
     # We use #_MARKER_# to hide %% while we do the other substitutions.
     substitutions = []
     substitutions.extend([('%%', '#_MARKER_#')])


It breaks a ton of tests; I guess clang (and the other llvm binaries) can't handle extended windows path names well.

Oh hm, it looks like %ERRORLEVEL% is 0 for a \\?\ path but it doesn't actually remove that directory then either.

(Fun fact, rmdir isn't a fan either:

C:\src>rmdir /s/q \\?\c:\src\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
The path \\?\c:\src\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA
~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\AAAAAA~1\
AAAAAA~1\AAAAAA~1\AAAAAA~1\.. is too long.
)

I replaced gnuwin's rm.exe with msys's, and now the first phase tests all pass: https://build.chromium.org/p/tryserver.chromium.win/builders/win_upload_clang/builds/25/steps/package%20clang/logs/stdio

Sadly, ExecutionEngine/OrcMCJIT/load-object-a.ll failed in the second phase, so still no green run on that bot :-( (also, the windows slave takes 2h10m for this, which is also not so great)

FAIL: LLVM :: ExecutionEngine/OrcMCJIT/load-object-a.ll (19661 of 26525)
******************** TEST 'LLVM :: ExecutionEngine/OrcMCJIT/load-object-a.ll' FAILED ********************
Script:
--
rm -rf E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2 E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3
mkdir -p E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2 E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3
E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-build/Release+Asserts/./bin\lli.EXE -mtriple=x86_64-pc-win32-elf -jit-kind=orc-mcjit -extra-module=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT/Inputs/multi-module-b.ll -extra-module=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT/Inputs/multi-module-c.ll -enable-cache-manager -object-cache-dir=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT\load-object-a.ll
find E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir -type f -name 'multi-module-?.o' -exec mv -v '{}' E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2 ';'
E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-build/Release+Asserts/./bin\lli.EXE -mtriple=x86_64-pc-win32-elf -jit-kind=orc-mcjit -extra-object=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2/multi-module-b.o -extra-object=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2/multi-module-c.o E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT\load-object-a.ll
E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-build/Release+Asserts/./bin\llvm-ar.EXE r E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3/load-object.a E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2/multi-module-b.o
E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-build/Release+Asserts/./bin\llvm-ar.EXE r E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3/load-object.a E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2/multi-module-c.o
E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-build/Release+Asserts/./bin\lli.EXE -mtriple=x86_64-pc-win32-elf -jit-kind=orc-mcjit -extra-archive=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3/load-object.a E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT\load-object-a.ll
--
Exit Code: -1073741819

Command Output (stdout):
--
Command 0: "rm" "-rf" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3"
Command 0 Result: 0
Command 0 Output:


Command 0 Stderr:


Command 1: "mkdir" "-p" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir3"
Command 1 Result: 0
Command 1 Output:


Command 1 Stderr:


Command 2: "E:/b/build/slave/win_upload_clang/build/src/third_party/llvm-build/Release+Asserts/./bin\lli.EXE" "-mtriple=x86_64-pc-win32-elf" "-jit-kind=orc-mcjit" "-extra-module=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT/Inputs/multi-module-b.ll" "-extra-module=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT/Inputs/multi-module-c.ll" "-enable-cache-manager" "-object-cache-dir=E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm\test\ExecutionEngine\OrcMCJIT\load-object-a.ll"
Command 2 Result: 0
Command 2 Output:


Command 2 Stderr:


Command 3: "find" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir" "-type" "f" "-name" "multi-module-?.o" "-exec" "mv" "-v" "{}" "E:\b\build\slave\win_upload_clang\build\src\third_party\llvm-build\Release+Asserts\test\ExecutionEngine\OrcMCJIT\Output\load-object-a.ll.tmp.cachedir2" ";"
Command 3 Result: -1073741819
Command 3 Output:


Command 3 Stderr:



That's the same exit code as above. The second stage build has a slightly longer build dir name, so I guess I'll try replacing find.exe and mv.exe too.

The test failed again on a second run (https://build.chromium.org/p/tryserver.chromium.win/builders/win_upload_clang/builds/27) but everything passed with msys find and mv and rm: https://build.chromium.org/p/tryserver.chromium.win/builders/win_upload_clang/builds/28 \o/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8aa292dd634877fc41e5844af23d1dee28694f41

commit 8aa292dd634877fc41e5844af23d1dee28694f41
Author: thakis <thakis@chromium.org>
Date: Tue Apr 26 00:24:36 2016

roll clang 266460:267383

Also switch to gnuwin-3, which is identical to gnuwin-1
(which is a zip of all binaries from GnuWin32 needed to
run llvm regression tests), except that rm.exe, mv.exe
and find.exe have been replaced with rm.exe, mv.exe,
and find.exe from msys (and three msys dlls needed
to run them have been added).  This new rm.exe is
able to delete paths longer than 260 characters, needed
for clang's Modules/dependency-dump.m test, and the new
mv.exe is needed for ExecutionEngine/OrcMCJIT/load-object-a.ll
which moves long paths around with `find.exe -exec mv.exe`

BUG= 604993 , 603364 

Review URL: https://codereview.chromium.org/1917853002

Cr-Commit-Position: refs/heads/master@{#389632}

[modify] https://crrev.com/8aa292dd634877fc41e5844af23d1dee28694f41/build/common.gypi
[modify] https://crrev.com/8aa292dd634877fc41e5844af23d1dee28694f41/build/config/compiler/BUILD.gn
[modify] https://crrev.com/8aa292dd634877fc41e5844af23d1dee28694f41/tools/clang/scripts/update.py