******************* TEST 'lit :: shtest-run-at-line.py' FAILED ********************
Script:
--
: 'RUN: at line 4';   not C:\b\depot_tools\win_tools-2_7_6_bin\python\bin\python.exe C:/b/rr/tmpk6ugaf/w/src/third_party/llvm\utils\lit\lit.py -j 1 -vv C:/b/rr/tmpk6ugaf/w/src/third_party/llvm-bootstrap/utils/lit/tests\..\tests\Inputs/shtest-run-at-line > C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out
: 'RUN: at line 5';   FileCheck --input-file C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out -Dpdbg0="echo '" -Dpdbg1="' > nul &&" C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\shtest-run-at-line.py
--
Exit Code: 1
Command Output (stdout):
--
$ ":" "RUN: at line 4"
$ "not" "C:\b\depot_tools\win_tools-2_7_6_bin\python\bin\python.exe" "C:/b/rr/tmpk6ugaf/w/src/third_party/llvm\utils\lit\lit.py" "-j" "1" "-vv" "C:/b/rr/tmpk6ugaf/w/src/third_party/llvm-bootstrap/utils/lit/tests\..\tests\Inputs/shtest-run-at-line"
$ ":" "RUN: at line 5"
$ "FileCheck" "--input-file" "C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out" "-Dpdbg0=echo '" "-Dpdbg1=' > nul &&" "C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\shtest-run-at-line.py"
# command stderr:
C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\shtest-run-at-line.py:19:10: error: expected string not found in input
# CHECK: [[pdbg0]]RUN: at line 4[[pdbg1]] true
         ^
C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out:5:1: note: scanning from here
--
^
C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out:5:1: note: with variable "pdbg0" equal to "echo '"
--
^
C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out:5:1: note: with variable "pdbg1" equal to "' > nul &&"
--
^
C:\b\rr\tmpk6ugaf\w\src\third_party\llvm-bootstrap\utils\lit\tests\Output\shtest-run-at-line.py.tmp.out:6:4: note: possible intended match here
: 'RUN: at line 4'; true

I think this is related to the bot running with a bash.exe availalbe in PATH, from C:\b\cipd_path_tools\bin

For some reason the outer test thinks it's running under cmd.exe but the inner lit invocation uses the bash instead (maybe)

Yes, the real conundrum is why the outer test doesn't see bash.exe but the inner does...

This is due to r333614 (+follow-ups). But since it only seems to fail in our environment, I don't want to revert yet.

Wohoo! Got a local repro. I copied cipd_path_tools from the bot, and in a vanilla cmd.exe:

set PATH=%PATH%;C:\src\tmp\cipd_path_tools\bin
python tools\clang\scripts\package.py

Reproduces the problem.

Hmm, but reproducing without package.py is still not obvious.

In a vanilla cmd.exe, in the llvm-bootstrap dir:

set PATH=%PATH%;C:\src\tmp\cipd_path_tools\bin

This reproduces the problem:

\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\bin\SetEnv.cmd /x64 && ninja check-all

But this doesn't: *shrug*

\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\bin\SetEnv.cmd /x64 && ninja check-lit

And this unfortunately doesn't either, even when run just after check-all above:

\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\bin\SetEnv.cmd /x64 && python bin\llvm-lit.py ..\llvm\utils\lit\tests\shtest-run-at-line.py

Aha! Starting to smell a rat...
This reproduces:

\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\bin\SetEnv.cmd /x64 && python bin\llvm-lit.py -vv ..\llvm\test\CodeGen\X86\tailcall.ll ..\llvm\utils\lit\tests\shtest-run-at-line.py


Or simply:

python bin\llvm-lit.py -vv ..\llvm\test\CodeGen\X86\tailcall.ll ..\llvm\utils\lit\tests\shtest-run-at-line.py

And this works in a vanilla LLVM build too, doesn't need to have been created by package.py.

My theory is that utils/lit/lit/llvm/config.py messes with the PATH. So if that runs *before* shtest-run-at-line.py, the "outer" lit will not see bash.exe on the PATH, or something to that effect.

So what exactly does utils/lit/lit/llvm/config.py do?

Oh, sneaky sneaky!

In order to add the tools path to PATH, it does:

            # Seek sane tools in directories and set to $PATH.
            path = self.lit_config.getToolsPath(config.lit_tools_dir,
                                                config.environment['PATH'],
                                                ['cmp.exe', 'grep.exe', 'sed.exe'])
            if path is not None:
                self.with_environment('PATH', path, append_path=True



But getToolsPath() has a side effect:


    def getToolsPath(self, dir, paths, tools):
        if dir is not None and os.path.isabs(dir) and os.path.isdir(dir):
            if not lit.util.checkToolsPath(dir, tools):
                return None
        else:
            dir = lit.util.whichTools(tools, paths)

        # bash
        self.bashPath = lit.util.which('bash', dir)
        if self.bashPath is None:
            self.bashPath = ''


It sets bashPath to the empty string!


So if getToolsPath() has run before getBashPath(), lit won't find bash.exe. Otherwise it will find it in PATH. Yay.


And that is what's happening:

- when running check-all, the LLVM lit config gets loaded
- that sets bashPath to the empty string
- utils/lit/tests/lit.cfg sets isWin32CMDEXE to true
- shtest-run-at-line.py starts a new lit invocation
- that doesn't load the LLVM config, so bashPath returns the path to bash.exe on PATH
- and so the internal test runs with that instead of cmd.exe, and the output doesn't match the FileCheck directives configured by the outer lit

r334100 relaxes the test.