Does this repro in a local goma build?

Building locally with r315520 and the same config as the bot reproduces the issue:

C:\src\chromium\src>ninja -j200 -C out\release obj/mojo/public/cpp/test_support/test_utils.lib
ninja: Entering directory `out\release'
[1/1] Regenerating ninja files
[198/362] LIB obj/mojo/public/c/test_support/mojo_public_test_support.lib
FAILED: obj/mojo/public/c/test_support/mojo_public_test_support.lib
c:/src/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False lib.exe /nologo /ignore:4221 /OUT:obj/mojo/public/c/test_support/mojo_public_test_support.lib @obj/mojo/public/c/test_support/mojo_public_test_support.lib.rsp
LINK : fatal error LNK1181: cannot open input file 'obj/mojo/public/c/test_support/test_support/test_support_private.obj'
[199/362] LIB obj/base/base_static.lib
FAILED: obj/base/base_static.lib
c:/src/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False lib.exe /nologo /ignore:4221 /OUT:obj/base/base_static.lib @obj/base/base_static.lib.rsp
LINK : fatal error LNK1181: cannot open input file 'obj/base/base_static/base_switches.obj'
[200/362] LIB obj/mojo/public/c/system/mojo_public_system.lib
FAILED: obj/mojo/public/c/system/mojo_public_system.lib
c:/src/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False lib.exe /nologo /ignore:4221 /OUT:obj/mojo/public/c/system/mojo_public_system.lib @obj/mojo/public/c/system/mojo_public_system.lib.rsp
LINK : fatal error LNK1181: cannot open input file 'obj/mojo/public/c/system/system/thunks.obj'
[201/362] LIB obj/third_party/modp_b64/modp_b64.lib
FAILED: obj/third_party/modp_b64/modp_b64.lib
c:/src/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False lib.exe /nologo /ignore:4221 /OUT:obj/third_party/modp_b64/modp_b64.lib @obj/third_party/modp_b64/modp_b64.lib.rsp
LINK : fatal error LNK1181: cannot open input file 'obj/third_party/modp_b64/modp_b64/modp_b64.obj'
[205/362] LIB obj/testing/gtest/gtest.lib
FAILED: obj/testing/gtest/gtest.lib
c:/src/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False lib.exe /nologo /ignore:4221 /OUT:obj/testing/gtest/gtest.lib @obj/testing/gtest/gtest.lib.rsp
LINK : fatal error LNK1181: cannot open input file 'obj/testing/gtest/gtest/empty.obj'
[358/362] CXX obj/base/base/win_util.obj
ninja: build stopped: subcommand failed.



empty.obj does indeed not exist, but the .tmp one does:

dir out\release\obj\testing\gtest\gtest\empty*
 Volume in drive C has no label.
 Volume Serial Number is 54A3-FBD9

 Directory of C:\src\chromium\src\out\release\obj\testing\gtest\gtest

10/12/2017  09:30 AM               376 empty-8b033bb0.obj.tmp
               1 File(s)            376 bytes
               0 Dir(s)  466,390,990,848 bytes free

dumpbin shows it is is a valid coff object.





The file was produced by a goma task cache hit, and indeed the "exec_output_file" shows "obj\testing\gtest\gtest\empty-8b033bb0.obj.tmp".

How did it conclude that??

The "flag" field shows:

../../third_party/llvm-build/Release+Asserts/bin/clang-cl.exe /nologo /showIncludes @obj/testing/gtest/gtest/empty.obj.rsp /c ../../testing/gtest/empty.cc /Foobj/testing/gtest/gtest/empty.obj /Fdobj/testing/gtest/gtest_cc.pdb -> ../../third_party/llvm-build/Release+Asserts/bin/clang-cl.exe /nologo /showIncludes -imsvcc:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\bin\..\..\win_sdk\include\10.0.15063.0\um -imsvcc:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\bin\..\..\win_sdk\include\10.0.15063.0\shared -imsvcc:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\bin\..\..\win_sdk\include\10.0.15063.0\winrt -imsvcc:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\bin\..\..\win_sdk\include\10.0.15063.0\ucrt -imsvcc:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\bin\..\..\vc\tools\msvc\14.11.25503\include -imsvcc:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\a9e1098bba66d2acccc377d5ee81265910f29272\win_sdk\bin\..\..\vc\tools\msvc\14.11.25503\atlmfc\include -DV8_DEPRECATION_WARNINGS -DUSE_AURA=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -DCR_CLANG_REVISION="315520-1" -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=1 -DWIN32 -D_SECURE_ATL -D_USING_V110_SDK71_ -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=0x0A000000 -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DUNIT_TEST -DGTEST_API_= -DGTEST_HAS_POSIX_RE=0 -DGTEST_LANG_CXX11=1 -I../.. -Igen -I../../third_party/googletest/src/googletest/include -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=1 /Gy /FS /bigobj /d2FastFail /Zc:sizedDealloc- -fmsc-version=1911 -m64 /W4 /WX /utf-8 /wd4091 /wd4127 /wd4251 /wd4275 /wd4312 /wd4324 /wd4351 /wd4355 /wd4503 /wd4589 /wd4611 /wd4100 /wd4121 /wd4244 /wd4505 /wd4510 /wd4512 /wd4610 /wd4838 /wd4995 /wd4996 /wd4456 /wd4457 /wd4458 /wd4459 -Wno-unknown-pragmas -Wno-microsoft-cast -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-tautological-unsigned-enum-zero-compare /O1 /Ob2 /Oy- /d2Zi+ /Zc:inline /Gw /Oi /MT -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang check-auto-raw-pointer -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare /wd4800 /TP /wd4577 /GR- /c ../../testing/gtest/empty.cc /Foobj/testing/gtest/gtest/empty.obj /Fdobj/testing/gtest/gtest_cc.pdb

So /Fo has the right object file name.


Did the rename not work after all, and somehow the goma executor doesn't fail, but returns whatever file clang outputs? How does it detect that?

Here's a theory:

My work-around in r315520 relies on Wine's SetFileInformationByHandle returning with an ERROR_CALL_NOT_IMPLEMENTED error code.

Contemporary Wine versions do that, but it seems this wasn't always the case. Older versions (2015 or so) simply returned false without setting an error code :-(

If goma is using an older version, that would explain it.

Looking at pack_clang-cl_in_chromium.sh, it seeme goma uses Wine from Ubuntu Trusty, which is pretty ancient. So that fits with my theory.

Committed LLVM r315597 to handle older Wines.

Pushed r315597 to goma and verified this works.