Not able to repro on Debian Testing.  Maybe this only repros on Trusty?

$ touch native_client/src/trusted/service_runtime/linux/nacl_bootstrap.c
$ ninja -C out/ClangToTLinuxUBSanVptr/ native_client/src/trusted/service_runtime/linux:bootstrap
ninja: Entering directory `out/ClangToTLinuxUBSanVptr/'
[11/11] STAMP obj/native_client/src/trusted/service_runtime/linux/bootstrap.stamp

Bot uses these args:

clang_use_chrome_plugins = false
is_clang = true
is_component_build = false
is_debug = false
is_ubsan_no_recover = true
is_ubsan_vptr = true
llvm_force_head_revision = true
sanitizer_coverage_flags = "trace-pc-guard"


Maybe it also repros with prebuilt clang (llvm_force_head_revision=false) -- possibly caused by is_ubsan_ flags

This issue is also affecting CFI Linux ToT.

https://build.chromium.org/p/chromium.fyi/builders/CFI%20Linux%20ToT/builds/7195

I just ssh'ed into slave42-c1 (which has been running only ClangToTLinuxUBSanVptr for the past 208 days without restart) and removed the sysroots, ran 'gclient sync', and now the build works. Also I killed some long-running processes that had accumulated 7250hr of CPU time.

Build 6841 looks good so far: https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinuxUBSanVptr/builds/6841

I'll try the same with slave59-c1 to fix CFI Linux ToT.

The build got a lot farther, but now clang is crashing.  When I ran gclient sync, I saw that it also downloaded a new clang.

FAILED: obj/third_party/webrtc/webrtc_common/common_types.o 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/third_party/webrtc/webrtc_common/common_types.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -DUNDEFINED_SANITIZER -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -DCR_CLANG_REVISION=\"314820\" -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DSANITIZER_COVERAGE -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DWEBRTC_ENABLE_PROTOBUF=1 -DWEBRTC_RESTRICT_LOGGING -DHAVE_SCTP -DENABLE_EXTERNAL_AUTH -DHAVE_WEBRTC_VIDEO -DHAVE_WEBRTC_VOICE -DLOGGING_INSIDE_WEBRTC -DUSE_WEBRTC_DEV_BRANCH -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DFEATURE_ENABLE_VOICEMAIL -DGTEST_RELATIVE_PATH -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_POSIX -DWEBRTC_LINUX -I../.. -Igen -I../../third_party/webrtc_overrides -I../../third_party/webrtc -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -pthread -fcolor-diagnostics -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=1 -m64 -march=x86-64 -Wall -Werror -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-tautological-unsigned-enum-zero-compare -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g1 -gline-tables-only -gcolumn-info -fno-omit-frame-pointer -fsanitize-coverage=trace-pc-guard -mllvm -sanitizer-coverage-prune-blocks=1 -fno-sanitize-recover=undefined -fsanitize=vptr -fsanitize-blacklist=../../tools/ubsan/vptr_blacklist.txt -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -std=gnu++14 -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include --sysroot=../../build/linux/debian_jessie_amd64-sysroot -fno-exceptions -fvisibility-inlines-hidden -c ../../third_party/webrtc/common_types.cc -o obj/third_party/webrtc/webrtc_common/common_types.o
clang-6.0: /b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:758: llvm::slpvectorizer::BoUpSLP::TreeEntry* llvm::slpvectorizer::BoUpSLP::newTreeEntry(llvm::ArrayRef<llvm::Value*>, bool, int&, const {anonymous}::InstructionsState&, llvm::ArrayRef<unsigned int>, int): Assertion `UserEntry->ShuffleMask[OpdNum].empty() && "Mask already present!"' failed.
#0 0x000000000211158a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x211158a)
#1 0x000000000210f66e llvm::sys::RunSignalHandlers() (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x210f66e)
#2 0x000000000210f7d2 (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x210f7d2)
#3 0x00007f52d30a7330 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x10330)
#4 0x00007f52d1c96c37 gsignal /build/eglibc-SvCtMH/eglibc-2.19/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:56:0
#5 0x00007f52d1c9a028 abort /build/eglibc-SvCtMH/eglibc-2.19/stdlib/abort.c:91:0
#6 0x00007f52d1c8fbf6 __assert_fail_base /build/eglibc-SvCtMH/eglibc-2.19/assert/assert.c:92:0
#7 0x00007f52d1c8fca2 (/lib/x86_64-linux-gnu/libc.so.6+0x2fca2)
#8 0x000000000223e2cd (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x223e2cd)
#9 0x00000000022422d7 llvm::slpvectorizer::BoUpSLP::buildTree_rec(llvm::ArrayRef<llvm::Value*>, unsigned int, int, int) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x22422d7)
#10 0x0000000002242402 llvm::slpvectorizer::BoUpSLP::buildTree(llvm::ArrayRef<llvm::Value*>, llvm::MapVector<llvm::Value*, llvm::SmallVector<llvm::Instruction*, 2u>, llvm::DenseMap<llvm::Value*, unsigned int, llvm::DenseMapInfo<llvm::Value*>, llvm::detail::DenseMapPair<llvm::Value*, unsigned int> >, std::vector<std::pair<llvm::Value*, llvm::SmallVector<llvm::Instruction*, 2u> >, std::allocator<std::pair<llvm::Value*, llvm::SmallVector<llvm::Instruction*, 2u> > > > >&, llvm::ArrayRef<llvm::Value*>) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x2242402)
#11 0x0000000002247b10 (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x2247b10)
#12 0x00000000022497a8 llvm::SLPVectorizerPass::vectorizeRootInstruction(llvm::PHINode*, llvm::Value*, llvm::BasicBlock*, llvm::slpvectorizer::BoUpSLP&, llvm::TargetTransformInfo*) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x22497a8)
#13 0x000000000224a24f llvm::SLPVectorizerPass::vectorizeChainsInBlock(llvm::BasicBlock*, llvm::slpvectorizer::BoUpSLP&) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x224a24f)
#14 0x000000000224aa86 (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x224aa86)
#15 0x000000000224bd94 (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x224bd94)
#16 0x0000000001cbe4f3 llvm::FPPassManager::runOnFunction(llvm::Function&) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x1cbe4f3)
#17 0x0000000001cbe59c llvm::FPPassManager::runOnModule(llvm::Module&) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x1cbe59c)
#18 0x0000000001cbee1f llvm::legacy::PassManagerImpl::run(llvm::Module&) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x1cbee1f)
#19 0x000000000229b84b (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x229b84b)
#20 0x000000000229cb22 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x229cb22)
#21 0x00000000029830db (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x29830db)
#22 0x0000000002d74a52 clang::ParseAST(clang::Sema&, bool, bool) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x2d74a52)
#23 0x000000000298271f clang::CodeGenAction::ExecuteAction() (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x298271f)
#24 0x00000000026368c6 clang::FrontendAction::Execute() (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x26368c6)
#25 0x000000000260dffe clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x260dffe)
#26 0x00000000026c728b clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0x26c728b)
#27 0x0000000000bc7648 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0xbc7648)
#28 0x0000000000b5db70 main (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0xb5db70)
#29 0x00007f52d1c81f45 __libc_start_main /build/eglibc-SvCtMH/eglibc-2.19/csu/libc-start.c:321:0
#30 0x0000000000bc2d49 _start (/b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0+0xbc2d49)
Stack dump:
0.	Program arguments: /b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/bin/clang-6.0 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -disable-free -main-file-name common_types.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -relaxed-aliasing -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debug-info-kind=line-tables-only -dwarf-version=4 -debugger-tuning=gdb -momit-leaf-frame-pointer -ffunction-sections -fdata-sections -coverage-notes-file /b/c/builder/ClangToTLinuxUBSanVptr/src/out/Release/obj/third_party/webrtc/webrtc_common/common_types.gcno -nostdinc++ -resource-dir /b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/lib/clang/6.0.0 -dependency-file obj/third_party/webrtc/webrtc_common/common_types.o.d -MT obj/third_party/webrtc/webrtc_common/common_types.o -isystem ../../buildtools/third_party/libc++/trunk/include -isystem ../../buildtools/third_party/libc++abi/trunk/include -D V8_DEPRECATION_WARNINGS -D USE_UDEV -D USE_AURA=1 -D USE_GLIB=1 -D USE_NSS_CERTS=1 -D USE_X11=1 -D UNDEFINED_SANITIZER -D FULL_SAFE_BROWSING -D SAFE_BROWSING_CSD -D SAFE_BROWSING_DB_LOCAL -D CHROMIUM_BUILD -D FIELDTRIAL_TESTING_ENABLED -D CR_CLANG_REVISION="314820" -D _FILE_OFFSET_BITS=64 -D _LARGEFILE_SOURCE -D _LARGEFILE64_SOURCE -D __STDC_CONSTANT_MACROS -D __STDC_FORMAT_MACROS -D SANITIZER_COVERAGE -D NDEBUG -D NVALGRIND -D DYNAMIC_ANNOTATIONS_ENABLED=0 -D WEBRTC_ENABLE_PROTOBUF=1 -D WEBRTC_RESTRICT_LOGGING -D HAVE_SCTP -D ENABLE_EXTERNAL_AUTH -D HAVE_WEBRTC_VIDEO -D HAVE_WEBRTC_VOICE -D LOGGING_INSIDE_WEBRTC -D USE_WEBRTC_DEV_BRANCH -D WEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -D FEATURE_ENABLE_VOICEMAIL -D GTEST_RELATIVE_PATH -D WEBRTC_CHROMIUM_BUILD -D WEBRTC_POSIX -D WEBRTC_LINUX -I ../.. -I gen -I ../../third_party/webrtc_overrides -I ../../third_party/webrtc -D __DATE__= -D __TIME__= -D __TIMESTAMP__= -isysroot ../../build/linux/debian_jessie_amd64-sysroot -internal-isystem ../../build/linux/debian_jessie_amd64-sysroot/usr/local/include -internal-isystem /b/c/builder/ClangToTLinuxUBSanVptr/src/third_party/llvm-build/Release+Asserts/lib/clang/6.0.0/include -internal-externc-isystem ../../build/linux/debian_jessie_amd64-sysroot/usr/include/x86_64-linux-gnu -internal-externc-isystem ../../build/linux/debian_jessie_amd64-sysroot/include -internal-externc-isystem ../../build/linux/debian_jessie_amd64-sysroot/usr/include -O2 -Wno-builtin-macro-redefined -Wall -Werror -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-tautological-unsigned-enum-zero-compare -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -std=gnu++14 -fdeprecated-macro -fdebug-compilation-dir /b/c/builder/ClangToTLinuxUBSanVptr/src/out/Release -ferror-limit 19 -fmessage-length 0 -fvisibility hidden -fvisibility-inlines-hidden -fsanitize-coverage-type=3 -fsanitize-coverage-trace-pc-guard -fsanitize=vptr -fsanitize-blacklist=../../tools/ubsan/vptr_blacklist.txt -fdepfile-entry=../../tools/ubsan/vptr_blacklist.txt -pthread -stack-protector 1 -stack-protector-buffer-size 4 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -mllvm -instcombine-lower-dbg-declare=1 -mllvm -sanitizer-coverage-prune-blocks=1 -o obj/third_party/webrtc/webrtc_common/common_types.o -x c++ ../../third_party/webrtc/common_types.cc 
1.	<eof> parser at end of file
2.	Per-module optimization passes
3.	Running pass 'Function Pass Manager' on module '../../third_party/webrtc/common_types.cc'.
4.	Running pass 'SLP Vectorizer' on function '@_ZNK6webrtc17BitrateAllocation8ToStringEv'
clang-6.0: error: unable to execute command: Aborted (core dumped)
clang-6.0: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 6.0.0 (trunk 314820)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /b/c/builder/ClangToTLinuxUBSanVptr/src/out/Release/../../third_party/llvm-build/Release+Asserts/bin
clang-6.0: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.
clang-6.0: note: diagnostic msg: 
********************
PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-6.0: note: diagnostic msg: /b/rr/tmpu_PHGt/t/common_types-639972.cpp
clang-6.0: note: diagnostic msg: /b/rr/tmpu_PHGt/t/common_types-639972.sh
clang-6.0: note: diagnostic msg: 
********************

https://www.mail-archive.com/llvm-bugs@lists.llvm.org/msg17146.html

Yeah, someone re-landed their patch (r314806) and I reverted in r314824.

There's been a lot of Clang breakages upstream this morning, hopefully it's better now.

ClangToTLinuxUBSanVptr and CFI Linux ToT are green now

Now seeing this on https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FClangToTLinux%2F9211%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

Does someone have to ssh in and clobber the sysroot there too?

What happened to require this step?

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FClangToTLinux__dbg_%2F7590%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout too

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FClangToTLinuxMSan%2F70%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout too

> What happened to require this step?

Not sure.. we didn't change anything.  I'll try to find out what clobbering the sysroot and gclient syncing actually change to make the build work again.

Update: it's not the sysroots.  It gets fixed simply by running gclient sync (which I thought the bots were supposed to do on each run?)

The only thing that happens during gclient sync is:
Updating Clang to 313786-1...
Downloading prebuilt clang
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-313786-1.tgz .......... Done.
Creating directory /b/c/builder/ClangToTLinux/src/third_party/llvm-build/Release+Asserts
clang 313786-1 unpacked

The gn args have llvm_force_head_revision=true, which I thought causes the build to use the clang built during gclient runhooks.  But the build still uses the one in third_party/llvm-build/Release+Asserts

ooooh that's not a real fix then, that makes the bot useless.

When the bot runs `gclient sync`, it has LLVM_FORCE_HEAD_REVISION set in the env. When you do this via ssh, you probably don't have that. So then you convert the bot from a ToT bot to a pinned bot, and it no longer tests what it's supposed to test.

But it should revert to being a ToT bot on the next build...so it's strange if that doesn't happen.

(This also means there's some upstream regression I suppose)

https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinuxUBSanVptr definitely reverted back to doing tot builds (good). It's currently red due to  issue 771670  (which is now fixed). If it cycles green, I'm very confused. If it stays red and switches to the error in comment 0 again, then at least I understand why ssh'ing in seemed to help.

Do those bots clobber on each build?  Since environment variables don't affect ninja's ability to detect if targets need to be rebuilt, do you think it's just reusing the files from native_client/src/trusted/service_runtime/linux:bootstrap from when I built it?

I can repro with a locally-built clang and these args.gn:

clang_base_path = getenv("HOME") + "/src/llvm-build-nolibcxx"
is_clang = true
is_component_build = false
is_debug = false
is_ubsan_no_recover = true
is_ubsan_vptr = true 
sanitizer_coverage_flags = "trace-pc-guard"
~                                                 



My local clang is at 314886

$ ninja -C out/gn  nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o 
ninja: Entering directory `out/gn'
[1/1] CC nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o
FAILED: nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o 
../../../../llvm-build-nolibcxx/bin/clang -MMD -MF nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o.d -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_POSIX_C_SOURCE=199506 -D_XOPEN_SOURCE=600 -D_GNU_SOURCE=1 -D__STDC_LIMIT_MACROS=1 -I../.. -fno-builtin -fno-stack-protector -fno-pic -fno-PIC -fno-pie -fno-PIE -m64 -march=x86-64 -Wall -Werror -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -O2 -fno-ident -fdata-sections -ffunction-sections -g0 --sysroot=../../build/linux/debian_jessie_amd64-sysroot -c ../../native_client/src/trusted/service_runtime/linux/nacl_bootstrap.c -o nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o
../../native_client/src/trusted/service_runtime/linux/nacl_bootstrap.c:434:16: error: section attribute is specified on redeclared variable [-Werror,-Wsection]
struct r_debug _r_debug __attribute__((nocommon, section(".r_debug")));
               ^
../../build/linux/debian_jessie_amd64-sysroot/usr/include/link.h:67:23: note: previous declaration is here
extern struct r_debug _r_debug;
                      ^
1 error generated.

They kind-of-clobber. They add a -D flag containing the revision of the currently-locally-built clang to the compile flags and then rely on ninja's command line tracking to make sure that all source files get rebuilt when the clang revision changes

I don't see the -D flag for the file that the build is failing on:

../../third_party/llvm-build/Release+Asserts/bin/clang -MMD -MF nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o.d -DCOMPONENT_BUILD -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_POSIX_C_SOURCE=199506 -D_XOPEN_SOURCE=600 -D_GNU_SOURCE=1 -D__STDC_LIMIT_MACROS=1 -I../.. -fno-builtin -fno-stack-protector -fno-pic -fno-PIC -fno-pie -fno-PIE -m64 -march=x86-64 -Wall -Werror -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-tautological-unsigned-enum-zero-compare -O2 -fno-ident -fdata-sections -ffunction-sections -g2 -ggnu-pubnames --sysroot=../../build/linux/debian_jessie_amd64-sysroot -c ../../native_client/src/trusted/service_runtime/linux/nacl_bootstrap.c -o nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o

Hm.. it appears nacl *purposely* resets the configs to [], which would explain why the -D isn't getting set

https://cs.chromium.org/chromium/src/native_client/src/trusted/service_runtime/linux/BUILD.gn?rcl=7f07816e463da403255f0ab4f6d88004450dd09d&l=57

It's added here, with a !is_nacl : https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?q=cr_clang_re&sq=package:chromium&l=262 IIRC that was because nacl code used to be built with its own compiler or something (?) So that !is_nacl is probably wrong...

https://codereview.chromium.org/972203003/diff/180001/build/config/compiler/BUILD.gn added the !is_nacl; not clear whybb

Maybe this was actually caused by the sysroot change.  It's possible that nacl_bootstrap.o didn't get rebuilt after the sysroot switch, and then some change in //build/config or nacl caused the command line for the target to change, which is why the build started failing a bit after the sysroot switch

It only repros with trunk clang for me, so it's not just the sysroot change at least. My local clang regression range atm is 314000:314300

r314262 ?

http://llvm.org/viewvc/llvm-project?view=revision&revision=314262 -- that looks very likely. Yes, confirmed.

Code is here: https://cs.chromium.org/chromium/src/native_client/src/trusted/service_runtime/linux/nacl_bootstrap.c?q=nacl_bootstrap.c&sq=package:chromium&dr&l=424


Since this works in gcc and used to work in clang, I suppose this is a real regression and not a bug on our side?

Started a discussion upstream: http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20171002/205616.html

Nice, looks like they fixed it https://llvm.org/viewvc/llvm-project?view=revision&revision=314939

Should we just remove the !is_nacl from c#22 so that the bots will rebuild the broken target?  I don't think it could possibly hurt, since nothing uses CR_CLANG_REVISION.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1ee5c0dcdb16b5365de11dd00617dbe99b98f9b6

commit 1ee5c0dcdb16b5365de11dd00617dbe99b98f9b6
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Thu Oct 05 05:28:50 2017

Set CR_CLANG_REVISION on NaCl builds too

BUG= 771191 
R=thakis@chromium.org

Change-Id: I7f29c4d0951ee4ec253261a39c2f87640156ef59
Reviewed-on: https://chromium-review.googlesource.com/701643
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506659}
[modify] https://crrev.com/1ee5c0dcdb16b5365de11dd00617dbe99b98f9b6/build/config/compiler/BUILD.gn

I worked with upstream to fix the compiler, and thomasanderson made it so that -DCR_CLANG_REVISION is passed to nacl targets too.