Note that you need to pass --nogoma to simple chrome ATM, since goma is broken.

I tested with clang++-6.0 on link, it also happens. I tested on samus, it also happens there.

#2, do you mean it crashes on clang-6.0 with similar back trace?

Stack dump:
0.      Program arguments: /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/bin/clang++-6.0.elf -cc1 -triple x86_64-cros-linux-gnu -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name cert_net_fetcher_impl.cc -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -relaxed-aliasing -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu corei7 -fdebug-info-for-profiling -dwarf-column-info -fno-split-dwarf-inlining -enable-split-dwarf -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -ggnu-pubnames -split-dwarf-file obj/net/net/cert_net_fetcher_impl.dwo -momit-leaf-frame-pointer -ffunction-sections -fdata-sections -coverage-notes-file /usr/local3/chrome/src/out_samus/obj/net/net/cert_net_fetcher_impl.gcno -resource-dir /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/lib64/clang/6.0.0 -dependency-file obj/net/net/cert_net_fetcher_impl.o.d -MT obj/net/net/cert_net_fetcher_impl.o -D _FORTIFY_SOURCE=2 -D V8_DEPRECATION_WARNINGS -D USE_UDEV -D USE_AURA=1 -D USE_NSS_CERTS=1 -D USE_OZONE=1 -D FULL_SAFE_BROWSING -D SAFE_BROWSING_CSD -D SAFE_BROWSING_DB_LOCAL -D CHROMIUM_BUILD -D FIELDTRIAL_TESTING_ENABLED -D _FILE_OFFSET_BITS=64 -D _LARGEFILE_SOURCE -D _LARGEFILE64_SOURCE -D CR_CLANG_REVISION="321529-2" -D __STDC_CONSTANT_MACROS -D __STDC_FORMAT_MACROS -D OS_CHROMEOS -D NDEBUG -D NVALGRIND -D DYNAMIC_ANNOTATIONS_ENABLED=0 -D DLOPEN_KERBEROS -D NET_IMPLEMENTATION -D USE_KERBEROS -D ENABLE_BUILT_IN_DNS -D GOOGLE_PROTOBUF_NO_RTTI -D GOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -D HAVE_PTHREAD -D U_USING_ICU_NAMESPACE=0 -D U_ENABLE_DYLOAD=0 -D U_STATIC_IMPLEMENTATION -D ICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -D UCHAR_TYPE=uint16_t -I .. -I gen -I ../third_party/protobuf/src -I gen/protoc_out -I ../third_party/protobuf/src -I ../third_party/boringssl/src/include -I /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/nss -I /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/nspr -I ../third_party/zlib -I ../third_party/ced/src -I ../third_party/icu/source/common -I ../third_party/icu/source/i18n -I ../third_party/brotli/include -D __DATE__= -D __TIME__= -D __TIMESTAMP__= -D __google_stl_debug_vector=1 -isysroot ../../.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz -internal-isystem /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/bin/../include/c++/v1 -internal-isystem ../../.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/local/include -internal-isystem /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/lib64/clang/6.0.0/include -internal-externc-isystem ../../.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/include -internal-externc-isystem ../../.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include -O2 -Wno-tautological-constant-compare -Wno-tautological-unsigned-enum-zero-compare -Wno-unknown-warning-option -Wno-section -Wno-unknown-warning-option -Wno-builtin-macro-redefined -Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-tautological-constant-compare -Wtautological-constant-out-of-range-compare -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wexit-time-destructors -Wno-header-guard -Wno-unknown-warning-option -Wno-inline-asm -std=gnu++14 -fdeprecated-macro -fdebug-compilation-dir /usr/local3/chrome/src/out_samus -ferror-limit 19 -fmessage-length 0 -fvisibility hidden -fvisibility-inlines-hidden -pthread -stack-protector 2 -fno-rtti -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -o obj/net/net/cert_net_fetcher_impl.o -x c++ ../net/cert_net/cert_net_fetcher_impl.cc -dwarf-debug-flags /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/bin/clang++-6.0 --driver-mode=g++ --sysroot=/usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz -Qunused-arguments -grecord-gcc-switches -Wno-tautological-constant-compare -Wno-tautological-unsigned-enum-zero-compare -Wno-unknown-warning-option -Wno-section -D _FORTIFY_SOURCE=2 -fstack-protector-strong -pie -fno-omit-frame-pointer -B /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.27.0-gold -Wno-unknown-warning-option -MMD -MF obj/net/net/cert_net_fetcher_impl.o.d -D V8_DEPRECATION_WARNINGS -D USE_UDEV -D USE_AURA=1 -D USE_NSS_CERTS=1 -D USE_OZONE=1 -D FULL_SAFE_BROWSING -D SAFE_BROWSING_CSD -D SAFE_BROWSING_DB_LOCAL -D CHROMIUM_BUILD -D FIELDTRIAL_TESTING_ENABLED -D _FILE_OFFSET_BITS=64 -D _LARGEFILE_SOURCE -D _LARGEFILE64_SOURCE -D CR_CLANG_REVISION="321529-2" -D __STDC_CONSTANT_MACROS -D __STDC_FORMAT_MACROS -D OS_CHROMEOS -D NDEBUG -D NVALGRIND -D DYNAMIC_ANNOTATIONS_ENABLED=0 -D DLOPEN_KERBEROS -D NET_IMPLEMENTATION -D USE_KERBEROS -D ENABLE_BUILT_IN_DNS -D GOOGLE_PROTOBUF_NO_RTTI -D GOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -D HAVE_PTHREAD -D U_USING_ICU_NAMESPACE=0 -D U_ENABLE_DYLOAD=0 -D U_STATIC_IMPLEMENTATION -D ICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -D UCHAR_TYPE=uint16_t -I .. -I gen -I ../third_party/protobuf/src -I gen/protoc_out -I ../third_party/protobuf/src -I ../third_party/boringssl/src/include -I /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/nss -I /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/nspr -I ../third_party/zlib -I ../third_party/ced/src -I ../third_party/icu/source/common -I ../third_party/icu/source/i18n -I ../third_party/brotli/include -fno-strict-aliasing -Wno-builtin-macro-redefined -D __DATE__= -D __TIME__= -D __TIMESTAMP__= -funwind-tables -fPIC -pipe -B ../third_party/binutils/Linux_x64/Release/bin -pthread -fcolor-diagnostics -no-canonical-prefixes -m64 -march=x86-64 -Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wno-unused-lambda-capture -Wno-user-defined-warnings -Wno-enum-compare-switch -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-tautological-constant-compare -Wtautological-constant-out-of-range-compare -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g2 -gsplit-dwarf -ggnu-pubnames -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wexit-time-destructors -Wno-header-guard -std=gnu++14 -fno-exceptions -fno-rtti --sysroot=../../.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz -fvisibility-inlines-hidden -pipe -pipe -pipe -march=corei7 -fno-split-dwarf-inlining -fdebug-info-for-profiling -D __google_stl_debug_vector=1 -Wno-unknown-warning-option -stdlib=libc++ -Wno-inline-asm -c ../net/cert_net/cert_net_fetcher_impl.cc -o obj/net/net/cert_net_fetcher_impl.o -B /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/bin --target=x86_64-cros-linux-gnu 
1.      ../net/cert_net/cert_net_fetcher_impl.cc:480:12: current parser token ')'
2.      ../net/cert_net/cert_net_fetcher_impl.cc:84:1: parsing namespace 'net'
3.      ../net/cert_net/cert_net_fetcher_impl.cc:86:1: parsing namespace 'net::(anonymous)'
4.      ../net/cert_net/cert_net_fetcher_impl.cc:445:55: parsing function body 'net::(anonymous namespace)::Job::StartURLRequest'
5.      ../net/cert_net/cert_net_fetcher_impl.cc:445:55: in compound statement ('{}')
6.      ../net/traffic_annotation/network_traffic_annotation.h:85:39: instantiating function definition 'net::DefineNetworkTrafficAnnotation<21, 1214>'
7.      ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<20>'
8.      ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<19>'
9.      ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<18>'
10.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<17>'
11.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<16>'
12.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<15>'
13.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<14>'
14.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<13>'
15.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<12>'
16.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<11>'
17.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<10>'
18.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<9>'
19.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<8>'
20.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<7>'
21.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<6>'
22.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<5>'
23.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<4>'
24.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<3>'
25.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<2>'
clang++-6.0: error: unable to execute command: Segmentation fault
clang++-6.0: error: clang frontend command failed due to signal (use -v to see invocation)
Chromium OS 6.0_pre317203_p20171212-r3 clang version 6.0.0 (/var/cache/chromeos-cache/distfiles/host/egit-src/clang.git 4d085086c74a8fbce197f61548f488a63f300933) (/var/cache/chromeos-cache/distfiles/host/egit-src/llvm.git 66af8bde13a515b8d0fa76201f0d5b428187fd13) (based on LLVM 6.0.0svn)
Target: x86_64-cros-linux-gnu
Thread model: posix
InstalledDir: /usr/local3/chrome/.cros_cache/chrome-sdk/tarballs/samus+10333.0.0+target_toolchain/usr/bin
clang++-6.0: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.
clang++-6.0: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang++-6.0: note: diagnostic msg: /tmp/cert_net_fetcher_impl-c61d7c.cpp
clang++-6.0: note: diagnostic msg: /tmp/cert_net_fetcher_impl-c61d7c.sh
clang++-6.0: note: diagnostic msg: 

********************
[17399/42168] CXX obj/net/net/quic_chromium_client_session.o
ninja: build stopped: subcommand failed.

I run the test case scope-info-1c832a.cpp inside chroot, and I could not get the crash. Dowloading chrome to try to run it with simple chrome workflow.

My simple chrome build for link works fine. Can you get the fresh chrome checkout and test it again?

Do you have any local changes that might break clang?

I don't have any local changes, the only "weird" thing I have to do is pass --nogoma.

I will make a new chrome checkout and test there.

Could you please show me where you passed '--nogoma'?
I tried it on cros chrome-sdk and autoninja command, and they both don't recognize this parameter.

I just tried, I also get the problem with a fresh chromium checkout.

Here is where I pass --nogoma:
cros chrome-sdk  --board=link --log-level=info --nogoma

Ah, you can make it reproducible with:

echo 0 > /proc/sys/kernel/randomize_va_space

I guess the address space randomization avoids the crash most of the time.

So does that mean you can reduce the occurrence of  crash by enabling ALSR? If that this the case, could you please lower the priority?

Yeah I can work. Although, it would certainly be good not to have a toolchain which crashes randomly... I have seen this crash before, but it was never reproducible, now I understand why.

agree. this is very annoying and could be happening in the buildbots.
Yunlian, have you been able to reproduce? 
Is there a possibility this only happen within marcheu@'s environment?

I could not reproduce it with ASLR after 5 runs.

How can I run echo 0 > /proc/sys/kernel/randomize_va_space on my desktop?
It seems I do not have permission to do that?

It's a root-only writable file; you have to use sudo, or be root etc.

sudo echo 0 > /proc/sys/kernel/randomize_va_space
bash: /proc/sys/kernel/randomize_va_space: Permission denied

sudo glinux-config list 
does not have this config knob.

How did you do that?

Yunlian, try this? 
echo 0 |sudo tee /proc/sys/kernel/randomize_va_space

Thanks Manoj, this works.

OKey, reproduced.
I will report back for additional findings.

There is a stackoverflow with our loader and ALSR disabled.
The stack size limit is 405504 in /proc/mappings and the stack grows over this limit.

Do you know the reason for stack overflow. Is it deep recursion in some function?

Is it normal that the stack size limit is reduced from 8M?

> Do you know the reason for stack overflow. Is it deep recursion in some function?

Looks like it; each of these is a template instantiation:

7.      ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<20>'
8.      ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<19>'
9.      ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<18>'
10.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<17>'
11.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<16>'
12.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<15>'
13.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<14>'
14.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<13>'
15.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<12>'
16.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<11>'
17.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<10>'
18.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<9>'
19.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<8>'
20.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<7>'
21.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<6>'
22.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<5>'
23.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<4>'
24.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<3>'
25.     ../net/traffic_annotation/network_traffic_annotation.h:14:20: instantiating function definition '(anonymous namespace)::recursive_hash<2>'

...And clang uses actual recursion to instantiate templates. There are limits to how deep clang will go, and clang goes out of its way to try to ensure that it has 8MB of stack available to it: https://github.com/llvm-mirror/clang/blob/master/tools/driver/cc1_main.cpp#L135-L162

Am I correct in my understanding that clang is being forced to live in ~400KB of stack here? Both inside and outside of my chroot, `ulimit -s` reports that 8MB of stack is available, and https://cs.chromium.org/search/?q=setrlimit+rlimit_stack+-file:llvm&sq=package:chromium&type=cs yields nothing interesting (modulo maybe service.py), so it's not clear to me where/how that's being set.