New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 687570 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Feb 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 685244

Blocking:
issue 651540



Sign in to add a comment

-O1 trace-pc-guard libFuzzer and AFL builds link uses 60GB of ram to link

Project Member Reported by aarya@google.com, Feb 1 2017

Issue description

https://build.chromium.org/p/chromium.fyi/builders/Libfuzzer%20Upload%20Linux%20ASan/builds/3178/steps/compile/logs/stdio
https://build.chromium.org/p/chromium.fyi/builders/Afl%20Upload%20Linux%20ASan/builds/1722/steps/compile/logs/stdio

[19982/20043] LINK ./webcrypto_ec_import_key_spki_fuzzer
FAILED: webcrypto_ec_import_key_spki_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_spki_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_spki_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[19983/20043] LINK ./webcrypto_ec_import_key_raw_fuzzer
FAILED: webcrypto_ec_import_key_raw_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_raw_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_raw_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
src/central_freelist.cc:333] tcmalloc: allocation failed 16384 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
clang: error: unable to execute command: Aborted (core dumped)
clang: error: linker command failed due to signal (use -v to see invocation)
[19984/20043] LINK ./webcrypto_ec_import_key_pkcs8_fuzzer
FAILED: webcrypto_ec_import_key_pkcs8_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_pkcs8_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_pkcs8_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
src/central_freelist.cc:333] tcmalloc: allocation failed 16384 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
clang: error: unable to execute command: Aborted (core dumped)
clang: error: linker command failed due to signal (use -v to see invocation)
ninja: build stopped: subcommand failed.
step returned non-zero exit code: 1
@@@STEP_FAILURE@@@

Are we running out of memory here ? Kostya, Nico, Hans - any idea on fixing this ?
 

Comment 1 by aarya@google.com, Feb 1 2017

Components: Infra

Comment 2 by kcc@chromium.org, Feb 1 2017

Cc: krasin@chromium.org
looks like parallel link steps are OOMing. 
+ krasin@, in case he has some ideas 
Components: -Infra
I don't think this should be owned by infra since OOM errors are likely caused by errors in the code or in GN. If you still think it should be owned by infra, please re-add a label with your rationale.
Blocking: 651540
Looks like this was caused by turning on trace-pc-guard

Attempting to build webcrypto_ec_import_key_spki_fuzzer locally with trace-pc-guard seems to lock up my machine. I'll revert https://codereview.chromium.org/2665243002/ for now.

Comment 6 by kcc@chromium.org, Feb 1 2017

Cc: kcc@chromium.org
Owner: aizatsky@chromium.org

Comment 7 by kcc@chromium.org, Feb 1 2017

This is pretty strange, given that trace-pc-guard creates less code than the
old coverage stuff... 
I tried to set "ulimit -v 52428800" to avoid machine lock up, but the build tries to run ASANified binary during the build (character_data_generator). Will spin up new GCE instance to experiment with this.

16 CPU/60G GCE instance has no problem building the fuzzer:

aizatsky@aizatsky-sandbox:~/src/chromium/src$ ./out/libfuzzer/webcrypto_ec_import_key_spki_fuzzer 
INFO: Seed: 1889592439
INFO: Loaded 2 modules (1524129 guards): [0x7f4095f9a1a0, 0x7f4095fb2ae0), [0xbfeb560, 0xc5a32a4), 
INFO: -max_len is not provided, using 64
INFO: A corpus is not provided, starting from an empty corpus
#0      READ units: 1
#1      INITED cov: 639 ft: 73 corp: 1/1b exec/s: 0 rss: 166Mb
#2      NEW    cov: 644 ft: 74 corp: 2/24b exec/s: 0 rss: 166Mb L: 23 MS: 1 InsertRepeatedBytes-

What's our typical bot HW config?
not sure about typical bot HW, but I was using goma with -j1500. 
It seems that gold consumes ~14G of RAM per invocation binary. I'm trying to figure out how to profile it. Any ideas?

Comment 12 by kcc@chromium.org, Feb 2 2017

was this caused by trace-pc-guard? Or just a coincidence? 
I just tried this again on my 64GB machine with the following GN args:

pdf_enable_xfa = true
is_asan = true
use_goma = true
use_libfuzzer = true
enable_nacl = false
is_debug = false
optimize_for_fuzzing = true

And managed to lock my machine again (top was showing ld.gold steadily chew through all my ram before it happened).

It doesn't seem to happen with trace-pc-guard.
It seems that the only difference between my & Oliver's configuration is "optimize_for_fuzzing = true" (I don't have it). Which changes '-O2' to '-O1' I believe.

Comment 15 Deleted

argh, s/with trace-pc-guard/without trace-pc-guard/
Confirming that optimize_for_fuzzing makes all the difference. Linker now fails with ulimit of 50G but succeeds with 60G. I will try to establish a baseline and measure linker memory consumption for the same binary without trace-pc-guard.

Comment 18 by aarya@google.com, Feb 2 2017

Summary: libFuzzer and AFL builds link failure (was: libFuzzer and AFL builds compile breakage)
AFL also now uses trace-pc-guard - https://codereview.chromium.org/2662883002/ and fails to link. If we don't have a workaround, i will revert it that one Friday as well.
Without trace-pc-guard (but with optimize_for_fuzzing) linker succeeds under ulimit of 7G but fails with 5G. So we are looking at ~10x memory usage increase.
@kcc the binary with trace-pc-guard is approximately 1/6th bigger (+50M). It doesn't look like it is less code. I'll have to investigate more to see where this comes from.
re c#9:

https://build.chromium.org/p/chromium.fyi/buildslaves/slave43-c1


architecture: amd64
bitness (userland): 64-bit
memory total: 29.45 GB
os family: Debian
os version: 14.04
processor count: 8
processor type: Intel(R) Xeon(R) CPU @ 2.30GHz
product name: Google Compute Engine


Looks like bots are n1-standard-8 instances (https://cloud.google.com/compute/pricing)
Summary: -O1 trace-pc-guard libFuzzer and AFL builds link uses 60GB of ram to link (was: libFuzzer and AFL builds link failure)
We just got upgraded to 104GB ram builders. Thanks for requesting the upgrade Max :)

I'll give another shot at relanding the trace-pc-guard change, but keeping this bug open since it seems like a pretty bad regression regardless.

Comment 23 by kcc@chromium.org, Feb 2 2017

perhaps the situation will get better once we move LLVM beyond 
http://llvm.org/viewvc/llvm-project?view=revision&revision=292862
Still fails with 104 gb ram
https://build.chromium.org/p/chromium.fyi/builders/Libfuzzer%20Upload%20Linux%20ASan/builds/3184/steps/compile/logs/stdio

[23239/23281] LINK ./webcrypto_rsa_import_key_pkcs8_fuzzer
FAILED: webcrypto_rsa_import_key_pkcs8_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_rsa_import_key_pkcs8_fuzzer" -Wl,--start-group @"./webcrypto_rsa_import_key_pkcs8_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23240/23281] LINK ./webcrypto_ec_import_key_raw_fuzzer
FAILED: webcrypto_ec_import_key_raw_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_raw_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_raw_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23241/23281] LINK ./webcrypto_ec_import_key_pkcs8_fuzzer
FAILED: webcrypto_ec_import_key_pkcs8_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_pkcs8_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_pkcs8_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23242/23281] LINK ./pdf_codec_jbig2_fuzzer
[23243/23281] LINK ./pdf_codec_fax_fuzzer
[23244/23281] LINK ./pdf_codec_jpeg_fuzzer
[23245/23281] LINK ./webcrypto_ec_import_key_spki_fuzzer
FAILED: webcrypto_ec_import_key_spki_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_spki_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_spki_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23246/23281] LINK ./webcrypto_rsa_import_key_spki_fuzzer
FAILED: webcrypto_rsa_import_key_spki_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_rsa_import_key_spki_fuzzer" -Wl,--start-group @"./webcrypto_rsa_import_key_spki_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23247/23281] LINK ./renderer_tree_fuzzer
FAILED: renderer_tree_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./renderer_tree_fuzzer" -Wl,--start-group @"./renderer_tree_fuzzer.rsp" ./libc++.so ./libfreetype.so.6 -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 -latk-1.0 -lcups -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lgcrypt -lpthread -lcrypt 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23248/23281] LINK ./renderer_fuzzer
FAILED: renderer_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./renderer_fuzzer" -Wl,--start-group @"./renderer_fuzzer.rsp" ./libc++.so ./libfreetype.so.6 -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 -latk-1.0 -lcups -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lgcrypt -lpthread -lcrypt 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23249/23281] LINK ./clear_site_data_fuzzer
FAILED: clear_site_data_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./clear_site_data_fuzzer" -Wl,--start-group @"./clear_site_data_fuzzer.rsp" ./libc++.so ./libfreetype.so.6 -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 -latk-1.0 -lcups -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lgcrypt -lpthread -lcrypt 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[23250/23281] LINK ./origin_trial_token_fuzzer
FAILED: origin_trial_token_fuzzer 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/b/build/slave/Libfuzzer_Upload_Linux_ASan/build/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./origin_trial_token_fuzzer" -Wl,--start-group @"./origin_trial_token_fuzzer.rsp" ./libc++.so ./libfreetype.so.6 -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 -latk-1.0 -lcups -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lgcrypt -lpthread -lcrypt 
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
ninja: build stopped: subcommand failed.
step returned non-zero exit code: 1
@@@STEP_FAILURE@@@
ToT Clang fails to link the binary at all:

FAILED: ../../../../../out/llvm/bootstrap/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -pthread -Werror -Wl,-O1 -Wl,--gc-sections --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -L/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/lib/x86_64-linux-gnu -L/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/x86_64-linux-gnu -L/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath-link=/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -Wl,-rpath-link=/usr/local/google/home/aizatsky/src/chrome/src/build/linux/debian_wheezy_amd64-sysroot/usr/lib -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -stdlib=libc++ -Wl,-rpath,\$ORIGIN/ -L. -o "./webcrypto_ec_import_key_spki_fuzzer" -Wl,--start-group @"./webcrypto_ec_import_key_spki_fuzzer.rsp" ./libc++.so -Wl,--end-group  -ldl -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lpangocairo-1.0 -lpango-1.0 -lcairo -lfontconfig -lfreetype -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lXss -lgconf-2 -lresolv -lasound -lm -lz -ldbus-1 
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_strcmp'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_strncmp'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_strcasecmp'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_strncasecmp'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_strstr'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_strcasestr'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_memmem'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: warning: Cannot export local symbol '__sanitizer_weak_hook_memcmp'
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: error: treating warnings as errors
Reverting once more... https://codereview.chromium.org/2675983002/
I rolled libfuzzer forward and tried to build with tot llvm. The linker still consumes ~50G of memory. Investigating object size increase since this might be relevant.
Actually with tot llvm I see small size decrease as Kostya suggested. 324M (currently) vs 320M (trace-pc-guard).

Comment 29 by aarya@google.com, Feb 6 2017

If we dont get a resolution to this in next 2-3 days, i will revert AFL update cl as well. Right now, it is also on compile failure.
I was able to localize the issue in gold code.
The culprit is "-gc-sections" flag. Without it linker process is fast and painless.

According to profiler, with gc-sections enabled, most time is spent in gc_process_relocs (in gc.h).

In particular, the hashtable insert in the very last loop in the function is where CPU is spent:

             for (Garbage_collection::Sections_reachable::iterator it_v
                     = cident_secn.begin();
                   it_v != cident_secn.end();
                   ++it_v)
                {
                  v.insert(*it_v);
                }

Looks like gold has a quadratic behavior in our particular case. The plan is:

- (proper fix) try adding COMDAT identifier (+hidden visibility) to coverage module constructors. We hope this will the problem completely.

- develop a separate reproducer for gold (and report)

- If proper fix fails try building without gc-sections.
Blockedon: 685244
The comdat fix is in and bots look happy:

https://reviews.llvm.org/rL294517

I have verified that linker doesn't have performance problem with the fix. Let's hope it gets into next clang roll ( issue 685244 ).
Project Member

Comment 34 by bugdroid1@chromium.org, Feb 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fd938bac4105603ff6a997d5de3cabe39d8869c0

commit fd938bac4105603ff6a997d5de3cabe39d8869c0
Author: aizatsky <aizatsky@chromium.org>
Date: Mon Feb 27 23:49:52 2017

Switching libfuzzer to trace-pc-guard instrumentation.

This reverts commit 23d6de2177b812285946bce6f459815799d825c5.

BUG=651540, 687570 

Review-Url: https://codereview.chromium.org/2721593003
Cr-Commit-Position: refs/heads/master@{#453396}

[modify] https://crrev.com/fd938bac4105603ff6a997d5de3cabe39d8869c0/build/config/sanitizers/sanitizers.gni

Status: Fixed (was: Assigned)
Buildbots are running fine.

Sign in to add a comment