This reproduces locally, try building app_shell_unittests locally. I see linking hanging
132853 aarya     20   0   16.6g  13.0g   9.5g S 100.0  20.7  80:32.68 ld.lld

Thanks for identifying a reproducer, I'll take a look why it's hanging.

This is lower priority, we still have  issue 851230  as a blocker.

I launched the reproducer and went to a meeting. Later on, I've realized that it has finished:

$ cat out/cov2/args.gn 
# Build arguments go here.
# See "gn args <out_dir> --list" for available build arguments.
use_sanitizer_coverage=false
use_goma = true
use_clang_coverage = true
dcheck_is_configurable = true
ffmpeg_branding = "ChromeOS"
is_component_build = false
is_debug = false
proprietary_codecs = true
strip_absolute_paths_from_debug_symbols = true
use_libfuzzer = true


$ time ninja -C out/cov2/ -j1000 app_shell_unittests
ninja: Entering directory `out/cov2/'
[31198/31198] LINK ./app_shell_unittests

real	83m0.126s
user	104m51.552s
sys	2m23.392s


I'm now rerunning again withuot use_sanitizer_coverage=false to make sure it's indeed faster.

Yeah, it's 20 times faster without use_sanitizer_coverage=false:

$ time ninja -C out/cov/ -j1000 app_shell_unittests
ninja: Entering directory `out/cov/'
[30928/30928] LINK ./app_shell_unittests

real	4m26.873s
user	10m57.160s
sys	4m3.764s

Below are exact link flags and their diff:

$ rm out/cov/app_shell_unittests && time ninja -v -C out/cov/ -j1000 app_shell_unittests
ninja: Entering directory `out/cov/'
[1/1] python "../../build/toolchain/gcc_link_wrapper.py" --output="./app_shell_unittests" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=lld -Wl,--color-diagnostics -m64 -Werror -Wl,-O2 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_sid_amd64-sysroot -L../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -fprofile-instr-generate -fsanitize-coverage=trace-pc-guard -rdynamic -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,-u_sanitizer_options_link_helper -fsanitize-coverage=trace-pc-guard -o "./app_shell_unittests" -Wl,--start-group @"./app_shell_unittests.rsp"  -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lresolv -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lpangocairo-1.0 -lpango-1.0 -lcairo -lXss -lasound -lm -lz -lpci -ldbus-1 -latk-1.0 -latk-bridge-2.0 -lcups 

real	1m37.269s
user	0m42.088s
sys	0m44.576s


$ rm out/cov2/app_shell_unittests && $ time ninja -v -C out/cov2/ -j1000 app_shell_unittests
ninja: Entering directory `out/cov2/'
[1/1] python "../../build/toolchain/gcc_link_wrapper.py" --output="./app_shell_unittests" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -Wl,--as-needed -fuse-ld=lld -Wl,--icf=all -Wl,--color-diagnostics -m64 -Werror -Wl,-O2 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_sid_amd64-sysroot -L../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -fprofile-instr-generate -rdynamic -Wl,-rpath-link=. -Wl,--disable-new-dtags -o "./app_shell_unittests" -Wl,--start-group @"./app_shell_unittests.rsp"  -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lresolv -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lpangocairo-1.0 -lpango-1.0 -lcairo -lXss -lasound -lm -lz -lpci -ldbus-1 -latk-1.0 -latk-bridge-2.0 -lcups 

real	77m58.924s
user	91m59.784s
sys	0m17.652s



>>> for f in cov:
...   if f not in cov2: print f
... 
-fsanitize-coverage=trace-pc-guard
-Wl,-rpath=\$ORIGIN/.
-Wl,-u_sanitizer_options_link_helper
-fsanitize-coverage=trace-pc-guard
>>> 
>>> 
>>> for f in cov2:
...   if f not in cov: print f
... 
-Wl,-z,defs
-Wl,--as-needed
-Wl,--icf=all

I would suspect "-Wl,--as-needed" to be a culprit, but actually removing "-Wl,--icf=all" gives a crazy speedup:


mmoroz@mmoroz2:~/Projects/new/chromium/src/out/cov2$ time python "../../build/toolchain/gcc_link_wrapper.py" --output="./app_shell_unittests" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -Wl,--as-needed -fuse-ld=lld -Wl,--color-diagnostics -m64 -Werror -Wl,-O2 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_sid_amd64-sysroot -L../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -fprofile-instr-generate -rdynamic -Wl,-rpath-link=. -Wl,--disable-new-dtags -o "./app_shell_unittests" -Wl,--start-group @"./app_shell_unittests.rsp"  -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lresolv -lgio-2.0 -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lpangocairo-1.0 -lpango-1.0 -lcairo -lXss -lasound -lm -lz -lpci -ldbus-1 -latk-1.0 -latk-bridge-2.0 -lcups

real	0m29.320s
user	0m29.580s
sys	0m17.860s
mmoroz@mmoroz2:~/Projects/new/chromium/src/out/cov2$ du -hs app_shell_unittests
1.7G	app_shell_unittests
mmoroz@mmoroz2:~/Projects/new/chromium/src/out/cov2$ du -hs ../cov/app_shell_unittests
3.5G	../cov/app_shell_unittests


Note, how significantly smaller is the binary file produced.

Sounds reasonable, as ICF likely has hard times handling all the instrumentation inserted by the compiler: https://ai.google/research/pubs/pub36912

I'm testing some targets locally and uploading a CL.

Cleaned up both out directories and re-built bunch of targets. Build time seems to be improved by 35%, binaries size by 50%.

mmoroz@mmoroz2:~/Projects/new/chromium/src$ time ninja -C out/cov/ -j1000 app_shell_unittests net_unittests crypto_unittests browser_tests chrome ulpfec_header_reader_fuzzer fingerprint_fuzzer audio_decoder_ilbc_fuzzer media_cenc_utils_fuzzer net_reporting_header_parser_fuzzer blink_json_parser_fuzzer payment_method_manifest_fuzzer rtp_header_fuzzer && time ninja -C out/cov2/ -j1000 app_shell_unittests net_unittests crypto_unittests browser_tests chrome ulpfec_header_reader_fuzzer fingerprint_fuzzer audio_decoder_ilbc_fuzzer media_cenc_utils_fuzzer net_reporting_header_parser_fuzzer blink_json_parser_fuzzer payment_method_manifest_fuzzer rtp_header_fuzzer 
ninja: Entering directory `out/cov/'
[1/1] Regenerating ninja files
[44822/44822] LINK ./browser_tests

real	8m53.258s
user	21m53.220s
sys	9m35.652s
ninja: Entering directory `out/cov2/'
[1/1] Regenerating ninja files
[45202/45202] LINK ./browser_tests

real	5m41.390s
user	19m55.136s
sys	4m48.704s
mmoroz@mmoroz2:~/Projects/new/chromium/src$ 
mmoroz@mmoroz2:~/Projects/new/chromium/src$ 
mmoroz@mmoroz2:~/Projects/new/chromium/src$ 
mmoroz@mmoroz2:~/Projects/new/chromium/src$ cd out/cov
mmoroz@mmoroz2:~/Projects/new/chromium/src/out/cov$ du -hs app_shell_unittests net_unittests crypto_unittests browser_tests chrome ulpfec_header_reader_fuzzer fingerprint_fuzzer audio_decoder_ilbc_fuzzer media_cenc_utils_fuzzer net_reporting_header_parser_fuzzer blink_json_parser_fuzzer payment_method_manifest_fuzzer rtp_header_fuzzer
3.5G	app_shell_unittests
905M	net_unittests
67M	crypto_unittests
4.9G	browser_tests
4.6G	chrome
40M	ulpfec_header_reader_fuzzer
740M	fingerprint_fuzzer
39M	audio_decoder_ilbc_fuzzer
225M	media_cenc_utils_fuzzer
405M	net_reporting_header_parser_fuzzer
3.2G	blink_json_parser_fuzzer
741M	payment_method_manifest_fuzzer
39M	rtp_header_fuzzer
mmoroz@mmoroz2:~/Projects/new/chromium/src/out/cov$ cd ../cov2/
mmoroz@mmoroz2:~/Projects/new/chromium/src/out/cov2$ du -hs app_shell_unittests net_unittests crypto_unittests browser_tests chrome ulpfec_header_reader_fuzzer fingerprint_fuzzer audio_decoder_ilbc_fuzzer media_cenc_utils_fuzzer net_reporting_header_parser_fuzzer blink_json_parser_fuzzer payment_method_manifest_fuzzer rtp_header_fuzzer
1.7G	app_shell_unittests
393M	net_unittests
35M	crypto_unittests
2.3G	browser_tests
2.3G	chrome
20M	ulpfec_header_reader_fuzzer
417M	fingerprint_fuzzer
19M	audio_decoder_ilbc_fuzzer
118M	media_cenc_utils_fuzzer
202M	net_reporting_header_parser_fuzzer
1.5G	blink_json_parser_fuzzer
417M	payment_method_manifest_fuzzer
19M	rtp_header_fuzzer


Now re-building stuff once again in a different order just to make sure there isn't any dependency (caused by goma) and my results are valid.

Well, it actually seems to be building 2 times faster as well:

$ time ninja -C out/cov2/ -j1000 app_shell_unittests net_unittests crypto_unittests browser_tests chrome ulpfec_header_reader_fuzzer fingerprint_fuzzer audio_decoder_ilbc_fuzzer media_cenc_utils_fuzzer net_reporting_header_parser_fuzzer blink_json_parser_fuzzer payment_method_manifest_fuzzer rtp_header_fuzzer && time ninja -C out/cov/ -j1000 app_shell_unittests net_unittests crypto_unittests browser_tests chrome ulpfec_header_reader_fuzzer fingerprint_fuzzer audio_decoder_ilbc_fuzzer media_cenc_utils_fuzzer net_reporting_header_parser_fuzzer blink_json_parser_fuzzer payment_method_manifest_fuzzer rtp_header_fuzzer 
ninja: Entering directory `out/cov2/'
[1/1] Regenerating ninja files
[45202/45202] LINK ./browser_tests

real	3m52.490s
user	19m57.568s
sys	4m37.924s
ninja: Entering directory `out/cov/'
[1/1] Regenerating ninja files
[44822/44822] LINK ./browser_tests

real	8m17.855s
user	22m9.596s
sys	8m57.548s

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/928c2b5c513e879475afec09ef14c3e2551d865c

commit 928c2b5c513e879475afec09ef14c3e2551d865c
Author: Max Moroz <mmoroz@chromium.org>
Date: Fri Jun 22 18:09:17 2018

Build: disable ICF when using clang_code_coverage=true.

When ICF (Identical Code Folding) is enabled, the linker has hard time figuring
out all identical code patterns inserted by compiler's instrumentation. That is
why this option is disabled when we use sanitizers. It also should be disabled
for code coverage instrumented build, as it has even more instrumentation code.

This change allows code coverage bots to use use_sanitizer_coverage=false which
would result in much faster and much smaller builds.

Performance improvements:
- builds 2x faster
- binaries 2x smaller
- run time 2x smaller (or even more on some targets, up to 6x)

Bug:  851337 
Change-Id: I20d4ad1e55730daf813e0f26060dd5830351dbcf
Reviewed-on: https://chromium-review.googlesource.com/1112199
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#569693}
[modify] https://crrev.com/928c2b5c513e879475afec09ef14c3e2551d865c/build/config/compiler/BUILD.gn

New reports look good, nothing seems to be broken after this change.