https://reviews.llvm.org/D42713 should fix.

I don't know if it's the same thing, but CFI Linux ToT is failing with the same assert: https://ci.chromium.org/buildbot/chromium.clang/CFI%20Linux%20ToT/1657

Hopefully D42713 fixes it; looks like it's ready to land :-)

Peter, will you commit D42713 or are there any other issues?

r324253 (sorry, was OOO part of last week)

Looks like the fix was not enough.
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FCFI_Linux_ToT%2F1722%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

I think I know what the problem is. Patch incoming.

https://reviews.llvm.org/D42934

The bot is now failing with out-of-disk errors.
From https://ci.chromium.org/buildbot/chromium.clang/ToTLinuxThinLTO/1137

[48710/49778] LINK ./vr_common_perftests
FAILED: vr_common_perftests 
python "../../build/toolchain/gcc_link_wrapper.py" --output="./vr_common_perftests" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -Wl,--no-as-needed -lpthread -Wl,--as-needed -fuse-ld=lld -Wl,--icf=all -flto=thin -Wl,--thinlto-jobs=8 -Wl,--thinlto-cache-dir=thinlto-cache -Wl,--thinlto-cache-policy,cache_size=10\% -Wl,--lto-O0 -fwhole-program-vtables -m64 -Werror -Wl,-O1 -Wl,--gc-sections -Wl,--gdb-index -Wl,--fatal-warnings -nostdlib++ --sysroot=../../build/linux/debian_stretch_amd64-sysroot -L../../build/linux/debian_stretch_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_stretch_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_stretch_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_stretch_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,--export-dynamic -o "./vr_common_perftests" -Wl,--start-group @"./vr_common_perftests.rsp"  -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lexpat -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lpci -lXss -lgio-2.0 -lresolv -lasound -lm -lz -ldbus-1 -latk-1.0 -latk-bridge-2.0 -lpangocairo-1.0 -lpango-1.0 -lcairo -lcups 
LLVM ERROR: IO failure on output stream: No space left on device
LLVM ERROR: Failed to rename temporary file thinlto-cache/Thin-3573d1.tmp.o to thinlto-cache/llvmcache-34A2D08ACEA7651846070F848F1CD93FD05F2304: No such file or directory
LLVM ERROR: Failed to rename temporary file thinlto-cache/Thin-809d27.tmp.o to thinlto-cache/llvmcache-F9800AF0E9B8B8C47BBBAF3CFE62CE2C9F11A1C4: No such file or directory
clang-7.0: error: linker command failed with exit code 1 (use -v to see invocation)


There is a huge number of files in the thinlto cache dir:

thinlto-cache$ ls | wc -l
322368

(Maybe we should be sharding that into separte subdirs?)

I'll try clobbering the dir to see if that helps.

$ rm *
-bash: /bin/rm: Argument list too long

Heh.

Cleared it now; let's see if that helps.

Still failing :-(

From https://ci.chromium.org/buildbot/chromium.clang/ToTLinuxThinLTO/2013

This is strange because when I checked there as ~130 GB free space. Are we using up all of that, or is there something else going on?

[50244/50270] LINK ./webkit_unit_tests
FAILED: webkit_unit_tests 
python "../../build/toolchain/gcc_link_wrapper.py" --output="./webkit_unit_tests" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -Wl,--no-as-needed -lpthread -Wl,--as-needed -fuse-ld=lld -Wl,--icf=all -flto=thin -Wl,--thinlto-jobs=8 -Wl,--thinlto-cache-dir=thinlto-cache -Wl,--thinlto-cache-policy,cache_size=10\% -Wl,--lto-O0 -fwhole-program-vtables -m64 -Werror -Wl,-O1 -Wl,--gc-sections -Wl,--gdb-index -Wl,--fatal-warnings -nostdlib++ --sysroot=../../build/linux/debian_stretch_amd64-sysroot -L../../build/linux/debian_stretch_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_stretch_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_stretch_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_stretch_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,--export-dynamic -o "./webkit_unit_tests" -Wl,--start-group @"./webkit_unit_tests.rsp"  -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lnss3 -lnssutil3 -lsmime3 -lplds4 -lplc4 -lnspr4 -lexpat -lgio-2.0 -lresolv -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lXtst -lXrandr -lpci -lXss -lasound -lm -lz -ldbus-1 -lpangocairo-1.0 -lpango-1.0 -lcairo -latk-1.0 -latk-bridge-2.0 -lcups 
LLVM ERROR: IO failure on output stream: No space left on device
LLVM ERROR: IO failure on output stream: No space left on device
clang-7.0: error: linker command failed with exit code 1 (use -v to see invocation)

We should be limiting the cache to 10% of available disk space. We could also be hitting other limits such as the number of inodes or the number of files in a directory. The latter was fixed by https://reviews.llvm.org/D40327 the last time we saw it, but it may have come back (I measured the per-directory limit in ext4 to be around 5000000, but it may be lower on the bot for some reason). I'll try reducing the limit further to 100000 and seeing if that helps.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/71f030aaa5d720985ed18610cb7f3df619117731

commit 71f030aaa5d720985ed18610cb7f3df619117731
Author: Peter Collingbourne <pcc@chromium.org>
Date: Wed Feb 07 08:50:30 2018

build: Limit the size of the ThinLTO cache further.

Fix a TODO, as well as hopefully the out of disk space errors on
ToTLinuxThinLTO.

Bug:  807402 
Change-Id: I90dfebcb070b1b14207dfef29d5aa922243ccc14
Reviewed-on: https://chromium-review.googlesource.com/906102
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534953}
[modify] https://crrev.com/71f030aaa5d720985ed18610cb7f3df619117731/build/config/compiler/BUILD.gn

So that didn't help. After having taken a closer look at the bot, I think the problem is that we are legitimately out of disk space as a result of building all targets at symbol_level=2.

We probably want the target coverage of that bot in order to catch ThinLTO bugs that may affect developers, but we most likely don't need to build everything with symbol level 2. I'll decrease it to 1.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a58d7afef2a0f1c599b5eee0f4530e1ff382af5b

commit a58d7afef2a0f1c599b5eee0f4530e1ff382af5b
Author: Peter Collingbourne <pcc@chromium.org>
Date: Thu Feb 08 01:46:49 2018

Set symbol level to 1 on ToTLinuxThinLTO builder.

A symbol level of 2 is causing out of disk space errors on the bot.
While it's probably worth building everything in order to catch ThinLTO
bugs that may affect developers, there probably isn't as much value
in building all of it at symbol level 2.

Bug:  807402 
Change-Id: I40930a0c14bb1fbee2b324f4463a666497dcec61
Reviewed-on: https://chromium-review.googlesource.com/907760
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Peter Collingbourne <pcc@chromium.org>
Cr-Commit-Position: refs/heads/master@{#535244}
[modify] https://crrev.com/a58d7afef2a0f1c599b5eee0f4530e1ff382af5b/tools/mb/mb_config.pyl

https://ci.chromium.org/buildbot/chromium.clang/ToTLinuxThinLTO/2024 built successfully, so let's mark this fixed.

The build then failed interactive_ui_tests, but that's either a flake or at least separate issue.