On win76-c1 (ToTWinThinLTO64):

thinlto-cache contains 390,249 files taking up ~67GB of space.
Of those, Thin-*.tmp.o accounts for 289,750 files, ~65GB of space.

For comparison, obj has ~56K files taking ~43GB, and the top-level
build output has 1,626 files taking up ~31GB.

This adds up to about 74GB for the obj and top-level output directory
combined, and 67GB for the thinlto-cache directory, for a total of
141GB in those 3 space-heavy locations.

It seems implausible that all the Thin-*.tmp.o files are from a single
build. As I understand it, we create one for each llvmcache-* file we are
about to create, then rename the .tmp.o file to the desired name once we
are done writing it. Files we don't preserve in that way are supposed to
be cleaned up (they are marked delete on close). I suspect that this cleanup is
not always happening. We also don't clean up those files during cache pruning, because that only considers files with names starting in "llvmcache-".

We are leaking tempfiles. This can be reproduced by doing a CFI build (ninja all) and stopping it with control+C during linking.

I also wrote a small reproducer that spawns 100 threads, each of which creates a TempFile and then tries to keep() it to the same final name as every other thread, and the main thread exits before all threads have finished.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6b340a5eff263abad65cecb22ce840baf34be2be

commit 6b340a5eff263abad65cecb22ce840baf34be2be
Author: inglorion <inglorion@chromium.org>
Date: Thu Aug 23 03:04:45 2018

Stop using the ThinLTO cache with lld-link

lld-link can use a cache to speed up ThinLTO builds. Unfortunately,
this cache leaks temporary files on Windows, causing storage to fill
up. Until this has been addressed, this disables the cache, avoiding
the problem.

Bug:  871962 
Change-Id: I415162233bc8855b3d7e237e80c23cbbc24bcfeb
Reviewed-on: https://chromium-review.googlesource.com/1185856
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Bob Haarman <inglorion@chromium.org>
Cr-Commit-Position: refs/heads/master@{#585380}
[modify] https://crrev.com/6b340a5eff263abad65cecb22ce840baf34be2be/build/config/compiler/BUILD.gn

We are no longer running out of disk space, so I'm closing this bug. For the longer term, we are re-architecting the way we do ThinLTO builds. crbug.com/877722 tracks that effort.