Alerts: https://groups.google.com/a/google.com/forum/#!searchin/chrome-troopers/%22Free$20disk$20space$20on$20E$3A%5C$20is$20below$205$25%22$20after$3A2016-05-26

A concrete example, vm715-m4:
https://build.chromium.org/p/tryserver.chromium.win/buildslaves/vm715-m4

two consecutive builds on that machine:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang/builds/28470
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/229013

in the beginning of the 1st build: 171GB/249G is used (68%)
in the beginning of the 2nd build: 242GB/249GB is used (97%)

will ssh

+people who may know how this works.

vm715-m4:/cygdrive/e/b/build/slave/win_clang/build/src/out/Debug_x64 takes 128 Gb.

Biggest files there:

1.3G browser_tests.ilk
1.2G interactive_ui_tests.ilk
1.2G sync_integration_tests.ilk
1.2G performance_browser_tests.ilk
1.2G sync_performance_tests.ilk
1.1G chrome_app_unittests.ilk
1.1G load_library_perf_tests.ilk
984M mini_installer.exe
956M next_version_mini_installer.exe
917M chrome.ilk
911M chrome.7z
692M components_unittests.ilk
627M content.ilk
515M browser_tests.exe.pdb
503M unit_tests.exe.pdb
477M interactive_ui_tests.exe.pdb
475M sync_integration_tests.exe.pdb
475M performance_browser_tests.exe.pdb
474M sync_performance_tests.exe.pdb
472M chrome_app_unittests.exe.pdb
472M load_library_perf_tests.exe.pdb

Is this normal?

Hm. I don't know how big these files usually are. Since they were fine for many weeks (as far as I know) I'll go out on a limp and claim that the ilk files are usually not that big.

I switched win_clang to a different linker for a while today (https://codereview.chromium.org/1881313002/), which I reverted after a while. The ilk files are the files that the linker we used before today and that we use again after the revert uses for incremental linking. It's possible that that linker's incremental link mode didn't expect that a different linker wrote the executable in between.

Is it possible to delete all ilk pdb exe files in all build directories of all slaves of the win_clang bot?

> Is it possible to delete all ilk pdb exe files in all build directories of all slaves of the win_clang bot?
I am not sure how to do it in automated way.
Labs, do we have such tools?

We don't have tools to auto delete things. If there's no way to clobber the output folder via the build system, then it has to be done using standard ssh access.

thakis, I assume it is safe to delete .ilk files during the build after compile step is complete?

Yes, deleting ilk files should be fine.

*.ilk files take 25 Gb
*.pdb files take 77 Gb

Deleting pdb files is fine as well. I'd delete exe and dll files too, just to be safe.

Deleted *.exe, *.dll, *.pdb, *.ilk
Now only 115G//249G is used (55%)

There is 177 machines affected. 

I was checking if a machine is not running win_clang build and then I was running
  ssh $vm df -h '&&' cd /cygdrive/e/b/build/slave/win_clang/build/src/out/Debug_x64/ '&&' rm *.exe *.dll *.pdb *.ilk '&&' df -h
on individual machines.

Most of machines run win_chromium_rel_ng, so I wrote a loop in bash that runs the aforementioned command for all machines that run win_clang.

Machines are cleaned up. Alerts for some machines are still active https://alertmanager.corp.google.com/#view=silenceDetails&query=2623631943136182272, but presumably will deactivate in a few

Alerts are not firing anymore.

thakis, apparently win_clang builder still generates a lot of files, for example I cleaned up vm840-m4, but it is full again.

*.(exe|dll|pdb|ilk) files in vm840-m4:/cygdrive/e/b/build/slave/win_clang/build/src/out/Debug_x64 take 102Gb, whereas entire src/out/Debug_x64 in "win" builder is 35M.
vm840-m4 has 250G disk, so 102Gb seems for exe,dll,ilk,pdb seems a lot.

Do you mean 35G instead of 35M?

It's expected that win_clang's build dir is somewhat larger as it builds all targets, instead of just the targets it runs tests for.

I don't know if the size of the files that both bots build is about the same. If not, I could take a look at what causes this.

If this is causing problems, we could disable incremental linking on win_clang, then none of the ilk files would exist as far as I know. We could also switch to fastbuild=2, then the pdbs wouldn't exist either. Both changes would mean that the bot would test a different config than people are using locally, but both changes can be made quickly (a small bot config change each).

If this isn't actively causing problems, we can investigate if and why clang/win causes link.exe to write larger binaries, but that'll take a while.

I would guess that we're now back at where the bot was for a while, so hopefully this is no longer P1.

I meant 35M.

Compilation in "win" directory on the machine completed today morning [1] and "/cygdrive/e/b/build/slave/win/build/src/out/Debug_x64" takes only 34M as of now.

"/cygdrive/e/b/build/slave/win/build/src/out/Debug_x64" takes 123G where 
*.pdb are 70G
*.ilk are 24G
*.exe are 8.4G
*.dll are 1004M

I don't know if [1] proves that non-clang compilation takes much less space because targets might be different.

[1]: https://uberchromegw.corp.google.com/i/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/220952/steps/compile%20%28with%20patch%29


---

Currently affected machines are

vm840-m4
vm247-m4
vm625-m4

I will clean them up again and we will see if the problem repeats.

I've deleted .pdb and .ilk files from those machines. If no alerts fire in an hour, I will close the bug. {Pri-1, Infra-Troopers} is appropriate for this bug.

Sorry, I don't believe that all build artifacts on Windows combined are 35M :-) chrome.dll alone is bigger than that. Maybe you checked on a slave that never did a windows build?

You looked in Debug_x64, but win might do 32-bit builds, in which case it build artifacts would be in Debug, not Debug_x64. win_clang does 64-bit builds, which means its artifacts are in Debug_x64, not Debug.

Indeed it builds something else, thanks.

win's Release_x64 takes 25G, where
*.pdb are 12G
*.exe are 4.6G
*.dll are 822M
no *.ilk files

Alerts are not firing anymore. https://alertmanager.corp.google.com/#view=silenceDetails&query=2623646820902895616

There is a chance that I actually did not clean up these 3 machines yesterday and there is no problem. I will disable alert silence for now.

Yes, release builds default to not doing incremental builds. win_debug_compile_only_ng is a non-clang debug build bot which will have ilk files as well. Could you check there too?

Did you mean win_chromium_compile_dbg_ng?

Yes :-)

vm653-m4 executed https://uberchromegw.corp.google.com/i/tryserver.chromium.win/builders/win_chromium_compile_dbg_ng/builds/197529 recently

its /cygdrive/e/b/build/slave/win/build/src/out/Debug is 34G
where ilk files are 9.1G
and pdb are 6.9G

Alerts are still not firing.

Ok, sounds like win_clang needs about 3x the disk of non-clang debug bots. Some of that is expected due to it building more targets, but 3x sounds a bit high. Maybe you can file a bug on me to investigate where the size goes. (Assuming that alerts stay off and that this bug here can be closed.)

 issue 615437 

Thanks! Do you happen to have out/ size for a win_clang build after deleting all the files once too? The 101GB above are for bots that (possibly) hadn't been cleaned yet from what I understand.

I think you meant "102GB above". Yes, that was before cleaning up.

vm840-m4:/cygdrive/e/b/build/slave/win_clang/build/src/out is 31G as of now.

Handing off to the current trooper (my shift ended yesterday). This seems to be pretty much fixed. 

Hm, that doesn't look necessarily bigger than regular win dbg bots, cool :-)

I'll still check next week on the other bug, to make sure.

This looks fixed (alerts are gone)