Issue 781366 has been merged into this issue.

Issue 781365 has been merged into this issue.

Issue 781364 has been merged into this issue.

Issue 781448 has been merged into this issue.

Issue 781447 has been merged into this issue.

Issue 781446 has been merged into this issue.

I've terminated swarm921-c4 and swarm925-c4 via the web ui, i.e. the button "shutdown gracefully" at https://chromium-swarm.appspot.com/bot?id=swarm925-c4

There's a problem where the terminate task is not getting scheduled in time (Again). Filed  issue 781487  for this. :(

So I speculate the bot has trouble cleaning up the caches because of undeleatable files, e.g. /b/swarming/c/sE/android_compile_x86_rel/src/build/linux/debian_jessie_amd64-sysroot/var/log/speech-dispatcher

I'll clean up a bit and will see how it behaves.

That's red herring, the files are owned by chrome-bot but I was under another account.

This was much more silly

https://chrome-internal-review.googlesource.com/c/infradata/config/+/496094 should fix it but I'm not in a mood to commit it on a Friday night just before going to bed without at least someone else to look at it.

If you see this and can monitor, feel free to do submit this CL.

I confirmed on swarm925-c4 that /b/swarming/c filed the disk. There's also something wrong in the way cache sizes is reported in the state, I'll have to take a look at this later.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ae1fdb1a4c14f7632600dab0ca09734caef78fb6

commit ae1fdb1a4c14f7632600dab0ca09734caef78fb6
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Sun Nov 05 21:19:52 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/081dcf38cee11ff586b139441bbd27a17f21b371

commit 081dcf38cee11ff586b139441bbd27a17f21b371
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Sun Nov 05 21:25:25 2017

Looks like this is fixed.