As John mentioned in the chat, this may have destabilizing effect on a fleet:

1. Imaging a compilation task took slightly longer than allowed time and were killed due to timed out.
2. Its cache is deleted.
3. Next time it runs, it will take even more longer, hitting a timeout for sure.
4. Goto 2.

In another scenario, if timeouts are caused by something shared and external (e.g. Swarming outage, Goma outage), many bots will remove they caches all at once, making the recovery slow (or overloading something else).

So it's not clear when we should delete caches automatically (if ever) :(

Is it possible to create a distinction between regular named caches and "disposable" named caches s.t. we could dispose of the latter but not the former in the event of a SIGTERM?

(e.g., we might want to dispose of the builder cache automatically, but we probably don't want to entirely dispose of the git cache, so the former would be disposable while the latter would not be)

More knobs means the next noogler will be surprised and in general recipes will start to drift to become misconfigured*, so we must trade off carefully.

* Especially that this kind of misconfiguration would only take effect under exceptional circumstances.