Don't know who this is best to route to/whether or not it's worth all of the effort; more just throwing the idea out there.

Issue 914583 brings up an interesting point: ccache is hard, and bugs in it are sometimes difficult to troubleshoot (and even notice, for that matter).

It seems reasonable to me for us to have a system that's capable of building some arbitrary TU both with and without ccache, then compares the output files. If there's a mismatch, yell/scream/complain with some ccache debugging info. (Naturally, we can skip the "without ccache" if ccache reported that it wasn't a cache hit)

On its own, this would be nice to have as a large hammer. I think the real value that it can provide, though, is if we can deploy it and turn it on for all users of ccache.

This would end up making all builds slower. This can be mitigated by only doing this verification on a subset of builds. Picking an arbitrary but deterministic subset doesn't seem too difficult; for example, we could hash the build command, and compare the last N bits of that to the last N bits of chromiumos-overlay's master hash (or origin/master, or ... anything local, reproable on another dev box, and sometimes-changing). Importantly, as much as possible, we should check the same set of build commands if the user emerges the same package repeatedly.

If we crank N up enough (6, 7, ...), we'd only have to do this verification on 1/64 or 1/128 or ... of all cache hits, which I'd naively hope would be close enough to noise, given that ccache already has to preprocess everything for every invocation anyway.

Any thoughts are appreciated