Puppet runs only outright fail if manifest compilation fails, otherwise you only get a failing resource, which might not be easily visible, especially since we still get failure flake (I believe it's apt failing to grab the lock).

I don't think I fully grasp the context, so correct me.

Yep, the ask here is for a dashboard that would let us detect when we start failing more resources.

An easier partial solution would be a dashboard to show sudden spike in number of failed resources. This happens when a very basic step fails (say, in my case something in profiles/base) because it leads to a lot of dependent packages to be skipped. So, if we tracked total number of packages failed + skipped, we'd see a sudden spike.

But there are also failure modes where an important but terminal package fails. Detecting this would require us to clearly flag _any_ failed packages if they start failing consistently.

So 
(1) A dashboard for # skipped + failed packages changing suddenly.
(2) A dashboard for # packages failing consistently (i.e., no pass in ~4 hours?)

 Issue 820242  has been merged into this issue.

Downgrading P2s that haven't been modified in more than 6 months, which have no component or owner.