This is the bug to track discussion on the recent outage where a bad image push resulted in Linux bot outage. 

- How do we determine when a canary image is good or bad? 
- Should we implement an automated monitoring system that would identify (specific test, specific system property) to correlate with consistent failures? 

Postmortem tracked under go/cit-pm-75. Assigning to nodir to comment as per Trooper discussion. 
 

This is the bug to track discussion on the recent outage where a bad image push resulted in Linux bot outage. 

- How do we determine when a canary image is good or bad? 
- Should we implement an automated monitoring system that would identify (specific test, specific system property) to correlate with consistent failures? 

Postmortem tracked under go/cit-pm-75. Assigning to nodir to comment as per Trooper discussion. 

More general, we need to be able to detect an increase of a failure rate associated with any parameter that may affect the build result, not necessarily image, but also, for example, versions of kitchen, git, bot id (all builds of a specific bot are failing, while other builds of the same builder on other bots do not fail that often), etc.