When you say we have alerting on the total number of failed builds, what are you referring to? I don't see anything quite like that in buildbot_alerts or the playbook, though we do alert on infra failures.

I was thinking of the OverallBuildInfraFailuresAlert, as you guessed.

Though, we should probably have alerts for when there's too many failed builds, period, as well.

Agreed, just wanted to make sure I understand the description.

Friendly ping. This is a blocking bug on cit-pm-55. Please update need and priority accordingly. 

The groundwork is being laid out in issue 873754, we can add this alert in addition to the infra failure alerts. The graphs already exist in http://vi/auto/prod:chrome-ops-client-infra/chrome_client/builds , though in absolute counts. I imagine it'd be more useful to have a graph of a percentage of failed builds, for easier diagnosis.

Thought of the day: the situation described in #0 sounds more like a sheriffing issue, since it's a src-side CL that broke builds. I'm not sure if it also broke CI builds for sheriffs to notice though.

These days we also have pending builds alerts for Chromium CQ, and we'd received an alert if CQ got overloaded.

Aside from that, I wonder, in general, if there is a meaningful action a trooper can take when lots of builds start failing, other than pinging a sheriff. In which case, maybe we need to alert sheriffs directly?

 Issue 674683  has been merged into this issue.

Per #9 - should we consider this bug closed?