I've just set the threshold to 2 hours. Going to see if that helps. 

This may be useful, by the way: https://g3doc.corp.google.com/cloud/gong/stackdriver/monitoring/pusher/g3doc/index.md?cl=head

The NextAction date has arrived: 2018-01-12

In the last two days we haven't had the noise we usually do in terms of single failures. I'll set the next check in a little later then. 

The NextAction date has arrived: 2018-01-15

The tuning appears to be working as intended! At least, it does not fire when the job fails and then succeeds on the next run. 

I had previously configured Stackdriver to send emails to chromium+alert@monorail-staging.appspotmail.com, and would expect to see some bugs there, but I don't. So now I'm trying to figure out how to troubleshoot. I don't see anything in the logs for the relevant time period (1/2-1/5) when I search for "/_ah/mail/chromium+ALERT@monorail-staging.appspotmail.com." I've asked zhangtiff@ for more advice on troubleshooting. 

The NextAction date has arrived: 2018-01-17

Blocking this on 798421 because if we use the cloud metrics within google3 infra we can incorporate the alerts in our Monarch configs (I think). This would be better anyway.

This bug refers to the cq-attempts failing policy in the project chrome-infra-events. We want to be able to configure the same alert in our Monarch configs. 

This is blocked on 798421 which aims to establish a consistent way of accomplishing this.