I think the reason we haven't implemented more step-specific monitoring is because Monarch can't handle much variability in metric values, and we have a lot of different step types.

Swarming does some monitoring by step: https://cs.chromium.org/chromium/infra/luci/appengine/swarming/ts_mon_metrics.py?q=spec_name&l=46 so it looks like we could easily add some kind of result monitoring there. If fact, we might have that already: https://cs.chromium.org/chromium/infra/luci/appengine/swarming/ts_mon_metrics.py?q=spec_name&l=46


For non-swarming tasks, we could do identify some common patterns we care most about, and add the monitoring for steps that match those patters. We currently do this for bot_update duration. (https://cs.chromium.org/chromium/build/scripts/master/status_logger.py?l=449)

Much of the monitoring infrastructure is still kinda opaque to me, and so I'm not sure what "Monarch can't handle much variability in metric values, and we have a lot of different step types" means?

Katie's absolutely right in #1.
dpranke: see go/monarch-high-cardinality-metrics for details.

We can do monitoring based on Dremel queries too. Data about builds and steps is available from the chrome_infra.completed_builds table [1]. I am not sure however, how to do alerts based on Dremel queries, but sergeyberezin@ may be able to give a pointer.


[1] https://plx.corp.google.com/#/table/chrome_infra::completed_builds

P.S. Another issue is that there is a 1-1.5 hour delay between the event and the moment data arrives to the Dremel table. So we may be getting an alert, but it may arrive too late.

Actually, Sergey has already commented on Dremel alerting earlier in https://bugs.chromium.org/p/chromium/issues/detail?id=659344#c3.

Removing Infra>Monitoring since this is a CQ related alert modification. Please reserve Infra>Monitoring for monitoring (ts_mon and event_mon) bugs. Added Ops-AddMonitoring label to track monitoring related tasks.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

It might still be relevant, especially the idea of monitoring steps that fail both with and without the patch (if we're still doing those). But it's likely still a P3.