hrm, I'll have to think about whether there's a way we could aggregate step times absent the step name that still gives us meaningful information. The expected runtimes of steps varies wildly.

How fine-grained could the buckets be? Shifts in the bucket distribution could potentially be interesting, though at that point we'd be using the buckets as a proxy for the step names.

*vary* wildly, argh

I think there are a couple of ways we could think about it. 

Steps with similar timeouts could be in the same bucket. That would require us to have not-overly-generic timeouts for our steps though, which I'm pretty sure is not always the case. 

We could look at a sample of steps and group them by how long they *should* take. 

We could just pick some buckets... Steps that should finish in < 1m, < 5m, <15m, <30m, <1h, >1h. 

ooo, interesting, I hadn't thought about that.

If we do something like that, what do we wind up alerting on? Can we alert on a combination of conditions that indicate that something has definitively regressed? (Increase in 15m bucket population + decrease in 5m bucket population + no significant change in other buckets, for example?)

Oh, interesting, I thought we'd send different metrics, one for each bucket. For example /chrome/infra/build/step/<bucket>/durations where we send the durations for steps that *should* be in <bucket>.

This is an interesting idea too though! I wonder if we could alert on the rate of increase in population size of a given bucket.  

I would suggest that alerting (in the form of a real-time interrupt) should be reserverd for stuckness, while gentle regressions can be analysed offline at a much lower frequency and perhaps generate a ticket for investigation.

The two data forms you can use for those are quite different, and the second can be  a lot more sophisticated.

We're discussing what metric should be used alerting on that stuckness. Any advice on how to do that given that the expected step duration varies from seconds to ~1h?

My intuition is that a stuckness alert looks something like 'total (all steps) time running > average of last 5 total completion times + 50%'

but that depends a lot on how quickly you need to respond to stuckness and how exceptional a condition it is. In particular, if you're seeing things stuck very often, more localised automatic responses are going to be necessary to avoid paging yourself to death, and that system could potentially act at the level of individual steps with retry logic and data about each step's typical time taken.

I spoke with packrat@ about this, and we decided that it would be helpful to write a lightweight design doc that outlines the context for this issue and highlights some of the related outages we've seen. This would help SRE advise us on a best path forward, and also just generally be helpful. 

Bumping the priority on this because it would allow us to *not* rely on completed_steps event data for a few graphs (Cycle time 50/90% and Success rate (2-hour/1-day window)) on Buildbot/buildbot.