Such a limit will prevent scheduling more builds than we have builders, and so prevent what I believe to be the main cause of PRECQ_LAUNCH_TIMEOUT errors.

This would replace the current concept of MAX_LAUNCHES_PER_CYCLE_DERIVATIVE, which limits how many jobs we launch in a single PreCQ launcher iteration.

Another option is to only timeout the builds not in 'scheduled' status in buildbucket.

A build will be marked as 'scheduled' in buildbucket once it's scheduled by pre-cq-launcher. when a buildslave picks up the build, it will mark it as 'started', and the build will be marked as 'failure' or 'success' after completion. we can only timeout the builds which are not in 'scheduled' status which mean they have already been picked up by buildslaves. so we don't need to add more logic to setup the cycle or get current running/ideal buildslaves.

SGTM.

PS: We are just starting to get accurate usage metrics, and we did briefly max out our PreCQ builders during the last 12 hours.

So... solving this will probably affect users.

A secondary nice-to-have feature. When there is a backlog, don't try to schedule 'speculative' PreCQ runs.

bmgordon has been thinking about this