All chromium.* try builders should have configured hard timeouts |
|||
Issue descriptionIn order to be able to ensure capacity, all chromium.* tryjob builders should have hard timeouts configured. Configuring a timeout will allow us to detect and reject jobs that either get stuck or slow down beyond our expectations. With a combination of this and a global rate limit / throttle for the luci.chromium.try bucket (see bug 812021) we should be able to calculate an upper bound on the number of builders needed to handle the load before we start rejecting load and thus be able to do better capacity forecasting.
,
May 22 2018
To the extent that try builders are mirroring CI builders, this means that the matching CI builders need to have timeouts as well. We also need to ensure that these builders are not sharing resources with builders that don't have timeouts (e.g., they shouldn't share "optional" bot pools with FYI builders).
,
Aug 29
Is this still needed? Dirk, can you comment? Note that this is a blocking bug for cit-pm-84. Thanks!
,
Aug 29
,
Aug 29
Yes, it's still needed.
,
Oct 2
all builders in luci.chromium.try DO have a timeout of 3h https://chromium.googlesource.com/chromium/src/+/b88a4a829fc69bc26315dceb4b8090ac09e65b2a/infra/config/global/cr-buildbucket.cfg#2757 did you mean a smaller timeout?
,
Oct 2
I don't think buildbot builders has the equivalent timeouts, so if this is fixed by moving to LUCI, then I think that's good enough for purposes of this bug. Most of the builders should probably have shorter timeouts, but that could easily be a follow-on issue. |
|||
►
Sign in to add a comment |
|||
Comment 1 by dpranke@chromium.org
, May 22 2018