Tune the BuildbucketProdTooManyPendingTryjobs alert |
||
Issue descriptionSee bug 739556 for the context; we started building up pending builds on the ios-device, ios-device-xcode-clang, and ios-simulator-xcode-clang bots at around 08:00 in the morning on 7/5, but the alert didn't fire until 17:00, at which point we had ~500 pending jobs across the three builders (each of which had only 10 bots in a pool). A useful alert should've fired much much earlier; we probably need to figure out how to change the alerts to be sensitive to the # of jobs we'd expect to have and/or the number of bots that are available in a given pool. shortn/_Sa2yb9s13N for the link to the pending queues in Viceroy, and o/e/m2546809710000039 for the alert that fired way too late :).
,
Jun 2 2018
Friendly ping. This is a blocking bug on cit-pm-55. Please update need and priority accordingly.
,
Aug 29
http://shortn/_TZTjWKNXlJ is the proper alert for this, which will soon fire per builder when there is a job pending >= 1h. We'll be tightening the threshold in the (hopefully near) future, as our fleet capacity improves to avoid perpetual pending queues. |
||
►
Sign in to add a comment |
||
Comment 1 by dpranke@chromium.org
, Jul 12 2017