New issue
Advanced search Search tips

Issue 740306 link

Starred by 2 users

Issue metadata

Status: Duplicate
Owner:
Closed: Aug 29
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug



Sign in to add a comment

Tune the BuildbucketProdTooManyPendingTryjobs alert

Project Member Reported by dpranke@chromium.org, Jul 8 2017

Issue description

See bug 739556 for the context; we started building up pending builds on the ios-device, ios-device-xcode-clang, and ios-simulator-xcode-clang bots at around 08:00 in the morning on 7/5, but the alert didn't fire until 17:00, at which point we had ~500 pending jobs across the three builders (each of which had only 10 bots in a pool).

A useful alert should've fired much much earlier; we probably need to figure out how to change the alerts to be sensitive to the # of jobs we'd expect to have and/or the number of bots that are available in a given pool.

shortn/_Sa2yb9s13N for the link to the pending queues in Viceroy, and o/e/m2546809710000039 for the alert that fired way too late :).

 
Labels: cit-pm-57

Comment 2 by efoo@chromium.org, Jun 2 2018

Friendly ping. This is a blocking bug on cit-pm-55. Please update need and priority accordingly. 
Mergedinto: 873754
Status: Duplicate (was: Assigned)
http://shortn/_TZTjWKNXlJ is the proper alert for this, which will soon fire per builder when there is a job pending >= 1h. We'll be tightening the threshold in the (hopefully near) future, as our fleet capacity improves to avoid perpetual pending queues.

Sign in to add a comment