Yesterday I tried to assess whether a specific CL reduced YouTube power consumption but discovered that the CL (submitted a day prior) still hadn't been tested on the perf bots. A little more digging showed that the root cause was that the Mac builders were something like 50-75 CLs behind, whereas a normal builder might be 3-5 CLs behind. The even rootier cause was that there were only 7 Mac buildslaves, whereas there were something like 20 Linux and Windows buildslaves.

After asking about this on the speed infra chat, martiniss@ was very helpful in getting more builders online by the end of the day. The builder is now catching up.

I'm thankful that the problem was resolved so quickly, but am wondering what next steps we can take to ensure that this doesn't happen again, as it's definitely disruptive and makes perfbot health sheriffing more difficult. Would it be possible to add monitoring on the builder, along the lines of:

if (time_to_build_each_cl * cls_to_build / number_of_buildslaves) > 2 hours:
  alert('Builder %s is behind' % platform)

It seems like, without this type of monitoring in place, it's a near inevitability that it'll happen again.