It's been suggested as part of bug 669297 that we simply have too many (>600?) slaves assigned to the master, and it isn't handling this well, causing its performance to fall off a cliff.

In theory we shouldn't need anywhere near this many slaves at the moment. I'm filing this bug to track reducing the number of slaves to 500 or 400 to see if that helps.

We should also dig into the implementation of scheduling in buildbot to see if there are any particular patterns to how buildbot is polling and scheduling things that might cause things to get starved ...