Ryan or Robbie, I think you may know buildbot internals enough to decide on the feasibility of some of these.

Personally, I think if we can limit loading of builds to the last ~50-100 that will mitigate this enough but I'm curious what you think.

Notably, I believe that the number of pickles loaded is "limited" to 200 per builder. The problem is that it has so many builders that that number becomes unreasonably large.

They are loaded on-demand, but the demand is "I want to view the Console page", which immediately requires loading builds from every builder. It's not clear to me why it loads 200 builds for every builder, instead of loading them until it reaches 25 revisions worth of builds or whatever it is that the console displays, but it's still all triggered by just a single pageload. (Loading the root json also triggers the same load-from-disk behavior.)

Without knowing the specifics, i know historically we increase the limit because of runtime issues, so any decrease should be done carefully and with an acknowledgement that we may be trading startup time for runtime performance.

I'm inclined not to put too much work into this since Dirk will be splitting up the offending masters. We should try to put checks in place to make sure masters don't get excessive numbers of builders or bots since this isn't the only aspect of buildbot that doesn't scale.