Hi! I think that this is because we're trying to upgrade the builders to Linux Trusty ( bug 660580 ) -- if so, this should be a one-off failure - sorry about this. Is there somewhere I should announce this?

Can we instead gracefully drain the builders and then rolling upgrade to the new version of linux instead of causing spurious failures and retries?

That would be better, if possible :-/

I'm not sure the best way for this to be done in general, and having the slaves wait until the current build is finished might be more troublesome (I think) and would also increase the amount of time where different slaves are using a different OS version.

Does anyone know if there's a better way, or if there's documentation about how groups of slaves have been upgraded in the past?

We can't easily drain things, no. Life will be rocky instead. Doing painless bulk upgrades the builders from one version of the operating system to the next is currently an unsolved problem in chrome-infra-land.

The alternative would be to just shut down the trees, but that alternative is worse than some failed builds.

However, we should send a PSA to chromium-dev telling people about this (ASAP).

Alright, now sending PSA to chromium-dev

Alright, closing now since the cause (rebooting groups of slaves for upgrading) is now finished.