I'm aware of this problem.
I'd like to say that buildslave maintenance is a separate concern from builder maintenance, and the latter is more importantly. I recently filed a bug to detect when a builder is compromised because of this / other issues:  issue 819419 

chrome-infra-labs should generally take care of buildslaves based on their own prirorities. For -c2 builders, we should probably have a separate administrative process as well, to monitor / manage the pool health.

Let's keep this bug for that.

Not currently on my plate.

https://uberchromegw.corp.google.com/i/chromeos/builders/eve-release

cros-beefy59-c2	offline in buildbot, online in GCP.  Will reboot and see.

Not sure what the semantics of "blocking" is

I don't currently have a builder in this state (just rebooted them all), but it is an ongoing issue.

dgarrett: are you expecting some amount of trooper intervention here?

Yes. We don't normally own or interact with the buildbot client.

I consider this lower priority than it was because we have fewer buildbot builders than we used too, but it does still cause build failures.

If you want to hand it back until we have another stuck builder, that would be very reasonable.

If you don't mind, I'll assign the bug to you Don, otherwise we keep on looking at triaging the issue.

We currently have two examples of this. 

https://uberchromegw.corp.google.com/i/chromeos/buildslaves/build173-m2
https://uberchromegw.corp.google.com/i/chromeos/buildslaves/cros-beefy71-c2


The first is associated with the CQ which has 2 redundant slaves, the second is causing an outage of veyron_jaq-release, so I'm going to capture logs and reboot it.

Actually, those appear to be symptoms appear to be different, I can't ssh into either machine.

As the number of buildbot builders goes down, this has been less of an issue.