Well, I guess it's good news that the detection is working? Ken, how do you feel about moving forward with having the Chrome-GPU pool quarantine the affected machines? I can get a patch out in a few minutes.

Let's get the auto-quaranting in place ASAP!

I hope the detection code is accurate. We'll need to monitor the fleet to make sure that we don't accidentally auto-quarantine all the bots.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/086e43efd1bb7fc5a6e8e5cbf6b010c71def85b2

commit 086e43efd1bb7fc5a6e8e5cbf6b010c71def85b2
Author: Brian Sheedy <bsheedy@google.com>
Date: Fri Sep 21 19:48:34 2018

Are you still seeing this? Any devices on Chrome-GPU should automatically be quarantined now instead of catching this at test run time, although I'm not seeing any that have gotten into that state yet.

No, I didn't see any new failures yet.

We can try bringing build706-m4 back online and see if it will be quarantined.

Sure, that would be a good test.

I've rebooted it, and now it's quarantined!
https://chromium-swarm.appspot.com/bot?id=build706-m4

Great, that tells us that the quarantining is indeed working, and that we shouldn't have to worry about it quarantining a large portion of the fleet accidentally since that's the only one that's been caught so far.

I'll keep monitoring for a bit longer and keep the other but open in the meantime, but I think that means this one is fixed.

Fantastic! Superb work Brian and thanks Yuly for reporting the problem and helping verify it!