chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: sudo -- vgchange -an cros_b+swarming+w+ir+cache+cbuild+repository+chroot_000
  Can't deactivate volume group "cros_b+swarming+w+ir+cache+cbuild+repository+chroot_000" with 1 open logical volume(s)

It looks like something is keeping the mount pinned.  A process left behind inside the chroot maybe?  Do these get rebooted in between builds?

They are supposed too. How long has it been up?

It looks like there's a leftover samus-compile-only-pre-cq from this morning that has the chroot pinned.  That suggests the machine isn't rebooting properly.

Here's the dead samus build: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8944267109024583552

I believe that the sequence is that the linked build was terminated, leaked a process, but didn't reboot as expected.

That leaked process held on to resources that caused later builds to fail before they started.

Not rebooting after the termination doesn't surprise me, since that's a new feature, but I thought we expected builders to reboot after a failed build.
That would have caused this builder to self-repair.

But that doesn't seem to be what's happening, on any of the builders I check. They reboot about once a day, but not after tasks complete pass or fail.

Ah, it's my fault. I told Xixuan to use 'ChromeOS' prefix but it should have been 'ChromeOSSkylab' at this CL:
https://chrome-internal-review.googlesource.com/#/c/636748/


The fix is to fix the string from 'ChromeOS' to 'ChromeOSSkylab'.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/2806094cf2ca95397b33e3198c1319083cd9fb9e

commit 2806094cf2ca95397b33e3198c1319083cd9fb9e
Author: Don Garrett <dgarrett@google.com>
Date: Mon Jun 11 18:24:49 2018

Believed fixed, looking for an example to confirm.

ChromeOS builders are now rebooting between builds again.