I've done a spot check of all the repair tasks on fizz DUTs
in the bvt pool.  There's a lot of them: in a 72 hour period,
there was a total of 63 repair events; that's enough to average
every DUT failing at least once on every release builder run.

Below are logs of a prototypical event:
    https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row4-rack5-host7/61981800-repair

The attached "status.log" file tells the tale.  The key is this part:
	FAIL	----	verify.ssh	timestamp=1513636301	localtime=Dec 18 14:31:41	No answer to ping from chromeos2-row4-rack5-host7
	START	----	repair.rpm	timestamp=1513636301	localtime=Dec 18 14:31:41	
		GOOD	----	verify.ssh	timestamp=1513636349	localtime=Dec 18 14:32:29	
		GOOD	----	verify.power	timestamp=1513636349	localtime=Dec 18 14:32:29	
	END GOOD	----	repair.rpm	timestamp=1513636349	localtime=Dec 18 14:32:29	

The "verify.ssh" line says that the DUT was offline.  The "repair.rpm"
action means that the system used an RPM device to unplug/replug AC
power to the DUT.  The logs show that power cycling AC caused the DUT
to boot up and return to working order.

The code for "repair.rpm" looks for and gathers crash dumps, if they're
found.  The logs show no dumps, so it looks like whatever caused the
problems, there were no crashes.

Deleted: status.log 5.5 KB

status.log 5.5 KB View Download

This smells like a system hang, so let's give it to the kernel.

Assigning to a sheriff to find a proper expert.

Haven't seen any more recurrences of this problem and it's been over a month without any updates.  Closing.