Repair results: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos4-row13-rack8-host10/2628323-repair/20172602123804/debug/

status.log:

START	----	repair	timestamp=1488141493	localtime=Feb 26 12:38:13	
	GOOD	----	verify.ssh	timestamp=1488141495	localtime=Feb 26 12:38:15	
	GOOD	----	verify.brd_config	timestamp=1488141496	localtime=Feb 26 12:38:16	
	GOOD	----	verify.ser_config	timestamp=1488141496	localtime=Feb 26 12:38:16	
	GOOD	----	verify.job	timestamp=1488141497	localtime=Feb 26 12:38:17	
	GOOD	----	verify.servod	timestamp=1488141502	localtime=Feb 26 12:38:22	
	GOOD	----	verify.pwr_button	timestamp=1488141502	localtime=Feb 26 12:38:22	
	GOOD	----	verify.lid_open	timestamp=1488141503	localtime=Feb 26 12:38:23	
	GOOD	----	verify.update	timestamp=1488141506	localtime=Feb 26 12:38:26	
	GOOD	----	verify.PASS	timestamp=1488141506	localtime=Feb 26 12:38:26	
	GOOD	----	verify.ssh	timestamp=1488141516	localtime=Feb 26 12:38:36	
	GOOD	----	verify.power	timestamp=1488141516	localtime=Feb 26 12:38:36	
	GOOD	----	verify.fwstatus	timestamp=1488141516	localtime=Feb 26 12:38:36	
	FAIL	----	verify.good_au	timestamp=1488141516	localtime=Feb 26 12:38:36	Last AU on this DUT failed
	START	----	repair.firmware	timestamp=1488141516	localtime=Feb 26 12:38:36	
		FAIL	----	repair.firmware	timestamp=1488141516	localtime=Feb 26 12:38:36	Firmware repair is not applicable to host chromeos4-row13-rack8-host10.
	END FAIL	----	repair.firmware	timestamp=1488141516	localtime=Feb 26 12:38:36	
	GOOD	----	verify.writable	timestamp=1488141517	localtime=Feb 26 12:38:37	
	GOOD	----	verify.tpm	timestamp=1488141517	localtime=Feb 26 12:38:37	
	GOOD	----	verify.ext4	timestamp=1488141518	localtime=Feb 26 12:38:38	
	GOOD	----	verify.rwfw	timestamp=1488141519	localtime=Feb 26 12:38:39	
	GOOD	----	verify.python	timestamp=1488141519	localtime=Feb 26 12:38:39	
	GOOD	----	verify.cros	timestamp=1488141523	localtime=Feb 26 12:38:43	
	START	----	repair.powerwash	timestamp=1488141523	localtime=Feb 26 12:38:43	
		START	----	reboot	timestamp=1488141523	localtime=Feb 26 12:38:43	
			GOOD	----	reboot.start	timestamp=1488141523	localtime=Feb 26 12:38:43	
			GOOD	----	reboot.verify	timestamp=1488141562	localtime=Feb 26 12:39:22	
		END GOOD	----	reboot	kernel=3.10.18	localtime=Feb 26 12:39:23	timestamp=1488141563	
		GOOD	----	verify.ssh	timestamp=1488142081	localtime=Feb 26 12:48:01	
		GOOD	----	verify.tpm	timestamp=1488142082	localtime=Feb 26 12:48:02	
		GOOD	----	verify.good_au	timestamp=1488142083	localtime=Feb 26 12:48:03	
		GOOD	----	verify.ext4	timestamp=1488142083	localtime=Feb 26 12:48:03	
		GOOD	----	verify.power	timestamp=1488142083	localtime=Feb 26 12:48:03	
		GOOD	----	verify.rwfw	timestamp=1488142084	localtime=Feb 26 12:48:04	
		GOOD	----	verify.python	timestamp=1488142084	localtime=Feb 26 12:48:04	
		GOOD	----	verify.cros	timestamp=1488142088	localtime=Feb 26 12:48:08	
	END GOOD	----	repair.powerwash	timestamp=1488142088	localtime=Feb 26 12:48:08	
	GOOD	----	verify.PASS	timestamp=1488142088	localtime=Feb 26 12:48:08	
	INFO	----	repair	timestamp=1488142088	localtime=Feb 26 12:48:08	Can't repair label 'board:nyan_kitty'.
	INFO	----	repair	timestamp=1488142088	localtime=Feb 26 12:48:08	Can't repair label 'cros-version:nyan_kitty-paladin/R58-9320.0.0-rc1'.
	INFO	----	repair	timestamp=1488142088	localtime=Feb 26 12:48:08	Can't repair label 'pool:cq'.
END GOOD	----	repair	timestamp=1488142088	localtime=Feb 26 12:48:08	chromeos4-row13-rack8-host10 repaired successfully

https://luci-milo.appspot.com/buildbot/chromeos/veyron_minnie-paladin/1789

		FAIL	provision_AutoUpdate	provision_AutoUpdate	timestamp=1488232182	localtime=Feb 27 13:49:42	Unhandled DevServerException: CrOS auto-update failed for host chromeos4-row9-rack9-host1: RootfsUpdateError: Failed to perform rootfs update: RootfsUpdateError('Update failed with unexpected update status: UPDATE_STATUS_IDLE',)
  Traceback (most recent call last):
    File "/usr/local/autotest/client/common_lib/test.py", line 804, in _call_test_function
      return func(*args, **dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 461, in execute
      dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
      postprocess_profiled_run, args, dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 376, in _call_run_once
      self.run_once(*args, **dargs)
    File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once
      force_full_update=force)
    File "/usr/local/autotest/server/afe_utils.py", line 232, in machine_install_and_update_labels
      *args, **dargs)
    File "/usr/local/autotest/server/hosts/cros_host.py", line 728, in machine_install_by_devserver
      full_update=force_full_update)
    File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 2013, in auto_update
      raise DevServerException(error_msg % (host_name, error_list[0]))
  DevServerException: CrOS auto-update failed for host chromeos4-row9-rack9-host1: RootfsUpdateError: Failed to perform rootfs update: RootfsUpdateError('Update failed with unexpected update status: UPDATE_STATUS_IDLE',)
	END FAIL	provision_AutoUpdate	provision_AutoUpdate	timestamp=1488232182	localtime=Feb 27 13:49:42	

I experienced a similar issue (http://cautotest/afe/#tab_id=view_job&object_id=106467907).
I selected a host "By specifying host labels", and choose veyron (platform) label.
Even with that label a reef DUT was selected, leading a to a provision error.

Using label: board:veryon_minnie works.

For DUTs going to status 'UPDATE_STATUS_IDLE' which causes provision failure, I don't have a clear solution for it. 

Temporarily mark it as wontfix. Reopen it if we meet the same problem frequently in the  future.

Issue 650520 has been merged into this issue.

Appeared again:

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/15188

This may qualify for Chase-Pending since t's caused failures.
However, it's not clear whether there's an easy thing we can
do to address this, or at least move it down the field.

The biggest problem is that it's not clear whether the specific
failure cited in c#7 is infrastructure, a latent product bug, or
a bad CL.  Frankly, none of those explanations hold water, but
one of them must be it.

So, my recommendation:
 1) Look at the update_engine logs from the failure, to see
    what they say, and
 2) Assuming 1) is inconclusive, add more logging somewhere
    in provisioning so that we'll know more the next time
    we see this.

P2, since the ask is for preventative measures against CQ flake.

Likely not a Chase bug, but xixuan@ can investigate. If this can be boiled down to a real Chase-Pening bug or dedupe this then we can reconsider for Chase.

jrbarnette@ is asking for a link to update_engine logs for this failure

Here is an example!

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/125377657-chromeos-test/chromeos4-row6-rack11-host4/

Could @jrbarnette or @dgarrett have a check and update the process of how to check in this bug?

 Issue 769990  is potentially a dup here.

Here's some recent occurrences (from the last 200 CQ runs, cyan provision failures):
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host3/1587042-provision
gs://chromeos-autotest-results/hosts/chromeos4-row12-rack11-host3/1587043-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host12/1587044-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host7/1587045-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host13/1587046-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host14/1587047-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host7/1585511-provision
gs://chromeos-autotest-results/hosts/chromeos4-row12-rack11-host3/1585512-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host14/1585513-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host13/1585514-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host12/1585515-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host3/1585516-provision
gs://chromeos-autotest-results/hosts/chromeos4-row11-rack11-host9/1585517-provision
gs://chromeos-autotest-results/hosts/chromeos4-row11-rack11-host9/1583963-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host17/1583964-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host12/1583965-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host14/1583966-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host13/1583967-provision
gs://chromeos-autotest-results/hosts/chromeos4-row12-rack11-host3/1583968-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host7/1583969-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host3/1583970-provision
gs://chromeos-autotest-results/hosts/chromeos4-row12-rack11-host3/1528936-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host12/1454447-provision
gs://chromeos-autotest-results/hosts/chromeos4-row6-rack9-host3/1372841-provision