Here's a provision job that should've marked the DUT as provision_failed but didn't: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos4-row10-rack9-host15/1462-provision/20170404124548/

Looks like the provision was trying to actually provision the ancient (missing) build onto the DUT, not a new one:

04/04 12:46:28.426 DEBUG|provision_AutoUpda:0059| Start provisioning <remote host: chromeos4-row10-rack9-host15> to quawks-release/R54-8743.44.0

This is because in my cl
https://chromium-review.googlesource.com/c/468087/
I didn't realize that test_push uses the http://shortn/_lGCoafFiyf stable_version board for testing as the target version. Worse, it asks the push_master what the stable version is.

So, for quawks, that is:
pprabhu@pprabhu:~$ atest stable_version list -w chromeos-autotest.hot 
==========================================================
board            | version
----------------------------------------------------------
beaglebone_servo | R55-8866.0.0
gandof           | R57-9202.18.0
angler           | git_oc-release/angler-userdebug/3790152
==========================================================

missing from the DB, hence global_config.ini value:
stable_cros_version: R54-8743.44.0


There is still the bug that it blew up before marking the DUT as provision failed.
Two problems.
(1) It blew up here: http://shortn/_XzkdOygznn before it ever got to machine_install{,_by_devserver}
(2) machine_install_by_devserver actually never marks the DUT correctly. Is that expected?

I changed the stable_version DEFAULT on the push master to chromeos-test@chromeos-autotest:~$ /usr/local/autotest/cli/atest stable_version list | grep DEFAULT
DEFAULT          | R58-9334.28.0


None of the underlying bugs here are fixed yet.