A plausible answer would be to run a reset/verify job prior
to running provisioning.  I think it's subtler than that,
in that some verify checks could maybe be skipped if the
next action will be provision.

That's a non-trivial change to the scheduler, and out of scope
for me right now.

Kevin maybe you can help explain this?


Here's the sequence of jobs on this DUT:

2017-02-23 23:54:35 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/782681-provision/
2017-02-23 23:31:41 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/782616-repair/
2017-02-23 23:23:54 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/782583-provision/
2017-02-23 16:17:00 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/103052397-chromeos-test/
2017-02-23 16:16:40 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/781907-reset/
2017-02-23 15:54:58 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/781765-repair/
2017-02-23 15:52:37 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/781758-reset/
2017-02-23 15:50:08 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/103049700-chromeos-test/
2017-02-23 15:40:21 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row7-rack3-host13/781739-provision/

I don't understand this output.  I see either "OK" or "--" as the result (? I assume) of a provision.  So "--" would mean the provision was not OK, i.e. it failed?  But for ###-chromeos-test I only see "--" results (even on other DUTs).  Is the meaning different?

In any case, here we have (order reversed):

- successful provision at 15:40
- test at 15:50.  Failed?  That would explain the immediately following:
-- reset (failed?)
-- repair OK
-- reset OK
-- another test (OK?)

and then nothing happens until 2323

- provision (fails)
- repair (works)
- provision (works)

Unfortunately /var/log/messages from the last repair doesn't seem to include the time of the previous failed provision.  It starts like this:

2017-02-24T00:24:08.856660+00:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset

and stays up until autotest reboots it:

2017-02-24T00:28:34.804452+00:00 NOTICE ag[5206]: autotest server[stack::run_op|reboot|run_background] -> ssh_run(( sleep 1; reboot & sleep 10; reboot -f ) </dev/null >/dev/null 2>&1 & echo -n $!)

then the clock goes back 1 hour (I've never seen that)

2017-02-23T23:54:00.434981-08:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset

and that's already too late.

May be part of Richard's provision re-write

Re c#2, the questions seem not entirely relevant to this bug.
For more details on how to interpret the output from dut-status,
try this:
    vi $(which dut-status)

The docstring has answers to the questions.

Re c#3:  I'm strongly considering implementing the suggestion
from c#1.  That is, schedule a verify/reset task prior to
provisioning.  That would prevent provisioning from running
on an offline DUT in most cases.  Done right, it could also
alleviate problems with "Provision failures are recorded as
test failures".

Status?

The requested behavior would be at least partly addressed as a
side-effect of this design proposal:
    https://docs.google.com/document/d/1Nr8qbqJlVm7eMeKaA6bgecRGsgfJfC1TaybpWTmDZvg/edit

See, specifically, the provisioning step entitled "Prepare the host".