In my short lived experience as long as there is hwtest failures, we cannot be sure paygen tests are caused by autoupdate :)

Given the current state of the builder, the Canary/Dev tests are basically identical.

I believe we have a lab problem, probably related to an isolated set of DUTs. Looking further....

Looking at the last 4 builds, the only DUTs mentioned are:

 chromeos2-row4-rack1-host7
 chromeos2-row4-rack1-host9
 chromeos2-row4-rack1-host10

They are all on the same rack.

The DUTs are provisioning and running tests, I don't understand what's happening.


Passing to an expert in DUT analysis.

There's a known problem with Eve DVT hardware, see bug 755060.
There was an initiative to rework the problem children, but I
can't find any evidence that those particular DUTs were ever
reworked.  So, there's a respectable chance that the failures
here are the known problem with VCCIO overshoot.

Passing to tbroch@ to answer this question:  How can we determine
whether the DUTs have the rework?

<sigh> :
    $ atest host list chromeos2-row4-rack1-host7 chromeos2-row4-rack1-host9 chromeos2-row4-rack1-host10 | count_labels -l phase
          3 EVT

The three problem devices aren't even DVT devices:  They're EVT
devices.  Lord only knows what sort of problems are expected.

At this time, there's a mix of EVT, DVT, _and_ PVT devices all
in the lab:
    $ atest host list -b board:eve | count_labels -l phase
         14 DVT
         12 EVT
         26 PVT

Really, that's just too much.

https://uberchromegw.corp.google.com/i/chromeos/builders/eve-release/builds/1033

chromeos2-row8-rack1-host11 :: needs vccio rework

234 | 2017-10-22 21:40:02 | Kernel Event | Clean Shutdown
235 | 2017-10-22 21:40:08 | System boot | 1484
236 | 2017-10-22 21:40:08 | Last post code in previous boot | 0x98 | Unknown
237 | 2017-10-22 21:40:08 | System Reset
238 | 2017-10-23 06:50:21 | System boot | 1485
239 | 2017-10-23 06:50:21 | Power Fail
240 | 2017-10-23 06:50:21 | ACPI Wake | Deep S5

	FAIL	150972698-chromeos-test/chromeos2-row8-rack1-host11/provision_AutoUpdate	provision	timestamp=1508765703	localtime=Oct 23 06:35:03	Unhandled AutoservSSHTimeout: ('ssh timed out', * Command: 

builds/1032-1029 :: same host chromeos2-row8-rack1-host11

Locked host for VCCIO rework or removal.


Duts in #3


chromeos2-row4-rack1-host7 -- has rework so should monitor
chromeos2-row4-rack1-host9 -- needs rework ... locked
chromeos2-row4-rack1-host10 -- needs rework ... locked

chromeos2-row8-rack1-host11 has been reworked.  Lets monitor 

https://uberchromegw.corp.google.com/i/chromeos/builders/eve-release/builds/1035

build #1035 passed.