This is ongoing, in fact eve-release hasn't had a successful build in a week.

xixuan@, can you take a look? It seems the provision failed to deploy a correct version?

@david, could you check this paygen_au_dev_full failure? seems sth wrong with verifying the hostlog.

This is causing Simple Chrome compile failures since there is not a recent LATEST file for anything > 9914.0.0 and there was a breaking GN configuration change post 9914.0.0.

Given the importance placed on eve, this is blocking a lot of devs.

Escalating priority on this.

There are a few different issues in this bug. Regarding this failure:

FAIL: The update appears to have completed successfully but we found a problem while verifying the hostlog of events returned from the update. Some attributes reported for the initial update check event are not what we expected: ['version']. The expected version is (9887.0.0) but reported version was (9914.0.0). The source payload we installed was probably incorrect or corrupt. Check the full hostlog for this update in the devserver_hostlog_rootfs file in the autoupdate_logs directory.

This can happen when we update rootfs from X -> Y. Then applying stateful fails. 
We retry to update to Y but since the rootfs was successfully updated we update from Y -> Y. We return the hostlog from this update which is not what the test expects. I have updated the error message for when this happens in the refactor CL:

https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/654064/

For eve though, the most common failure reason for paygen_au_canary and paygen_au_dev lately is this one:

Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row4-rack1-host10: RootfsUpdateError: After update and reboot, update-engine failed to call chromeos-setgoodkernel within 120 seconds

Which can happen if an update failed or it failed to apply the source image in the test. Checking the logs for eve...

and this failure reason too:
Unhandled DevServerException: CrOS auto-update failed for host <DUT HOSTNAME>: SSHConnectionError: ssh: connect to host <DUT HOSTNAME> port 22: Connection timed out

Both of these failures are because we try to apply the source image in the autoupdate test and the device doesn't come back to life after we apply stateful. So it times out with ssh or applying chromeos-setgoodkernel, whatever comes first. 

So it seems like the test is behaving correctly and there is a problem with the eve images and/or stateful lately. 

Not sure who to assign this to. ahassani@ do you know if there was any changes related to eve here?

Eve has been flaky since Aug 11. Looks like there was a change to the eve touch firmware in the first failed build: https://crosland.corp.google.com/log/9831.0.0..9832.0.0

Could this be a culprit, jkwang@?

Could not think of any reason that the fw update will trigger this.

look at: crbug.com/761259

One failure is fixed by #7 another is fixed by #12 so closing this