After some digging, the test_that.DEBUG file has the interesting bits:
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/rambi-paladin/R54-8613.0.0-rc1/vm_test_results_2/test_harness/failed/SimpleTestUpdateAndVerify/2_autotest_tests/debug/

From what I understand, cros_au_test_harness.py kicks off test_that on the VM.  It'll look for a successful run of smoke:suite (rc=0) and if it fails in anyways, it'll mark the stage as a failure.  The confusing part is the summary reports all tests pass, digging into the debug log shows which tests failed.

Search for the line "chromite.lib.timeout_util.TimeoutError: Timeout occurred- waited 300 seconds"

Two tests failed with that:
- desktopui_ScreenLocker
- login_LoginSuccess

Nice how the summary leaves both of those tests out.  Probably because they error'd out instead of flat out fail.


In anycase, we suspect the problem was replacing the existing AFE with RetryingAFEs, I have a cl up that reverts those changes (https://chromium-review.googlesource.com/#/c/362021/) and a tryjob to see if that will help (https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/1195).  If that tryjob succeeds, we'll chump the change in.

The tryjob succeeded so it looks like the fix works, but why does the fix work? 

¯\_(ツ)_/¯