Xixuan - can you take a look? Sounds provision related. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ec801e3a36456e5dc6e1d1ed665e575fd6713787

commit ec801e3a36456e5dc6e1d1ed665e575fd6713787
Author: xixuan <xixuan@google.com>
Date: Thu Aug 25 17:20:22 2016

Autotest: Add host class check, and change parameter of
machine_install_by_devserver.

This CL handles two cases:
1. if the DUT has job_url, and pass it into provision_AutoUpdate test as
update_url, the machine_install_by_devserver func will fail since it does
not have a parameter 'update_url'.
2. Only trigger devserver to do auto-update if the host is a CrosHost.

BUG= chromium:638728 
TEST=Run /usr/local/autotest/server/autoserv -p -r
/tmp/chromeos2-row3-rack3-host16-16-provision -m chromeos2-row3-rack3-host16
--verbose --lab True --provision --job-labels
cros-version:gandof-release/R54-8630.0.0 on hot.
Run repair/provision_AutoUpdate.double on local autotest.

Change-Id: Ia411f91727516b61e8a9af69b56b1759115f2031
Reviewed-on: https://chromium-review.googlesource.com/375939
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/ec801e3a36456e5dc6e1d1ed665e575fd6713787/server/hosts/cros_host.py
[modify] https://crrev.com/ec801e3a36456e5dc6e1d1ed665e575fd6713787/server/afe_utils.py
[modify] https://crrev.com/ec801e3a36456e5dc6e1d1ed665e575fd6713787/server/hosts/base_classes.py

Seems like same failure is seen in M-54 -> https://wmatrix.googleplex.com/failures/bvt?tests=provision_AutoUpdate.double&hide_experimental=True&releases=54

Xixuan, Do we need to merge to M-54?

I believe so. Seems the package don't have the newest autotest code.

https://chromium-review.googlesource.com/#/c/391221/, This CL is created for merge the fix to R54.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6bd68d2bb4950257ecf6d75016cd8f2c611fb9a9

commit 6bd68d2bb4950257ecf6d75016cd8f2c611fb9a9
Author: xixuan <xixuan@google.com>
Date: Thu Aug 25 17:20:22 2016

Autotest: Add host class check, and change parameter of
machine_install_by_devserver.

This CL handles two cases:
1. if the DUT has job_url, and pass it into provision_AutoUpdate test as
update_url, the machine_install_by_devserver func will fail since it does
not have a parameter 'update_url'.
2. Only trigger devserver to do auto-update if the host is a CrosHost.

BUG= chromium:638728 
TEST=Run /usr/local/autotest/server/autoserv -p -r
/tmp/chromeos2-row3-rack3-host16-16-provision -m chromeos2-row3-rack3-host16
--verbose --lab True --provision --job-labels
cros-version:gandof-release/R54-8630.0.0 on hot.
Run repair/provision_AutoUpdate.double on local autotest.

Change-Id: Ia411f91727516b61e8a9af69b56b1759115f2031
Reviewed-on: https://chromium-review.googlesource.com/375939
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>
Reviewed-on: https://chromium-review.googlesource.com/391221
Commit-Queue: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/6bd68d2bb4950257ecf6d75016cd8f2c611fb9a9/server/hosts/cros_host.py
[modify] https://crrev.com/6bd68d2bb4950257ecf6d75016cd8f2c611fb9a9/server/afe_utils.py
[modify] https://crrev.com/6bd68d2bb4950257ecf6d75016cd8f2c611fb9a9/server/hosts/base_classes.py

Issue 652145 has been merged into this issue.

Issue 645804 has been merged into this issue.

Issue 645760 has been merged into this issue.

Is this flakey? I saw another occurence during the recent push to prod, see http://chromeos-drone2-staging.hot.corp.google.com/results/27397-chromeos-test/

My guess is that this is fixed in that branch, but at an older release than the one being used for this provision autoupdate test. Looks like the cherry-pick CL landed in 8765, but the provision autoupdate double test is running is 8743.

jrbarnette@, shuqianz@, xixuan@, what sets the image that we are using for this provision_AutoUpdate and why is it so out of date?

There is a script named 'assign_stable_images.py' that running every day I think to update the stable_images for all the boards. It will update the stable image to most recent version on the *Bet channel. jrbarnette@ knows more about this script.

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/site_utils/assign_stable_images.py?q=assign_sta&sq=package:%5Echromeos_public$&l=164

However, this 'update_url' issue happened on last Friday too. I tried R54, R55 builds, both didn't have the fix. Then xixuan@ pushed the fix to both R54 and R55 on Friday evening. I think the fix hasn't got a chance to built into a beta image yet, so the push failed. It is just my guess.

Beta channel, sorry for the typo

This is still failing in latest M54 across multiple devices 
https://wmatrix.googleplex.com/unfiltered?suites=bvt-inline&builds=R54-8743.65.0

bvttriage is the right label.

 Issue 657017  has been merged into this issue.

Any update on this? we are seeing lots of  error "has reached AU retry limit"

See full list in
https://cros-goldeneye.corp.google.com/chromeos/console/qaRelease?releaseName=M55-BETA-CHROMEOS-5

The original error in this bug has not reappeared. However the bug that was marked as a duplicate in #21 has been going on a long time. The "has reached AU retry limit" error.

We are seeing this test failure with "has reached AU retry limit" on all boards on M55 and M54. M56 and M57 look all green

If you look at both of results links below you can see that everything started failing on October 13th and has failed on every build since

M55:
https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=provision_AutoUpdate.double&releases=55&days_back=60

M54:
https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=provision_AutoUpdate.double&releases=54&days_back=90

Did something change around that date? 

btw, feel free to close this bug and reopen  issue 657017  if that is easier

It's fixed in a new branch, but may still happens in version older than 8765. 

So if we can have enough time to let these old-version go, this would be better.

Since this test is in bvt-inline, it failing means the out of band suites like CTS and GTS are not automatically run on stable builds. This is delaying the release process. Is this an easy fix?

Cherry-pick https://chromium-review.googlesource.com/#/c/391221/ this CL and apply it to the oldest branch you can find in the current test.

*oldest failed build version/branch.

The failing builds (and branch) are newer than 8765. They are in the 8872 branch:

https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=provision_AutoUpdate.double&releases=55&days_back=60

As mentioned in #24, the original error in this bug is not reproduced and it is a new error message: "AU retry limit".

The cherry pick you linked to in #28 was to the M54 branch. it fixed the original error in this bug but the new "AU retry limit" error message started then. 

It's real provision error, just not be shown as expected. 

https://chromium-review.googlesource.com/#/c/393886/ cleans this types of failure and show the right errors and traceback.

https://chromium-review.googlesource.com/#/c/394268/ move the auto-update logs from sysinfo to autoupdate_logs.

Since these are old branches, the auto-update logs are still in sysinfo, whose name is like 'CrOS_update_chromeos2-row6-rack7-host22_1070.log'. If you see 'reach retry limit' error, go to the log directory and check such CrOS_update log in it. This will help to find why the provision fails (https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/provisioning_debug_instruction, change the log dir from log_dir/autoupdate_logs/ to log_dir/sysinfo/ for old branches).


Because that lots of CLs are not merged into very old branches, the older the branch is, the bigger possibility that provision fails, and the harder to detect why it fails.

Stable results in 59