Issue metadata
Sign in to add a comment
|
autotest: cros-version label verification is consistently failing after DUT provisioning |
||||||||||||||||||||||
Issue descriptionChrome OS Version : R67-10575.39.0 (moblab release) What steps will reproduce the problem? (1) Load R67-10575.39.0 on Moblab device (2) Stage an official image with custom tests on a local Moblab as defined in https://docs.google.com/document/d/e/2PACX-1vQKDTDTQFKjNxJatFkFUSjCPdVzgry9vkLLvxL8vwqasrKMP2KReEMZ3iva9GX8EzQYo-kANnzPlFG_/pub#h.pn77cclnhla8 (3) Provision the staged image on DUT using autotest. What is the expected result? Provisioning is successful. Post-Provisioning verification steps must PASS. What happens instead? Provisioning is sucessful. Host verification checks failed during post provisioning. cros-version check to be specific Please provide any additional information below. Attach a screenshot if possible. In /usr/local/autotest/server/hosts/cros_host.py::verify_cros_version_label() Build release path from the DUT (/etc/lsb-release) is matched against value of "cros-version" label of DUT. where, CHROMEOS_RELEASE_BUILDER_PATH is typically <boardname>-<buildertype>/<buildname> and cros-version label when official build with custom tests are staged should be <boardname>-custom/<buildname> As a result, use-cases where builder path on Moblab have "-custom" suffix will never pass, thereby blocking device provisioning. I appears that change made to fix https://bugs.chromium.org/p/chromium/issues/detail?id=800562 broke the scenarios that involve custom tests/images.
,
May 18 2018
Can you attach the logs from the provision job?
,
May 18 2018
Hello,
Please find the provisioning log attached.
Also attached autoserv log for reference.
05/17 17:00:34.869 WARNI| cros_host:1468| cros-version label
"cros-version:gale-custom-mandapaka/R68-10654.0.0" does not match
release_builder_path gale-release/R68-10654.0.0. Removing the label.
05/17 17:00:34.912 ERROR| repair:0354| Failed: Legacy host
verification checks
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line
351, in _verify_host
self.verify(host)
File "/usr/local/autotest/server/hosts/repair.py", line 55, in verify
host.verify_software()
File "/usr/local/autotest/server/hosts/cros_host.py", line 1629, in
verify_software
self.verify_cros_version_label()
File "/usr/local/autotest/server/hosts/cros_host.py", line 1472, in
verify_cros_version_label
raise error.AutoservError('The host has wrong cros-version label.')
AutoservError: The host has wrong cros-version label.
05/17 17:00:34.914 INFO | server_job:0224|
FAIL ---- verify.cros timestamp=1526601634 localtime=May 17
17:00:34 The host has wrong cros-version label.
Regards,
Harsha
,
May 18 2018
I'm not sure I understand the issue with cros-version label you mentioned, but even before that the log indicate that provision is failing due to an error staging provision artifacts for gale-custom-mandapaka/R68-10654.0.0 from GS I'm not familiar with the custom staging process on moblab, but can you try staging to GS?
,
May 18 2018
First attempt to stage the provision artifacts to GS had failed but when retried with boto file as extra environment, staging to GS has passed and build flashing has eventually passed.
// staging if provisioning artifacts [log: provision_AutoUpdate.DEBUG]
05/17 16:51:47.784 WARNI|provision_AutoUpda:0106| Unable to stage quick provision payload:
// staging passed during retry [log: provision_AutoUpdate.DEBUG]
cmd=['/mnt/moblab/cros_cache/common/gsutil_4.30.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'ls', '--', u'gs://chromeos-image-archive/gale-custom-mandapaka/R68-10654.0.0'], extra env={'BOTO_CONFIG': '/home/moblab/.boto'}
05/17 16:51:47.899 DEBUG|provision_AutoUpda:0120| Installing image
05/17 16:53:08.661 DEBUG| jetstream_host:0087| Jetstream: Prepare for update
05/17 16:53:36.565 DEBUG| utils:0282| [stdout] Resetting stateful update state.
05/17 16:53:36.593 INFO | autoupdater:0312| Updating image via: /usr/bin/update_engine_client --update --omaha_url=http://192.168.231.1:8080/update/gale-custom-mandapaka/R68-10654.0.0
05/17 16:58:24.122 INFO | autoupdater:0516| Updating stateful partition...
05/17 16:58:54.540 INFO | autoupdater:0577| Update complete.
05/17 16:58:54.541 INFO | autoupdater:0588| Update engine log has downloaded in sysinfo/update_engine dir. Check the lastest.
05/17 17:00:29.452 DEBUG|provision_AutoUpda:0131| Finished provisioning <remote host: 192.168.231.101> to gale-custom-mandapaka/R68-10654.0.0
// cros_version verification done after provisioning [log: autoserv.DEBUG]
05/17 17:00:33.797 INFO | repair:0349| Verifying this condition: Legacy host verification checks
05/17 17:00:34.869 WARNI| cros_host:1468| cros-version label "cros-version:gale-custom-mandapaka/R68-10654.0.0" does not match release_builder_path gale-release/R68-10654.0.0. Removing the label.
,
Jun 11 2018
Any workarounds for this issue ?
,
Jun 14 2018
I took a look. The problem is that we're checking to make sure the cros-version label matches what is inside the /etc/lsb-release file for CHROMEOS_RELEASE_BUILDER_PATH I think this behavior is perfectly sensible on the test side, so the problem should be fixed on the build side if possible. How are these custom images being built? The build process should be putting the correct value in /etc/lsb-release
,
Jun 14 2018
They are generated using 'cros stage', as mentioned in the Moblab user manual: https://sites.google.com/a/chromium.org/dev/chromium-os/testing/moblab The 'custom' designation has been part of cros stage for a long time. The addition of the username is Jetstream-specific, but has been part of its developer flow for a couple years now. The 'custom' bit is here: http://cs/chromeos_public/chromite/cli/cros/cros_stage.py?l=32&rcl=de5f8a610a0a4407a1e02787e08410080b75a593
,
Jun 15 2018
We don't want to revert the change that caused this regression, so we'll have to bite the bullet and actually figure out how to fix the cros-version verification logic properly. jrbarnette has "volunteered" to hold onto this bug as he has more of an idea in mind about what needs to happen here than I do.
,
Jun 18 2018
> They are generated using 'cros stage', as mentioned in the Moblab user manual: IIUC, this command is rewriting a CrOS image with custom content. Does the command also change the content of /etc/lsb-release to mark the build as custom? If it doesn't change the content why, and can we fix that? My preferred solution would be for `cros stage` to mark its images in /etc/lsb-release. I can make specific recommendations as to the kinds of changes that would be most useful.
,
Jun 18 2018
'cros stage' only modifies autotest packages. The device image is unaltered, so lsb-release is not changed.
,
Jun 18 2018
> 'cros stage' only modifies autotest packages. The device image is unaltered, > so lsb-release is not changed. <sigh> Hokay. Plan B is that we hard-code the "$BOARD-custom/R..." pattern in Autotest, and change the verification test to skip image names that match the pattern.
,
Jun 18 2018
If going with plan B, we'd appreciate support for any build type that contains 'custom', as in comment #3 where it is 'gale-custom-mandapaka/R68-10654.0.0'. We could consider changing 'cros stage' to modify the lsb-release, but it would be a larger change.
,
Jun 30 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c92805bfec5613974999b255fb604095c2e43095 commit c92805bfec5613974999b255fb604095c2e43095 Author: Richard Barnette <jrbarnette@chromium.org> Date: Sat Jun 30 00:50:21 2018 [autotest] Handle version labels from `cros stage` The tool `cros stage` updates and uploads test image content to a new location in googlestorage. In uploading to a new location, the process invalidates the value of CHROMEOS_RELEASE_BUILDER_PATH in /etc/lsb-release in the root file system. Provisioning believes that the URL from which the image is downloaded and the setting of CHROMEOS_RELEASE_BUILDER_PATH are matched, and includes a sanity check to make sure the image installed has the expected setting. Images from `cros stage` break this check, resulting in failed provisioning tasks. This amends the check to exclude the specific label names produced by `cros stage` from the provisioning check. BUG= chromium:844276 TEST=None Change-Id: Ic7865f804fda2fcbed4711441806dbabeff486df Reviewed-on: https://chromium-review.googlesource.com/1115715 Commit-Ready: Richard Barnette <jrbarnette@chromium.org> Tested-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> Reviewed-by: Laurence Goodby <lgoodby@chromium.org> [modify] https://crrev.com/c92805bfec5613974999b255fb604095c2e43095/server/hosts/cros_host.py
,
Jul 3
,
Jul 15
Hello Richard, How can I figure out the Moblab build label in which this fix landed? Regards, Harsha
,
Jul 15
> How can I figure out the Moblab build label in which this fix landed? go/crosland is your friend. The CL is the one in #c15, that is, 1115715. So: https://crosland.corp.google.com/cl?q=chromium:1115715
,
Jul 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6ef69038d3af64ae0a91d0070dfdea42890486c7 commit 6ef69038d3af64ae0a91d0070dfdea42890486c7 Author: Richard Barnette <jrbarnette@chromium.org> Date: Mon Jul 16 21:15:31 2018 [autotest] Handle version labels from `cros stage` The tool `cros stage` updates and uploads test image content to a new location in googlestorage. In uploading to a new location, the process invalidates the value of CHROMEOS_RELEASE_BUILDER_PATH in /etc/lsb-release in the root file system. Provisioning believes that the URL from which the image is downloaded and the setting of CHROMEOS_RELEASE_BUILDER_PATH are matched, and includes a sanity check to make sure the image installed has the expected setting. Images from `cros stage` break this check, resulting in failed provisioning tasks. This amends the check to exclude the specific label names produced by `cros stage` from the provisioning check. BUG= chromium:844276 TEST=None Change-Id: Ic7865f804fda2fcbed4711441806dbabeff486df Reviewed-on: https://chromium-review.googlesource.com/1115715 Commit-Ready: Richard Barnette <jrbarnette@chromium.org> Tested-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> Reviewed-by: Laurence Goodby <lgoodby@chromium.org> (cherry picked from commit c92805bfec5613974999b255fb604095c2e43095) Reviewed-on: https://chromium-review.googlesource.com/1138914 Reviewed-by: Keith Haddow <haddowk@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [modify] https://crrev.com/6ef69038d3af64ae0a91d0070dfdea42890486c7/server/hosts/cros_host.py
,
Jul 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/2e921e58db9533ebe2186aa9dcc1d09ff3c20a67 commit 2e921e58db9533ebe2186aa9dcc1d09ff3c20a67 Author: Richard Barnette <jrbarnette@chromium.org> Date: Mon Jul 16 21:20:01 2018 [autotest] Handle version labels from `cros stage` The tool `cros stage` updates and uploads test image content to a new location in googlestorage. In uploading to a new location, the process invalidates the value of CHROMEOS_RELEASE_BUILDER_PATH in /etc/lsb-release in the root file system. Provisioning believes that the URL from which the image is downloaded and the setting of CHROMEOS_RELEASE_BUILDER_PATH are matched, and includes a sanity check to make sure the image installed has the expected setting. Images from `cros stage` break this check, resulting in failed provisioning tasks. This amends the check to exclude the specific label names produced by `cros stage` from the provisioning check. BUG= chromium:844276 TEST=None Change-Id: Ic7865f804fda2fcbed4711441806dbabeff486df Reviewed-on: https://chromium-review.googlesource.com/1115715 Commit-Ready: Richard Barnette <jrbarnette@chromium.org> Tested-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> Reviewed-by: Laurence Goodby <lgoodby@chromium.org> (cherry picked from commit c92805bfec5613974999b255fb604095c2e43095) Reviewed-on: https://chromium-review.googlesource.com/1138915 Reviewed-by: Keith Haddow <haddowk@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [modify] https://crrev.com/2e921e58db9533ebe2186aa9dcc1d09ff3c20a67/server/hosts/cros_host.py |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by mandapaka@chromium.org
, May 18 2018