New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 807369 link

Starred by 1 user

[Enguarde, Veyron_mighty, hana, likely others] Recovery is failing with unexpected error message.

Project Member Reported by sontis@chromium.org, Jan 30 2018

Issue description

Chrome Version:10323.9.0
OS:Chrome

What steps will reproduce the problem?
Try to recover the device using recovery USB stick.

What is the expected result?
Recovery should be successful.

What happens instead?
Recovery is failing with unexpected error message.


From Recovery logs:
==========================
Updating Partition Table Attributes using CgptManager...
Updated kernel 2 with Successful = 1 and NumTriesLeft = 6
Checking /mnt/stateful_partition/unencrypted permission.
RemovePackFiles Failed
Touch(/mnt/stateful_partition/.install_completed) FAILED
Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=recovery)
Command: //usr/sbin/chromeos-firmwareupdate --mode=recovery
Starting Google_Enguarde firmware updater v4 (recovery)...
 - Updater package: [RO:Google_Enguarde.5216.201.8 RW:Google_Enguarde.5216.201.10 / EC:enguarde_v1.6.172-89d5e7c]
 - Current system:  [RO:Google_Enguarde.5216.201.8 , ACT:Google_Enguarde.5216.201.8 / EC:enguarde_v1.6.172-89d5e7c]
 - Write protection: Hardware: ON, Software: Main=off EC=off
findString(): Vpd data 'whitelabel_tag' was not found.
ERROR: Execution failed: ./updater4.sh (error code = 3)
Finished after 3 seconds.
Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=recovery - Exit Code 3
Firmware can't be updated. Booted from RW Firmware B (error code: 3)
Rolling back update due to failure installing required firmware.
Successfully updated GPT with all settings to rollback.
PostInstall Failed
===============================================
 
Summary: [Enguarde, Veyron_mighty] Recovery is failing with unexpected error message. (was: [Enguarde] Recovery is failing with unexpected error message.)
Observed the same issue on Veyron_Mighty.  Logs uploaded to the link in C#2
Cc: dhadd...@chromium.org sdantul...@chromium.org mkarkada@chromium.org
Labels: -Type-Bug -ReleaseBlock-Stable ReleaseBlock-Dev Type-Bug-Regression
Reprod with AU too. attached log file.

Log info
*********************************************************************************

PostInstall Configured: (A, /dev/mmcblk0p3, /dev/mmcblk0p2, /dev/mmcblk0p12)

 Current Kernel Info: sysname(Linux) nodename(localhost) release(4.4.64-09098-g4504f6dcbb08) version(#1 SMP PREEMPT Wed Jul 26 23:32:53 PDT 2017) machine(x86_64)

lsb-release inside the new rootfs:
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={BB86912A-29BE-A59B-31EF-8F4DD1C023E2}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={BB86912A-29BE-A59B-31EF-8F4DD1C023E2}
CHROMEOS_RELEASE_BOARD=enguarde-signed-mp-v25keys
CHROMEOS_RELEASE_BRANCH_NUMBER=9
CHROMEOS_RELEASE_BUILDER_PATH=enguarde-release/R65-10323.9.0
CHROMEOS_RELEASE_BUILD_NUMBER=10323
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_DESCRIPTION=10323.9.0 (Official Build) dev-channel enguarde 
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=dev-channel
CHROMEOS_RELEASE_VERSION=10323.9.0
DEVICETYPE=CHROMEBOOK
GOOGLE_RELEASE=10323.9.0

Set boot target to /dev/mmcblk0p3: Partition 3, Slot A
SetImage
KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 3584000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=3584000 alg=sha1 root_hexdigest=a4606e57cb09071a95940f795a3e656b236c70dc salt=36cda804bf52781cb969260567348475440577ab95cda7c8d88d3934cc0851ef" noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic  
Setting up verity.
Finished after 30 seconds.
Clearing network driver boot cache: /var/lib/preload-network-drivers.
Syncing filesystems before changing boot order...
Finished after 1 seconds.
Updating Partition Table Attributes using CgptManager...
Updated kernel 2 with Successful = 0 and NumTriesLeft = 6
Checking /mnt/stateful_partition/unencrypted permission.
Permission is ok.
Unlinked file /var/lib/ureadahead/pack
Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=autoupdate)
Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate
Starting Google_Enguarde firmware updater v4 (autoupdate)...
 - Updater package: [RO:Google_Enguarde.5216.201.8 RW:Google_Enguarde.5216.201.10 / EC:enguarde_v1.6.172-89d5e7c]
 - Current system:  [RO:Google_Enguarde.5216.201.8 , ACT:Google_Enguarde.5216.201.10 / EC:enguarde_v1.6.172-89d5e7c]
 - Write protection: Hardware: ON, Software: Main=ON
findString(): Vpd data 'whitelabel_tag' was not found.
ERROR: Execution failed: ./updater4.sh (error code = 3)
Finished after 7 seconds.
Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate - Exit Code 3
Firmware can't be updated. Booted from RW Firmware B (error code: 3)
Rolling back update due to failure installing required firmware.
Successfully updated GPT with all settings to rollback.
PostInstall Failed

[0130/124334:ERROR:postinstall_runner_action.cc(291)] Postinst command failed with code: 3
[0130/124334:ERROR:postinstall_runner_action.cc(328)] Postinstall action failed.
[0130/124334:INFO:action_processor.cc(116)] ActionProcessor: finished PostinstallRunnerAction with code ErrorCode::kPostinstallBootedFromFirmwareB
[0130/124334:INFO:action_processor.cc(121)] ActionProcessor: Aborting processing due to failure.
[0130/124334:INFO:update_attempter.cc(901)] Processing Done.
[0130/124334:INFO:cpu_limiter.cc(71)] Setting cgroup cpu shares to  1024
[0130/124334:INFO:cpu_limiter.cc(79)] CPU shares = 1024
[0130/124334:ERROR:update_attempter.cc(1336)] Update failed.
update_engine.20180130-122743.txt
83.9 KB View Download
Cc: kmshelton@chromium.org philipchen@chromium.org vineeths@chromium.org
seen same issue on HANA.

https://buganizer.corp.google.com/issues/72663246
Owner: hungte@chromium.org
Probably from this commit:
https://chromium-review.googlesource.com/c/chromiumos/platform/firmware/+/844003
Cc: sjg@chromium.org hungte@chromium.org nsanders@chromium.org shapiroc@chromium.org
That is neat, I am not sure how this was only noted now, the logic that uses the fake vpd command was all in just before the branch, so we should have seen this on the last dev.

Does this repro back to 10321?

Can we revert https://chromium-review.googlesource.com/#/c/chromiumos/platform/firmware/+/844003/ on the 65 release branch to fix this?
Can someone whom is able to repro this try on 10321, or perhaps more pointedly the last dev push (10323.1)?

If this happens there, there is no point in blocking the dev update here.
yes, reprod on last dev RC(10323.1.0) too.
Summary: [Enguarde, Veyron_mighty, hana, likely others] Recovery is failing with unexpected error message. (was: [Enguarde, Veyron_mighty] Recovery is failing with unexpected error message.)
Labels: -ReleaseBlock-Dev ReleaseBlock-Beta
In that case punting to beta.

Comment 11 by dchan@chromium.org, Jan 31 2018

+hungte, please take a look, M65 beta is 2/1
Project Member

Comment 13 by bugdroid1@chromium.org, Feb 1 2018

Labels: merge-merged-release-R65-10323.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/39c490d92f304a38f9697215a6ecbbd38cc4068e

commit 39c490d92f304a38f9697215a6ecbbd38cc4068e
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Feb 01 01:19:07 2018

updater4: Ignore VPD failure when specified key value is not found.

On devices with key selection, 'vpd -g' may return error code if the
whitelabel_tag is missing so we should ignore the return code and only
check if the returned value is empty.

In shell scripting, 'local a=$(cmd)' is different from 'a=$(cmd)' in
return value. For example,

a() { local v=$(false); echo $?; }
b() { local v; v=$(false); echo $?; }

Executing a will print '0' while b will print '1'. That's why our
retrieval of customization_id (in local v=$(command) form) won't fail
while the new whitelabel_tag would.

The right solution is to add '|| true' to v=$(command).

BUG= chromium:807369 
TEST=./pack_firmware_functest.py

Change-Id: I29c462beddc503b08fffbdb162ae9030cbc15bc5
Reviewed-on: https://chromium-review.googlesource.com/895852
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Trybot-Ready: Hung-Te Lin <hungte@chromium.org>

[modify] https://crrev.com/39c490d92f304a38f9697215a6ecbbd38cc4068e/pack_dist/updater4.sh

Project Member

Comment 14 by bugdroid1@chromium.org, Feb 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/3c0497acb562de993243f079cd3a0a99b33cd3c6

commit 3c0497acb562de993243f079cd3a0a99b33cd3c6
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Feb 01 06:39:02 2018

updater4: Ignore VPD failure when specified key value is not found.

On devices with key selection, 'vpd -g' may return error code if the
whitelabel_tag is missing so we should ignore the return code and only
check if the returned value is empty.

In shell scripting, 'local a=$(cmd)' is different from 'a=$(cmd)' in
return value. For example,

a() { local v=$(false); echo $?; }
b() { local v; v=$(false); echo $?; }

Executing a will print '0' while b will print '1'. That's why our
retrieval of customization_id (in local v=$(command) form) won't fail
while the new whitelabel_tag would.

The right solution is to add '|| true' to v=$(command).

BUG= chromium:807369 
TEST=./pack_firmware_functest.py

Change-Id: I29c462beddc503b08fffbdb162ae9030cbc15bc5
Reviewed-on: https://chromium-review.googlesource.com/894926
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/3c0497acb562de993243f079cd3a0a99b33cd3c6/pack_dist/updater4.sh

Status: Fixed (was: Started)
merged to ToT & M65. Let me know if you see more failures.
Project Member

Comment 16 by bugdroid1@chromium.org, Feb 7 2018

Labels: merge-merged-factory-coral-10122.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/f86d230907f61c8772e261276a9095e4fe3f37e0

commit f86d230907f61c8772e261276a9095e4fe3f37e0
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Feb 07 20:31:08 2018

updater4: Ignore VPD failure when specified key value is not found.

On devices with key selection, 'vpd -g' may return error code if the
whitelabel_tag is missing so we should ignore the return code and only
check if the returned value is empty.

In shell scripting, 'local a=$(cmd)' is different from 'a=$(cmd)' in
return value. For example,

a() { local v=$(false); echo $?; }
b() { local v; v=$(false); echo $?; }

Executing a will print '0' while b will print '1'. That's why our
retrieval of customization_id (in local v=$(command) form) won't fail
while the new whitelabel_tag would.

The right solution is to add '|| true' to v=$(command).

BUG= chromium:807369 
TEST=./pack_firmware_functest.py

Change-Id: I29c462beddc503b08fffbdb162ae9030cbc15bc5
Reviewed-on: https://chromium-review.googlesource.com/894926
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
(cherry picked from commit 3c0497acb562de993243f079cd3a0a99b33cd3c6)
Reviewed-on: https://chromium-review.googlesource.com/906924
Reviewed-by: YH Lin <yueherngl@chromium.org>
Commit-Queue: Patrick Berny <pberny@chromium.org>
Tested-by: Patrick Berny <pberny@chromium.org>
Trybot-Ready: Patrick Berny <pberny@chromium.org>

[modify] https://crrev.com/f86d230907f61c8772e261276a9095e4fe3f37e0/pack_dist/updater4.sh

Status: Verified (was: Fixed)
Verified.

Sign in to add a comment