Issue metadata
Sign in to add a comment
|
Regression : Recovery is failing with an Unexpected error message while installing R63-10016.0.0 via USB |
||||||||||||||||||||||
Issue descriptionChrome Version: 63.0.3230/10016.0.0 dev-channel Gnawty and Parrot OS: Chrome What steps will reproduce the problem? (1)Connect the USB with 63.0.3230/10016.0.0 (2)Try to recover the build(Esc+reload+power) and observe. Expected: Should be able to recover the build. Actual: Instead Unexpected Error message is seen This is a Regression issue as build recovery is successful with 63.0.3223.0/9978.0.0 dev-channel Daisy Good Build:63.0.3223.0/9978.0.0 Bad Build: 63.0.3223.0/9979.0.0 Note: 1.As per comment #7 from issue 769602 Filing this Issue 2.Checked on Daisy,Kip,Peppy,Paine,Blaze,Candy and Reks 3.Checked with 2 USB sticks[a)SUPER * TALENT USB 3.0 Express ST1-2 & b)Patriot Memory,supersonic 8GB ] Attaching Log files
,
Oct 9 2017
Is that really the last good build? Hard to believe that there were 4 other tot QA's after 9978 and this wasn't caught... https://cros-goldeneye.corp.google.com/chromeos/console/listBuild?milestone=63#%2F
,
Oct 9 2017
Tested on Gnawty, AU and Recovery both are working as excepted. Chrome Version: 63.0.3230/10016.0.0 dev-channel Gnawty
,
Oct 9 2017
After offline chat with Ansar, not reprod on devices listed. Maybe the USB stick was broken?
,
Oct 9 2017
Able to reproduce the issue on 63.0.3230/10016.0.0 dev-channel Gnawty.
,
Oct 9 2017
Please ignore my comment #3, I was tested on Gnawty+ with Gnawty image. Recovery failed on Gnawty with 10016.0.0 but successful auto updated to 10016.0.0 Please find the update_engine log as attached.
,
Oct 9 2017
From Recovery logs: =============================== Updating Partition Table Attributes using CgptManager... Updated kernel 2 with Successful = 1 and NumTriesLeft = 6 Checking /mnt/stateful_partition/unencrypted permission. RemovePackFiles Failed Touch(/mnt/stateful_partition/.install_completed) FAILED Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=recovery) Command: //usr/sbin/chromeos-firmwareupdate --mode=recovery Starting Google_Gnawty firmware updater v4 (recovery)... - Updater package: [RO:Google_Gnawty.5216.239.109 RW:Google_Gnawty.5216.239.123 / EC:gnawty_v1.6.206-3acdf9e] - Current system: [RO:Google_Gnawty.5216.239.3 , ACT:Google_Gnawty.5216.239.3 / EC:gnawty_v1.6.206-3acdf9e] - Write protection: Hardware: ON, Software: Main=ON EC=ON recovery: update main/RW:A,B,SHARED * invoke: flashrom -p host -r _current/bios.bin ERROR: Failed to get RW firmware information ERROR: Execution failed: ./updater4.sh (error code = 1) Finished after 9 seconds. Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=recovery - Exit Code 1 Firmware update failed (error code: 1). Rolling back update due to failure installing required firmware. Successfully updated GPT with all settings to rollback. PostInstall Failed ===================================
,
Oct 9 2017
,
Oct 9 2017
yes, that error in comment #7 indicates something is going wrong with flashrom. someone with an actual failing gnawty device will need to debug it though. cros_get_rw_firmware_info is sending all of its error output to /dev/null.
looks like this bit of code:
src/platform/firmware/pack_dist/updater4.sh
# Get RW firmware information
local fw_info
fw_info="$(cros_get_rw_firmware_info "$DIR_TARGET/$TYPE_MAIN/VBLOCK_A" \
"$DIR_TARGET/$TYPE_MAIN/FW_MAIN_A" \
"$target_image")" || fw_info=""
[ -n "$fw_info" ] || die "Failed to get RW firmware information"
src/platform/firmware/pack_dist/crosutil.sh
cros_get_rw_firmware_info() {
check_param "cros_get_rw_firmware_info(vblock, fw_main, image)" "$@"
local vblock="$1"
local fw_main="$2"
local image="$3"
local rootkey="_rootkey"
silent_invoke futility gbb -g --rootkey="$rootkey" "$image" 2>/dev/null ||
return
local fw_info
fw_info="$(futility vbutil_firmware \
--verify "$vblock" \
--signpubkey "$rootkey" \
--fv "$fw_main" 2>/dev/null)" || fw_info=""
echo "$fw_info"
}
,
Oct 9 2017
could also bisect down the version range too. 9979.0.0 is known to be broken (and the fix didn't land until 9984.0.0). so check versions from 9984.0.0 through 10016.0.0.
,
Oct 9 2017
Hunge Te wondering if this is related to your change that landed in 9978.0.0? https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/668352/ Although I would expect this to fail for all devices.
,
Oct 9 2017
can someone w/a gnawty just do a "quick" run through the current versions and see which work/fail ? you can use `cros flash` to pull down known working versions.
,
Oct 10 2017
Re#11 I think that is not related because 9978 was reported to be "good" in this issue. One of the change landed in 9979 and touching the code listed in #9 is https://chromium-review.googlesource.com/c/chromiumos/platform/firmware/+/668318 , but I can't see if that would break anything. But if anyone has a gwanty, can you check a build with 668318 reverted would have this problem fixed? Re#12 I don't think anything has been fixed in 10016. According to #6, "Recovery failed on Gnawty with 10016.0.0 but successful auto updated to 10016.0.0". That means it succeeded in AU mode instead of recovery mode. Maybe we should remove the 2>/dev/null to find out what's wrong.
,
Oct 10 2017
I believe this error is related to this recent change: https://chromium-review.googlesource.com/c/chromiumos/platform/firmware/+/668318 i.e. use of old vbutil_firmware was replaced with futility vbutil_firmware. However, the futility version that is included in the updater is futility_s i.e. the static futility binary. And if you do a futility_s help, it indicates that vbutil_firmware is not supported. This has to do with the FUTIL_STATIC_SRCS in https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/master/Makefile#664. Though, the solution is not as simple as just adding FUTIL_SRCS to FUTIL_STATIC_SRCS. All the dependencies on crypto libraries, lz and ldl need to be taken care of.
,
Oct 10 2017
,
Oct 10 2017
i posted a CL to drop the 2>/dev/null looks like we need just vbutil_firmware & vbutil_key added: https://chromium-review.googlesource.com/708236
,
Oct 10 2017
i think before my CL 668318 we were running the vbutil programs from $PATH which in turn were links to the dynamic futility. i.e. we should have been bundling them already, but we weren't.
,
Oct 10 2017
As updated in C#0, this issue is seen on Daisy,Kip,Peppy,Paine,Blaze,Candy and Reks. Also, reconfirming that issue persists on latest ToT build 63.0.3230/10018.0.0 Peppy,Kip,Reks,Candy,Quawks,Celes and Falco. i.e.Unexpected error message is seen.
,
Oct 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/5d5b2f16109662dceb55dfbe62772ee41887be82 commit 5d5b2f16109662dceb55dfbe62772ee41887be82 Author: Mike Frysinger <vapier@chromium.org> Date: Tue Oct 10 12:31:03 2017 futility_s: add vbutil_firmware & vbutil_key The firmware updater relies on these, so include them. BUG= chromium:772862 TEST=build works and futility_s includes new commands BRANCH=None Change-Id: I973e339d11df4d5b72f70d3a9e28916cca4ebbf6 Reviewed-on: https://chromium-review.googlesource.com/708236 Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Furquan Shaikh <furquan@chromium.org> [modify] https://crrev.com/5d5b2f16109662dceb55dfbe62772ee41887be82/Makefile
,
Oct 10 2017
should be fixed by above commit
,
Oct 10 2017
AU failure on parrot device from 9901.35.0 ==> M63 Dev RC 10009.0.0
,
Oct 10 2017
anything <10020.0.0 is known to be broken. please retest with 10020.0.0+.
,
Oct 10 2017
Recovery installation is failing for10009.0.0_winky. Getting unexpected error message. Recovery logs are present at https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cros/772862/?debugUI=CLOUD
,
Oct 10 2017
reprod on chell too.
,
Oct 10 2017
again, please don't bother testing images older than 10020.0.0. reporting them as broken isn't useful when we already know they're broken. please update the images you're testing.
,
Oct 10 2017
Tested on cros:10020 but still AU is failed on parrot. AU log info: -------------------------------------------------------------- TO: CHROMEOS_AUSERVER=https://tools.google.com/service/update2 CHROMEOS_BOARD_APPID={9D137383-EB72-4BA9-A523-91AC0853F8AD} CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1} CHROMEOS_DEVSERVER= CHROMEOS_RELEASE_APPID={9D137383-EB72-4BA9-A523-91AC0853F8AD} CHROMEOS_RELEASE_BOARD=parrot-signed-mp-v4keys CHROMEOS_RELEASE_BRANCH_NUMBER=0 CHROMEOS_RELEASE_BUILDER_PATH=parrot-release/R63-10020.0.0 CHROMEOS_RELEASE_BUILD_NUMBER=10020 CHROMEOS_RELEASE_BUILD_TYPE=Official Build CHROMEOS_RELEASE_CHROME_MILESTONE=63 CHROMEOS_RELEASE_DESCRIPTION=10020.0.0 (Official Build) dev-channel parrot CHROMEOS_RELEASE_NAME=Chrome OS CHROMEOS_RELEASE_PATCH_NUMBER=0 CHROMEOS_RELEASE_TRACK=dev-channel CHROMEOS_RELEASE_VERSION=10020.0.0 DEVICETYPE=CHROMEBOOK GOOGLE_RELEASE=10020.0.0 ChromeosChrootPostinst(10020.0.0) Set boot target to /dev/sda5: Partition 5, Slot B SetImage KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-1 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="2 vboot none ro 1,0 2579200 bootcache PARTUUID=%U/PARTNROFF=1 2579200 8d0802652a04fcb17e59127ac568ce87e5a60800 512 20000 100000, vroot none ro 1,0 2539520 verity payload=254:0 hashtree=254:0 hashstart=2539520 alg=sha1 root_hexdigest=c852ea435ddfee0c60db64f618fe7075f51bffd1 salt=95fd56a1c6ddfb9f0ee7bf2efdf46b04d848ee953f25d93621402922457ecc6a" noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic iTCO_vendor_support.vendorsupport=3 Setting up verity. Finished after 23 seconds. Clearing network driver boot cache: /var/lib/preload-network-drivers. Syncing filesystems before changing boot order... Finished after 2 seconds. Updating Partition Table Attributes using CgptManager... Updated kernel 4 with Successful = 0 and NumTriesLeft = 6 Checking /mnt/stateful_partition/unencrypted permission. RemovePackFiles Failed Touch(/mnt/stateful_partition/.install_completed) FAILED Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=autoupdate) Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate Starting Google_Parrot firmware updater v3 (autoupdate)... - Updater package: [Google_Parrot.2685.54.0 / 00BE107A00] - Current system: [RO:Google_Parrot.2685.37.0 [RO_NORMAL], ACT:Google_Parrot.2685.37.0 / 00BE107A00] Warning: wpsw_cur is not availble, using wpsw_boot (1) Segmentation fault (core dumped) - Write protection: Hardware: ON, Software: Main=off Warning: EC FMAP layout file is overridden by ec.layout. * invoke: flashrom -p host -r _current/bios.bin Execution failed (139): flashrom -p host -r _current/bios.bin Messages: Calibrating delay loop... OK. flashrom v0.9.9 : b17e9e4 : Sep 27 2017 19:58:03 UTC on Linux 3.8.11 (x86_64) flashrom v0.9.9 : b17e9e4 : Sep 27 2017 19:58:03 UTC on Linux 3.8.11 (x86_64) Segmentation fault (core dumped) ERROR: Execution FAILED. ERROR: Execution failed: ./updater3.sh (error code = 1) Finished after 5 seconds. Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate - Exit Code 1 Firmware update failed (error code: 1). Rolling back update due to failure installing required firmware. Successfully updated GPT with all settings to rollback. PostInstall Failed
,
Oct 10 2017
,
Oct 10 2017
,
Oct 10 2017
Reopened bug. Cl in #19 landed in 10020 but issue still seen. Mike could you please take a look?
,
Oct 11 2017
that's a different issue. that's flashrom segfaulting, not vboot/futility programs not working. this bug is only about the vbutil subprograms missing. is recovery failing too, or only AU ?
,
Oct 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/firmware/+/222889a5a58a0fe05ee1b01edda506f03c9fa366 commit 222889a5a58a0fe05ee1b01edda506f03c9fa366 Author: Mike Frysinger <vapier@chromium.org> Date: Wed Oct 11 05:13:46 2017 drop stderr silencing If these tools fail, we abort, so we don't want to throw away their stderr output. They've been this way since they started. BUG= chromium:772862 TEST=precq passes Change-Id: Ib56d010f63d6dfc51868b2e7ad53aa071e4e4627 Reviewed-on: https://chromium-review.googlesource.com/708254 Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/222889a5a58a0fe05ee1b01edda506f03c9fa366/pack_dist/crosutil.sh
,
Oct 11 2017
Able to install build in Dev Mode for 63.0.3236.0/10021.0.0 dev-channel Peppy,Kip and in verified mode unable to install build instead "You are using an out-of-Date official Chrome OS recovery image" error Message is seen(Please refer Screenshot) Attaching log files for verified Mode
,
Oct 11 2017
that bug isn't related to this. this bug is only about firmware-updater. that code failure is the recovery init itself which is issue 764753 . there's already a CL in flight to fix it though.
,
Oct 11 2017
sorry, issue 765844 is tracking the futility_s usage in the recovery initramfs
,
Oct 11 2017
According to abodeti@ both AU and recovery are failing.
,
Oct 11 2017
the only bug i'm looking at is the parrot/flashrom crash. all other issues should be resolved (or there's a CL in flight for it), so i'm not looking at any existing reports. they need to be revalidated with ToT (although you prob should wait for issue 765844 first).
,
Oct 11 2017
the flashrom crash isn't related to my changes or this bug. it's been crashing on parrot since 9981.0.0. i've moved that to issue 773801 . since all the issues raised here are fixed or tracked in a diff bug, i'm closing this out. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by mmanchala@chromium.org
, Oct 9 2017