New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 894324 link

Starred by 15 users

Issue metadata

Status: Fixed
Owner:
Closed: Oct 12
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocked on:
issue 897218

Restricted
  • Only users with EditIssue permission may comment.



Sign in to add a comment

Daisy: Cannot update firmware (images in bin/ are not signed by signerbot)

Project Member Reported by hungte@chromium.org, Oct 11

Issue description

OS: Chrome OS M69+

What steps will reproduce the problem?
(1) Get a MP signed MPx2 (or known as x16) device 
(2) Auto-update to M69+ or run recovery shim

What is the expected result?
Success

What happens instead?
Failure in firmware update: the image is signed with DEV key.

Error messages:

Firmware update available: Google_Snow.2695.132.0. 
Current key: a026a7a4a0bf0fa32d6b7aa90a80d5ef01a3b799 
Target  key: b11d74edd286c144e1135b49e7f0bc20cf041f10 DEV-signed rootkey

  Incompatible firmware image (Root key is different).

--

This is an issue continued from    issue #881034   .

Since crosreview.com/798773, every extra/additional files are moved to bin/ folder. This has introduced several issues, including that customization need to look at new path (solved by    issue 881034   ), and that signer bot must look into bin/ as well.

I also tried to extract the content of M69 firmware updater:

./bin/bios-snow-2695.132.ro.bin # rootkey = DEV = b11d74edd286c144e1135b49e7f0bc20cf041f10
./bin/bios-snow-2695.132.rw.bin # rootkey = DEV = b11d74edd286c144e1135b49e7f0bc20cf041f10
./bios.bin # rootkey = MP = a026a7a4a0bf0fa32d6b7aa90a80d5ef01a3b799
./bios_rw.bin # rootkey = MP = a026a7a4a0bf0fa32d6b7aa90a80d5ef01a3b799

So it seems clear that signer bot is not signing bin/bios-*.bin.

There may be several solutions. Note that on M71+, we've eliminated the need of getting new images in bin/ by merging both firmware (x8 and x16) into a single image, but needs both changes in updater code (which only futility version was implemented) and images.

So we may need some solution for M69 & M70.

Mike, what do you think? Will it be very easy to let signer bot take bin/bios-* for M69/70, or do you think we should just rework M69/M70 firmware updater script so it can handle the M71 style special images?
 

Comment 1 Deleted

I believe this bug is in response to my comment on a different  bug #881034 .  If that is the case, then 893702, which I opened in response to comment 30 should be merged, somehow linked to this one, or closed.  Sorry for any confusion my improper labeling or unfamiliarity with this bug tracker has caused.
 Issue 893702  has been merged into this issue.
Description: Show this description
Description: Show this description
Another possible approach is to revise CL:798773, that bios*.bin should be moved to top folder instead of bin/. That may be even easier.
Hmmm, that's probably easier. https://chromium-review.googlesource.com/c/chromiumos/platform/firmware/+/1275766 is probably the most easy solution.
Labels: Merge-Request-70 Merge-Request-69
Request for merge of crosreview.com/1275766 for M69 and M70.

Note M71+ does not need it.
Project Member

Comment 10 by sheriffbot@chromium.org, Oct 11

Labels: -Merge-Request-70 Merge-Review-70 Hotlist-Merge-Review
This bug requires manual review: We are only 4 days from stable.
Please contact the milestone owner if you have questions.
Owners: benmason@(Android), kariahda@(iOS), geohsu@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -Merge-Review-70 Merge-Approved-70
Approved for M70. Please check with cindyb for M69, it's already in Stable and rolling out.
Cc: vapier@chromium.org
Owner: cindyb@chromium.org
lets just skip M69
Project Member

Comment 15 by bugdroid1@chromium.org, Oct 12

Labels: merge-merged-release-R70-11021.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/697a4648233394678c9d65c7d50c3c140b1aa684

commit 697a4648233394678c9d65c7d50c3c140b1aa684
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Oct 12 05:28:49 2018

pack_firmware: Keep extra firmware images in top level folder.

The signer bot does not process bin/*.bin, but currently all files
listed in EXTRA (which will be copied by CopyElfs) will live in bin/ sub
folder.

"Extra firmware images" is eliminated in M71+, so we want some simple
work around for M69 and M70, by copying files directly if the file
name looks like *.bin.

BUG= chromium:894324 
TEST=Revert CL:*684050, then run: emerge-daisy chromeos-firmware-daisy;
     chromeos-firmwareupdate -V | grep '\.bin'
     # 7f5b0669d9f43b0075cded5688a0a343 *./bios-snow-2695.132.ro.bin
     # 673c3efd476ac9a803cfc2f724c24806 *./bios_rw.bin
     # 0d2c9045017a843ad16d492897481d88 *./ec-2695.132.bin
     # 3849d1915bc50fb1a9a168c648517d37 *./bios-snow-2695.132.rw.bin
     # e9306cb76c8e636b4f5847cc7702abe7 *./bios.bin

Change-Id: Ie718e7a7e14606f01f7fc934c8f8c7ece0e63be0
Reviewed-on: https://chromium-review.googlesource.com/c/1278545
Tested-by: Hung-Te Lin <hungte@chromium.org>
Trybot-Ready: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>

[modify] https://crrev.com/697a4648233394678c9d65c7d50c3c140b1aa684/pack_firmware.py

Cc: -vapier@chromium.org cindyb@chromium.org
Labels: -M-69 -Merge-Request-69 -Merge-Approved-70
Owner: vapier@chromium.org
Status: Fixed (was: Assigned)
Ok, closing the issue now - M69 will be considered as WONTFIX then (#c14).
Project Member

Comment 17 by bugdroid1@chromium.org, Oct 12

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/bfae02ed17b04dbf30a147a7beb70e2941452248

commit bfae02ed17b04dbf30a147a7beb70e2941452248
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Oct 12 22:17:11 2018

pack_firmware: Keep extra firmware images in top level folder.

The signer bot does not process bin/*.bin, but currently all files
listed in EXTRA (which will be copied by CopyElfs) will live in bin/ sub
folder.

"Extra firmware images" is eliminated in M71+, so we want some simple
work around for M69 and M70, by copying files directly if the file
name looks like *.bin.

BUG= chromium:894324 
TEST=Revert CL:*684050, then run: emerge-daisy chromeos-firmware-daisy;
     chromeos-firmwareupdate -V | grep '\.bin'
     # 7f5b0669d9f43b0075cded5688a0a343 *./bios-snow-2695.132.ro.bin
     # 673c3efd476ac9a803cfc2f724c24806 *./bios_rw.bin
     # 0d2c9045017a843ad16d492897481d88 *./ec-2695.132.bin
     # 3849d1915bc50fb1a9a168c648517d37 *./bios-snow-2695.132.rw.bin
     # e9306cb76c8e636b4f5847cc7702abe7 *./bios.bin

Change-Id: Ie718e7a7e14606f01f7fc934c8f8c7ece0e63be0
Reviewed-on: https://chromium-review.googlesource.com/1275766
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/bfae02ed17b04dbf30a147a7beb70e2941452248/pack_firmware.py

Cc: josa...@chromium.org
+josafat

Hi Josafat, please let GE owners know that M69 daisy can't be updated (also recovery image won't work). Please skip them (again).
Hello all, a customer is affected by this bug and still unable to recover the device with a USB. They get the "Chrome OS is missing or damaged" screen. They have downloaded a Chrome OS image (70.0.3538.63) via direct download link, which we were advised included the fix, but the problem persists. What version/build is the fix being merged to? Some details below:

- Model: Samsung Chromebook (daisy) 
- Firmware version: Google_Snow.2695.132.0 
- HWID: Daisy Test A-A 9382 --> Not found in Recovery app. 

- Pressing Tab key on recovery screen shows recovery_reason: 
"0x17 RW firmware version rollback detected" (Screenshot attacked).
- Inserting a recovery USB results in message "The device you inserted does not contain Chrome OS".

Thanks for any information you can share.
20181016_095554_resized.jpg
1.0 MB View Download

Comment 20 Deleted

The HWID shows that the device should have been already corrupted, due to HWID went wrong, rootkey is DEV key.

It was probably had WP removed, and then recovered with a DEV signed recovery image before the fix, which results in such state.

Please recovery with a MP signed recovery image, or send to RMA.
I've downloaded the signed 11021.46.0 image and found all its 4 host firmware images were signed with
 { "root": "a026a7a4a0bf0fa32d6b7aa90a80d5ef01a3b799", "recovery": "13b0ddf343bb0c325b178e5be138d4969a9e02be"}

And apparently not what the screenshot said.


I, too, am affected by this.  Additionally, several other posts in a Google+ group identify this problem that happened within the last several days.  Is there a fix for this?  Otherwise a perfectly good Chromebook just self-destructed.  Write Protect sticker is intact.  I see the exact same information that was reported in the screenshot.  0x17 RW Firmware Version Rollback Detected.
We have had several devices become inoperable in the past 48 hours due to this.  While we knew they were listed as AUE we expected to be able to receive limited use for the remainder of this school year until we had funds available to replace them.  They're less than four years old and in fine condition.  The identical symptoms as described in comment 19.
Labels: Merge-Approved-69
Please merge to M69, merge is approved.
Able to reproduce this issue on daisy MPx16 device with 10895.78.0 stable build.

Recovery logs are present at https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cros/894324/?debugUI=CLOUD
Project Member

Comment 27 by bugdroid1@chromium.org, Oct 19

Labels: merge-merged-release-R69-10895.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/7fda010e75ce067f85558c9cc2b39ba3341874e4

commit 7fda010e75ce067f85558c9cc2b39ba3341874e4
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Oct 19 01:10:03 2018

pack_firmware: Keep extra firmware images in top level folder.

The signer bot does not process bin/*.bin, but currently all files
listed in EXTRA (which will be copied by CopyElfs) will live in bin/ sub
folder.

"Extra firmware images" is eliminated in M71+, so we want some simple
work around for M69 and M70, by copying files directly if the file
name looks like *.bin.

BUG= chromium:894324 
TEST=Revert CL:*684050, then run: emerge-daisy chromeos-firmware-daisy;
     chromeos-firmwareupdate -V | grep '\.bin'
     # 7f5b0669d9f43b0075cded5688a0a343 *./bios-snow-2695.132.ro.bin
     # 673c3efd476ac9a803cfc2f724c24806 *./bios_rw.bin
     # 0d2c9045017a843ad16d492897481d88 *./ec-2695.132.bin
     # 3849d1915bc50fb1a9a168c648517d37 *./bios-snow-2695.132.rw.bin
     # e9306cb76c8e636b4f5847cc7702abe7 *./bios.bin

Change-Id: Ie718e7a7e14606f01f7fc934c8f8c7ece0e63be0
Reviewed-on: https://chromium-review.googlesource.com/1275766
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit bfae02ed17b04dbf30a147a7beb70e2941452248)
Reviewed-on: https://chromium-review.googlesource.com/c/1290412
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>
Trybot-Ready: Hung-Te Lin <hungte@chromium.org>

[modify] https://crrev.com/7fda010e75ce067f85558c9cc2b39ba3341874e4/pack_firmware.py

Blockedon: 897218
Labels: -Merge-Approved-69 Merge-Merged Restrict-AddIssueComment-EditIssue
we're tracking breakage of x16 devices in  issue 894324 .  this bug as filed has been fixed, so closing it out.
Will the recovery image for 69 contain the fix? or should we wait until 70 is available for recovery?
If your device was not already bricked, recovery images since M69 10895.87.0 are working fine.

If your device is already bricked, a crafted recovery image is needed.
Labels: Hotlist-ConOps-CrOS
hungte@ 

How do users get a crafted recovery image? I want to be sure we are able to provide instructions to our Chromebook Central users. 

Comment 33 Deleted

Re#32

Sorry I have to delete the link in c#33. That was an internal doc targeting Google teams and not for public. 
Let me paste the related info here:

 1. Download a MP signed Daisy recovery image, for example chromeos_11021.46.0_daisy_recovery_stable-channel_snow-mp-v4.bin
 2. Get a complete Chrome OS source tree (chroot), and enter it (cros_sdk)
 3. cd trunk/src/platform/vboot_reference/scripts/image_signing
 4. sudo ./make_dev_ssd.sh --recovery -i PATH_TO/chromeos_11021.46.0_daisy_recovery_stable-channel_snow-mp-v4.bin --partitions 2

Then flash the image to USB stick and do recovery with it.

I've made a prebuilt version, but not sure if sharing direct link to it is a good idea.

@Josafat how do we usually deal with such case? Should we upload the image to some where and change recovery tool to use it?

@hungte and @josafat 

Would it be possible to craft an image on https://cros-updates-serving.appspot.com/#daisy in the recovery column? If so, I am happy to write instructions for our users. 
I think that page is automatically generated by Omaha config, so Josafat may know better.
@josafat, do we have any update on being able to share the premade image with customers? Cases 17291250 and 17291250 are being affected and customers are asking.
We are still validating final image to recover the units .. I'll update with location in the next few days 

I have a recovery image from M70, stable has started to release. Testing is going to double-check on Daisy and then I will load it to a drive folder.

Comment 40 Deleted

since there's no secrets in that image, and running it on a non-broken system won't break it, we should be able to copy it to a public GS bucket for anyone to access (if you want to share it with external users)
Would it be possible to allow the Drive image to be shared publicly with customers for now? We're hoping to allow a couple of customers test it to help confirm.
Update: Image was tested on a customer device and the attached screenshot was shown. A couple of customer devices are being sent for testing as well.
IMG_20181029_145126.jpg
4.4 MB View Download
Customer at 17281673 also tested the recovery image but error persisted. Image attached.
Still not recovering.JPG
3.6 MB View Download
Re#43 #44,  which image did you get and how that was flashed to USB stick?
Recovery is available: https://goo.gl/HLj7Y3
The customer at 17291250 tried downloading the recovery image at #46, copying it to a USB and inserting it into the affected device. It did not work. Is there any different instruction we should give them?
One of our customers tried the image from https://goo.gl/HLj7Y3 and he got the error message seen on the screenshot.
IMG_4940.JPG
77.0 KB View Download
Re#47
"copying it to a USB" is not clear enough. Did they use dd command? or did they use recovery tool? Mount + copy via file browser won't work. They should use the recovery tool.

Re#48
That is a corp enrolled device. Did they try to recovery using Ctrl-U? They should use recovery (Esc-Refresh-Power).
Here is the response from our customer:

That is a corp enrolled device. Did they try to recovery using Ctrl-U? They should use recovery (Esc-Refresh-Power)."

Yes - we used the proper recovery process, and tried it multiple times. Since it is an enrolled device we disable developer mode to prevent students from altering the devices so the custom recovery image fails validation as noted.
--------------------------------------------------
Re#50 Can you check with cindyb@ and send them recovery_m1.bin and see if that helps?
Both 17281673 and 17291250 confirmed that they have been using the recovery utility to create the flash drive. The error is the same as #48.

Comment 53 Deleted

Does write-protect need to be disabled for this image? 
it does, but if WP was enabled, then they wouldn't have gotten into this bad state in the first place.

if they have a system where the WP screw is loose, and so sometimes is enabled & sometimes disabled, then we don't have a way of reliably recovering their bricked system via software.  unfortunately, some internal analysis indicates there are a number of systems in such a state.

so if they have a system with WP enabled, they'll have to wait until we finish http://go/cros-wp-status.  which unfortunately might not be soon.
17281673: I can't figure out why they can't even boot the system. Maybe just re-create the flash drive and try again, then insert / remove flash drive, or try different USB slots after entering recovery mode.

17291250: Need more information how they failed. Was it the same as c#48?
To recover block_dev enabled device, we may need to hack the initramfs/init in recovery kernel, to prevent setting unofficial root.

I've crafted an image to do so - recovery_m3.zip. @cindyb, can you test it and release to the users if that helps?
Re#48 

Can you check if they did use recovery tool to create the flash drive?
I'm wondering if they simply use dd or something and got a corrupted flash drive due to auto-mount features on Linux host.
Is there anything we can tell customer at the moment regarding the progress of the investigation? They are asking for an update
Re#59

For most people whose devices are broken, use recovery tool to build a recover stick using image from #c46 then starting the recovery flow would be the right solution.

If failed to boot like #c44, please check if the image was created properly, and try different USB stick / USB slots.

If failed to recover like #c48, please check again if the image was created properly. Meanwhile we're experimenting some workaround and will check and update if that helps recovering devices in this case.

Comment 61 Deleted

@hungte - We still have multiple users on our Chromebook forum having issues with this image. 

For more information: https://productforums.google.com/forum/#!topic/chromebook-central/qctwuZ1hBRk

Do you have any other suggestions? 
From the thread I think

1. Developer mode is not needed. We should address this.
2. Seems like some people have problem creating a valid recovery image. Maybe we should try to find a "good" combination of environment, or ask Recovery Tool to do something better, for example verify again after flashed.

3. For any one failed during the recovery process (unexpected error), please ask them to attach the flash drive to a Linux machine, mount first partition, and upload the recovery logs on that partition.

I need the logs to figure out why it failed on their units.
hungte@ thank you for replying so quickly and for your partnership to get this resolved for our users. Unfortunately, many of our users will not be able to follow the process you outlined in number 3, as this is most likely the only device they own.

Is there alternative information we could ask users to provide? This is impacting a lot of devices and I am not sure all of our users will be able to follow this complex process to get their devices working again.
Re#64

Unfortunately we can't find any devices (yet) to reproduce their error (all devices we have today can be fixed by either m2 or m3 images), so getting logs is probably the most easy approach to figure out what went wrong. Of if we can get some ninja's help in getting those broken devices?

For people with two or more Chromebooks, I think they can also get logs easily - the working Chromebook should be able to read the USB stick and find out the files.

Another possible solution is when the "unknown error" screen displayed,  press Ctrl+Alt+F2,F3,F4 (forward,refresh, fulscreen, ... etc) keys - that should reveal different debug consoles.

However, the screen will only show last few lines (where the error may happened earlier and get scrolled up), and we may need users to scroll back (shift + up) to figure out where the error really happened - and that may need more interaction.
Re#65 

Customer (mtlaurelschools.org) sent two devices as per Sylvia instructions (sylcat@). These devices were delivered on October 31st in order to test the image, but we haven't received any update. The case number is: 17227144, please make reference to comment #43.

Hello,

It looks like one user was able to pull logs and can be found here: https://drive.google.com/open?id=1-3bedk2zxF_o632Rn9-3h77DWf0tA36N. 

hungte@ - can you take a look and see if this helps.

Thanks,
Alisha 
Re#67

Yes I saw the logs and found that these devices have WP enabled so we can't fix them.
These logs revealed that the device is in a state we were concerned about - unstable HW WP...

In the logs, write protection was enabled (Hardware: ON, Software: Main=ON EC=ON).
This implies the WP screw was temporary loosen when the AU starts, and corrupted device; and now it's fastened (or connected) again.

The proper way to recover these devices would be
 - Remove HW WP screw (
   https://www.chromium.org/a/chromium.org/dev/chromium-os/developer-information-for-chrome-os-devices/samsung-arm-chromebook#TOC-Firmware-Write-Protect )
 - Boot with the recovery m3 image to compete recovery
 - Fasten HW WP screw again

I think this is probably the solution to everyone who sees Unexpected Error.
And for people who want to create the special recovery images on their own (instead of trusting or downloading the files on web):

0. Get a signed daisy_snow recovery image (the recovery tool can help that)

Method 1: Resign kernel #2 as DEV

This can be done by re-signing only kernel, in following steps inside a Chrome OS source tree chroot:

 # Download a MP signed Daisy recovery image, for example
 # chromeos_11021.46.0_daisy_recovery_stable-channel_snow-mp-v4.bin,
 # let's call it recovery_image.bin
 cd trunk/src/platform/vboot_reference/scripts/image_signing
 sudo ./make_dev_ssd.sh --recovery -i PATH_TO/recovery_image.bin --partitions 2
 
Method 2: Add a DEV signed kernel #6 (keep #2 as MP signed)

The image by method 1 will only boot for devices in broken state, and not bootable for a MP device. So if we want to publish that as "recovery tool for all daisy", may need a special hack by cloning that to partition C (#6) and enable the partition:

 # Assume the original MP signed recovery image is `recovery_image.bin`, then inside chroot:
 IMAGE="PATH_TO/recovery_image.bin"
 truncate -s $(( $(stat -c '%s' "${IMAGE}") + 32768 * 512 )) "${IMAGE}"
 ~/trunk/src/platform/factory/bin/pygpt repair "${IMAGE}"
 cgpt add -i 6 -P 15 -T 15 -S 1 -b $(( $(cgpt show -i 1 -b "${IMAGE}") + \
  $(cgpt show -i 1 -s "${IMAGE}") )) -s 32768 "${IMAGE}"
 LO=$(sudo losetup -P --show --find "${IMAGE}")
 echo "${LO}" # should print something like /dev/loop0
 sudo dd if=${LO}p2 of="${LO}p6"
 sudo losetup -d "${LO}"
 ~/trunk/src/platform/vboot_reference/scripts/image_signing/make_dev_ssd.sh --recovery \
   -i "${IMAGE}" --partitions 6 --edit_config
 # In the editor or command line edit buffer, change NROFF=1 to NROFF=-3


Method 3: Add a DEV signed kernel #6 and disable set_unofficial_root in initramfs

It is found that on devices with block_dev set, the DEV signed kernel may not work properly. For such case, we have to disable the set_unofficial_root in recovery initramfs init script. To do that, apply http://crosreview.com/1312421, build a DEV signed recovery image, copy the recovery kernel #2 to the kernel #6 of an image prepared by Method 2, and keep the method 2 kernel command line unchanged:

 #!/bin/sh
 # build_m3.sh
 # usage: build_m3.sh ~/trunk/src/build/images/daisy/latest/recovery_image.bin

 die() {
   echo "$*" >&2
   exit 1
 }

 set -e

 VBOOT=/mnt/host/source/src/platform/vboot_reference
 [ -d "${VBOOT}" ] || die "Need to run inside chroot."
 [ -f "$1" ] || die "Usage: $0 PATH_TO/recovery_image.bin"

 src="$1"
 tmp="tmp"
 dest="recovery_m3.bin"
 cfg="${tmp}/cfg.6"  # make_dev_ssd adds kernel index.
 kern2="${tmp}/kern.2"
 mkdir -p ${tmp}

 # Extract recovery kernel
 lo=$(sudo losetup -P --show --find ${src})
 sudo dd if=${lo}p2 of=${kern2}
 sudo losetup -d ${lo}

 lo=$(sudo losetup -P --show --find ${dest})sudo dump_kernel_config ${lo}p2 >${cfg}
 sudo dd of=${lo}p6 if=${kern2}
 sudo losetup -d ${lo}

 sed -i 's/PARTNROFF=1/PARTNROFF=-3/' ${cfg}

 # Resign and change cmdline
 ${VBOOT}/scripts/image_signing/make_dev_ssd.sh \
   --recovery -i ${dest} --partitions 6 --set_config ${cfg%.*}

Build a daisy image, apply mod_image_for_recovery, do a Method 2 image first, copy that as recovery_m3.bin, and run the script above as:

 ./build_m3.sh ~/trunk/src/build/images/daisy/latest/recovery_image.bin


hungte@ thank you for this information in #68. 

We have one user that was able to successfully complete the process that was previously having issues. 
Cc: marcore@chromium.org
@hungte I don't have access to the log shared on https://bugs.chromium.org/p/chromium/issues/detail?id=894324#c67 so I can't compare with mine's.
here are the recovery log from case# 17261584 https://drive.google.com/open?id=1ZS2IEo3lVdywZTchMQ_h8mZO017StgrQ

should I advice the customer to "The proper way to recover these devices would be
 - Remove HW WP screw (
   https://www.chromium.org/a/chromium.org/dev/chromium-os/developer-information-for-chrome-os-devices/samsung-arm-chromebook#TOC-Firmware-Write-Protect )
 - Boot with the recovery m3 image to compete recovery
 - Fasten HW WP screw again" ?

Re#71

The logs in #17261584 have different issues.

From the log it failed in setting up root:

 dmsetup create -r vroot --table 0 2539520 verity payload=PARTUUID=29c07264-36ad-234a-958b-906ec726c701/PARTNROFF=1 hashtree=PARTUUID=29c07264-36ad-234a-958b-906ec726c701/PARTNROFF=1 hashstart=2539520 alg=sha1 root_hexdigest=99dd8531831400d4cbd6debdee83c04cc8132327 salt=7830cfe8ea58b08d33947261ca3fb68d9c97d615755ed6fcd25b14336584059a error_behavior=eio
device-mapper: reload ioctl on vroot failed: Argument list too long
Command failed
+ dlog Failed to configure device mapper root

Was that using m2 ( https://goo.gl/HLj7Y3) or m3 ( https://storage.cloud.google.com/chromeos-releases-public/recovery-images/recovery_m3.zip?folder=true&organizationId=433637338589 ) image?

My recommendation:

1. Try m2 image. If not work, try m3.
2. If both does not work, try enter developer mode first:
   a. Esc-F3-Power to trigger recovery mode manually, and then Ctrl-D, Enter.
   b. If recovery success, leave developer mode.

Response from customer in case 17361465

1. I created the recovery image from the download. I formatted the USB
drive through the recovery tool and then installed the local image from the
download.
2. Removed the WP screw and removed the washer.
3. Entered recovery mode and installed the recovery image.
4. It came up with the error message that was listed before (same as in #48)

Any advise will be appreciated
Re#73

So did they try m2 or m3, or both? that was not answered.

Please try both, and upload the recovery logs separately (the error message on screen does not help - we always need the log files)
(i.e., m2 and m3 used different kernel, so I'd assume the mapper issue may have caused problem on one of them, but I can't figure out which, and the user may have encountered different problems on different recovery images)
Project Member

Comment 75 by bugdroid1@chromium.org, Nov 19

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/9071205452595027cc8d52588f0c6fdf7581f905

commit 9071205452595027cc8d52588f0c6fdf7581f905
Author: Hung-Te Lin <hungte@chromium.org>
Date: Mon Nov 19 15:09:16 2018

futility: updater: Need --force when re-keying to DEV keys

For dogfood devices, we usually will only re-key from DEV to PreMP, and then
PreMP to MP.  It was found that for retail devices, if WP was disabled
(unintended), user may accidentally re-key to DEV keys if they (1)
recover with a DEV-signed image, or (2) received an AU that didn't have
right signing keys.

As a result, we want to make it harder when recovering to DEV keys.

BUG= chromium:894324 
TEST=make futil; tests/futility/run_test_scripts.sh $(pwd)/build/futility
BRANCH=None

Change-Id: Id3f7788e6c86d12b6e37b77818a1b4c2ceda1e2f
Signed-off-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1312596
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/9071205452595027cc8d52588f0c6fdf7581f905/futility/updater.c

#74 the ct has tried both versions (m2 and m3) .

He states the error message at #48 persists.  Is there any other method or solution we can try?
No idea unless if we can find machines to reproduce that.

Are there any other other units reporting same issues?
Also, please try the developer mode as I've indicated.

Try enter developer mode first:
 1. Esc-F3-Power to trigger recovery mode manually, and then Ctrl-D, Enter.
    This should turn on developer mode.
 2. Continue with recovery flow. If recovery success, leave developer mode.

Sign in to add a comment