In the system log I spotted this:

Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=autoupdate)
Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate
Starting Google_Snow firmware updater v4 (autoupdate)...
 - Updater package: [Google_Snow.2695.117.0 / EC:snow_v1.3.139-375eb9f]
 - Current system:  [RO:Google_Snow_Rev4.2695.128.0 [RO_NORMAL], ACT:Google_Snow_Rev4.2695.128.0 / EC:snow_v1.3.139-375eb9f]
 - Write protection: Hardware: ON, Software: Main=ON
mv: cannot stat 'bios-snow-2695.132.ro.bin': No such file or directory
ERROR: Execution failed  [Reason: ALERT!!:Internal error] : ./updater4.sh (error code = 1)

Could this be the reason for the update to fail?
Also, could it be very similar to this old issue https://bugs.chromium.org/p/chromium/issues/detail?id=449356

Issue still present after chromeos 69.0.3497.95 / 10895.56.0 has been released for the Samsung Chromebook (daisy).

Could anyone please have a look?

Chrome 69 has been released and I'm stuck on 67!

Running an XE303C12-A01US dated April 2014.  I have the same or similar bug.  It is also preventing an update to v69.  Originally, I tried doing a recovery a few weeks ago when it refused to update to 68 (which didn't resolve the issue) but now I can't update to 69 either.  Logs attached, but pertinent part seems to be:

Set boot target to /dev/mmcblk0p5: Partition 5, Slot B
SetImage
KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 2539520 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=2539520 alg=sha1 root_hexdigest=1d0f3d736360b85b1017ea574a8d784786f7a84a salt=c45d5f50553690c291b717c951441eb2121c6838cf476cb8b52b597f45d277f2" noinitrd vt.global_cursor_default=0 kern_guid=%U  
Setting up verity.
Finished after 36 seconds.
Clearing network driver boot cache: /var/lib/preload-network-drivers.
Syncing filesystems before changing boot order...
Finished after 3 seconds.
Updating Partition Table Attributes using CgptManager...
Updated kernel 4 with Successful = 0 and NumTriesLeft = 6
Checking /mnt/stateful_partition/unencrypted permission.
Permission is ok.
Unlinked file /var/lib/ureadahead/pack
Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=autoupdate)
Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate
Starting Google_Snow firmware updater v4 (autoupdate)...
 - Updater package: [Google_Snow.2695.117.0 / EC:snow_v1.3.139-375eb9f]
 - Current system:  [RO:Google_Snow.2695.132.0 [RO_NORMAL], ACT:Google_Snow.2695.132.0 / EC:snow_v1.3.139-375eb9f]
 - Write protection: Hardware: ON, Software: Main=ON
mv: cannot stat 'bios-snow-2695.132.ro.bin': No such file or directory
ERROR: Execution failed: ./updater4.sh (error code = 1)
Finished after 2 seconds.
Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate - Exit Code 1
Firmware update failed (error code: 1).
Rolling back update due to failure installing required firmware.
Successfully updated GPT with all settings to rollback.
PostInstall Failed

[0918/175049:ERROR:postinstall_runner_action.cc(304)] Postinst command failed with code: 1
[0918/175049:ERROR:postinstall_runner_action.cc(354)] Postinstall action failed.

Deleted: update_engine.log.txt 153 KB

update_engine.log.txt 153 KB View Download

ChromeOS 69.0.3497.95 release is not 100% yet, which may be why devices are still seeing the issue.

@hungte, Is this issue fixed in ChromeOS 69.0.3497.95 when crbug.com/863789 is fixed?

Do we have some idea when this starts to fail?
I think it looks like the unibuild stuff may have changed updater packer and broke the logic there long ago, and it'll be much easier to figure out what happened if some one knows when it starts to fail (which build).

Ok, I found the problem. Extracting a ToT updater gives me this list:

 ./bin/crossystem
 ./bin/ec-2695.132.bin
 ./bin/crossystem.elf
 ./bin/futility
 ./bin/vpd.elf
 ./bin/flashrom.elf
 ./bin/bios-snow-2695.132.ro.bin
 ./bin/futility.elf
 ./bin/mosys.elf
 ./bin/flashrom
 ./bin/bios-snow-2695.132.rw.bin
 ./bin/updater_custom.sh
 ./bin/mosys
 ./bin/vpd

As you can see, by Mike's change of moving EXTRA to bin/, even the firmware images are now in bin instead of top folder, which creates the problem.

I think this implies all firmware packages doing multiple firmware are also broken; fortunately I've just done a clean up recently, and only daisy is left due to its having two models.

I can make some hack to get it work again.

A fix in https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-daisy-private/+/683571

it wasn't the intention to copy the .bin files over into bin/ too

looking at CROS_FIRMWARE_EXTRA_LIST= in the tree, daisy was the only one to use it this way

i guess rather than try to fix the firmware updater code at this point, we accept this and move forward with updater5 which wouldn't be affected by any of this ?

also somewhat related to  issue 851774  in that things broke in the same way

Yes I think we should just leave it there using updater4, since daisy not only used multiple images but also need to deal  with the complicated "RO-normal" logic.

We should just make it "not broken", and wait for EOL, and then deprecate the updater & extra list.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-daisy-private/+/7a0c324b6a2a9e1333a68c35b22334a034e0dbbc

commit 7a0c324b6a2a9e1333a68c35b22334a034e0dbbc
Author: Hung-Te Lin <hungte@chromium.org>
Date: Sat Sep 22 08:43:16 2018

This should have been fixed on R71 11089.0.0 or newer builds.
Feel free to reopen if you see other issues.

reopen. I think we want to make sure ToT works, and then see if we should backport this to 68, 69 even 70.

68 has sailed, but 69/70 sound fine since this prevents people from being able to update to it, and the CL in question only affects the daisy board

This bug requires manual review: M70 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: benmason@(Android), kariahda@(iOS), geohsu@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

 Issue 888550  has been merged into this issue.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-daisy-private/+/72402b015bc83d66080ad4982af956828a4e1e1c

commit 72402b015bc83d66080ad4982af956828a4e1e1c
Author: Hung-Te Lin <hungte@chromium.org>
Date: Tue Sep 25 02:02:16 2018

Will sheriffbot add R69 owners, or should we add that explicitly?

+cindy who should know about R69?

Re-assign to cindy - please let us know if we're OK to merge into R69.

Merge approved, M69.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-daisy-private/+/c3bb3677b899ad1a6498a4ff6dec3e6b85cfd913

commit c3bb3677b899ad1a6498a4ff6dec3e6b85cfd913
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Sep 27 23:37:22 2018

So M68 would still fail
M69, M70 and ToT should have been fixed.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/04e441e5f74e107d05578622909410259f6d6f99

commit 04e441e5f74e107d05578622909410259f6d6f99
Author: Hung-Te Lin <hungte@chromium.org>
Date: Tue Oct 02 12:19:09 2018

futility: updater: Add quirk 'daisy_snow_dual_model' for daisy_snow

The target AUE for daisy_snow is 74 or even longer, so we need to get a
better solution to get rid of script based updater customization (and the
painful EXTRA list in updater configuration).

The new quirk 'daisy_snow_dual_model' is assuming the input firmware image
has both daisy_snow x8 and x16 firmware packed into a single image
(because in vboot1, RW_A is identical to RW_B), and will modify A/B
contents according to target system.

BRANCH=None
BUG= chromium:881034 
TEST=make futil; tests/futility/run_test_scripts.sh $(pwd)/build/futility
     # Provide a fake mosys and output both MP / MPx16 to:
     futility update -i bios-snow-2695.132.117-rw.bin \
      --quirks daisy_snow_dual_model --emu emu.bin --sys_props 0,0x0000,0

Change-Id: I8af1b6c3117a703aed4da59902aaecb1009101f2
Signed-off-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1239798

[modify] https://crrev.com/04e441e5f74e107d05578622909410259f6d6f99/futility/updater.h
[modify] https://crrev.com/04e441e5f74e107d05578622909410259f6d6f99/futility/updater.c
[modify] https://crrev.com/04e441e5f74e107d05578622909410259f6d6f99/futility/updater_quirks.c

Hello, I saw there was an update available for my Chromebook and when I tried to update to 10895.78.0, I still get errors.  Log is attached but the pertinent part is below:

Setting up verity.
Finished after 45 seconds.
Clearing network driver boot cache: /var/lib/preload-network-drivers.
Syncing filesystems before changing boot order...
Finished after 2 seconds.
Updating Partition Table Attributes using CgptManager...
Updated kernel 4 with Successful = 0 and NumTriesLeft = 6
Checking /mnt/stateful_partition/unencrypted permission.
Permission is ok.
Unlinked file /var/lib/ureadahead/pack
Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=autoupdate)
Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate
Starting Google_Snow firmware updater v4 (autoupdate)...
 - Updater package: [Google_Snow.2695.117.0 / EC:snow_v1.3.139-375eb9f]
 - Current system:  [RO:Google_Snow.2695.132.0 [RO_NORMAL], ACT:Google_Snow.2695.132.0 / EC:snow_v1.3.139-375eb9f]
 - Write protection: Hardware: ON, Software: Main=ON
Detected x16/MP2 hardware - switching to x16/MP2 firmware now
RO BIOS: 2695.132.0, RW BIOS: 2695.132.0, EC: 2695.132.0
 * invoke: flashrom -p host -r _current/bios.bin
Firmware update available: Google_Snow.2695.132.0. 
Current key: a026a7a4a0bf0fa32d6b7aa90a80d5ef01a3b799 
Target  key: b11d74edd286c144e1135b49e7f0bc20cf041f10 DEV-signed rootkey

  Incompatible firmware image (Root key is different).

  You may need to disable hardware write protection and perform a factory
  install by '--mode=factory_install' or recovery by '--mode=recovery'.
  
ERROR: Incompatible Rootkey.
ERROR: Execution failed: ./updater4.sh (error code = 1)
Finished after 15 seconds.
Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate - Exit Code 1
Firmware update failed (error code: 1).
Rolling back update due to failure installing required firmware.
Successfully updated GPT with all settings to rollback.
PostInstall Failed

[1009/125039:ERROR:postinstall_runner_action.cc(304)] Postinst command failed with code: 1

Deleted: update_engine.log.txt 59.8 KB

update_engine.log.txt 59.8 KB View Download

this bug report is only about systems that have this error:
  mv: cannot stat 'bios-snow-2695.132.ro.bin': No such file or directory

your report doesn't have that problem which means it's a different issue.  please file a new report for us.

Yes that should be another issue.

A quick note: from the log it seems like the image pushed was not properly signed (it's DEV). @Mike, are we still signing bios_rw.bin on signer bot? I'm slightly concerned if that has been broken in the Unibuild migration.

The rw thing is eliminated by c#28 anyway, but that only lands in M71.

well it seems like the case. signer bot is not signing images in bin/. We need another issue. Created  issue 894324  .