logs are present at https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cros/810541/?debugUI=CLOUD

AU also failed.

log info
********************************************

CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_DESCRIPTION=10323.24.0 (Official Build) dev-channel lars 
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=dev-channel
CHROMEOS_RELEASE_VERSION=10323.24.0
DEVICETYPE=CHROMEBOOK
GOOGLE_RELEASE=10323.24.0

Set boot target to /dev/mmcblk0p5: Partition 5, Slot B
SetImage
KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 3584000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=3584000 alg=sha1 root_hexdigest=c616652080795b9dfbd66959e79ea0b585bf3b69 salt=745111c85797f63b748931b350985e09b25dc2bb5bf1d352901a88c8397c19fa" noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic intel_idle.max_cstate=7  
Setting up verity.
Finished after 13 seconds.
Clearing network driver boot cache: /var/lib/preload-network-drivers.
Syncing filesystems before changing boot order...
Finished after 1 seconds.
Updating Partition Table Attributes using CgptManager...
Updated kernel 4 with Successful = 0 and NumTriesLeft = 6
Checking /mnt/stateful_partition/unencrypted permission.
Permission is ok.
Unlinked file /var/lib/ureadahead/pack
Starting firmware updater (//usr/sbin/chromeos-firmwareupdate --mode=autoupdate)
Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate
Starting Google_Lars firmware updater v4 (autoupdate)...
 - Updater package: [RO:Google_Lars.7820.298.0 RW:Google_Lars.7820.317.0 / EC:lars_v1.9.375-8829f10 / PD:lars_pd_v1.9.375-8829f10]
 - Current system:  [RO:Google_Lars.7820.127.0 , ACT:Google_Lars.7820.127.0 / EC:lars_v1.9.210-4472c78 / PD:lars_pd_v1.9.210-4472c78]
 - Write protection: Hardware: ON, Software: Main=off
./updater4.sh: 46: ./updater4.sh: cros_spi_descriptor: not found
./updater4.sh: 61: ./updater4.sh: cros_spi_descriptor: not found
autoupdate(recovery): update RO+RW
 * invoke: flashrom -p host --fast-verify -w bios.bin
 Execution failed (255): flashrom -p host --fast-verify -w bios.bin
 Messages:
Calibrating delay loop... OK.
coreboot table found at 0x7aa4c000.
Erasing and writing flash chip... Verifying flash... FAILED
flashrom v0.9.9  : 831c609 : Nov 30 2017 02:04:47 UTC on Linux 3.18.0-12637-gab5706d (x86_64)
flashrom v0.9.9  : 831c609 : Nov 30 2017 02:04:47 UTC on Linux 3.18.0-12637-gab5706d (x86_64)
WARNING: SPI Configuration Lockdown activated.
VERIFY FAILED at 0x000001f5! Expected=0x84, Read=0x82, failed byte count from 0x00000000-0x00ffffff: 0x1
Your flash chip is in an unknown state.
Get help on IRC at irc.freenode.net (channel #flashrom) or
mail flashrom@flashrom.org with FAILED: your board name in the subject line!
-------------------------------------------------------------------------------
DO NOT REBOOT OR POWEROFF!
ERROR: Execution FAILED.
ERROR: Execution failed: ./updater4.sh (error code = 1)
Finished after 106 seconds.
Failed Command: //usr/sbin/chromeos-firmwareupdate --mode=autoupdate - Exit Code 1
Firmware update failed (error code: 1).
Rolling back update due to failure installing required firmware.
Successfully updated GPT with all settings to rollback.
PostInstall Failed

[0208/143846:ERROR:postinstall_runner_action.cc(291)] Postinst command failed with code: 1
[0208/143846:ERROR:postinstall_runner_action.cc(328)] Postinstall action failed.
[0208/143846:INFO:action_processor.cc(116)] ActionProcessor: finished PostinstallRunnerAction with code ErrorCode::kPostinstallRunnerError
[0208/143846:INFO:action_processor.cc(121)] ActionProcessor: Aborting processing due to failure.
[0208/143846:INFO:update_attempter.cc(909)] Processing Done.
[0208/143846:INFO:cpu_limiter.cc(71)] Setting cgroup cpu shares to  1024
[0208/143846:INFO:cpu_limiter.cc(79)] CPU shares = 1024
[0208/143846:ERROR:update_attempter.cc(1333)] Update failed.
[0208/143846:INFO:payload_state.cc(247)] Updating payload state for error code: 5 (ErrorCode::kPostinstallRunnerError)

Deleted: update_engine.20180208-142851.txt 78.4 KB

update_engine.20180208-142851.txt 78.4 KB View Download

Can we try this on a different Lars device?

I don't think we have done much to change Lars specifically in a while, the message sounds like it tried to write to some space it could not, my initial guess is that this particular Lars may have a different ME version on it which is locked. 

...
VERIFY FAILED at 0x000001f5! Expected=0x84, Read=0x82, failed byte count from 0x00000000-0x00ffffff: 0x1
...

I think this is underlying the issue:

./updater4.sh: 46: ./updater4.sh: cros_spi_descriptor: not found
./updater4.sh: 61: ./updater4.sh: cros_spi_descriptor: not found

I can't see anything that would have changed to make cros_spi_descriptor not exist in /usr/sbin as it comes from chipset-skl-private overlay and doesn't look to have changed.

Perhaps instead what is happening is the PATH is messed up and running cros_spi_descriptor without a fully qualified path is failing?

Tried Recovery and AU on two different devices and both gets failed with same error.

I downloaded 10323.27.0 lars image and can't find cros_spi_descriptor inside.

Seems like the problem is chromeos-base/chromeos-bsp-chipset-skl-private is added by overlay-glados-private/virtual/chromeos-bsp/chromeos-bsp-3.ebuild, but not overlay-kunimitsu.

And we didn't see this because firmware updater used to add that explicitly (inside bundle), but we don't do it anymore.

A fix here https://chrome-internal-review.googlesource.com/#/c/chromeos/overlays/overlay-kunimitsu-private/+/567439/

This bug requires manual review: M65 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Still able to reproduce this issue on M65 build 10323.30.0

Re#11 CLs haven't merged. Hope the merge can happen today.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-kunimitsu-private/+/f1ed5820332c16854529643031179ff4da2f34da

commit f1ed5820332c16854529643031179ff4da2f34da
Author: Hung-Te Lin <hungte@chromium.org>
Date: Sat Feb 10 04:51:28 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-kunimitsu-private/+/3b3bf6fbc42ba1f1692f08aa85982d1e4d1a0566

commit 3b3bf6fbc42ba1f1692f08aa85982d1e4d1a0566
Author: Hung-Te Lin <hungte@chromium.org>
Date: Tue Feb 13 02:59:31 2018

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

 Issue 815243  has been merged into this issue.

Able to reproduce the issue on Chrome 65.0.3325.114/ CrOS 10323.48.0 - Lars. I'm not able to install Recovery image with unexpected error message.

Please paste the error message from recovery error logs. Sorry that I don't have lars in hand.

Ok I found the problem...

kunimitsu layout is slightly different from glados.
I thought building kunimitsu would be enough, but seems like lars is not taking bsp from kunimitsu.

So we have to add three CLs into all kunimitsu boards, each adding skl-private into bsp.

Changes in
 https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-lars-private/+/584113
 https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-asuka-private/+/584076
 https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-sentry-private/+/584075

I'll wait for these to be merged on ToT before sending them to R65.

+vapier

Hi Vapier, I found that I did put the cros_spi_descriptor and additional programs into firmware updater itself, so it should not be a problem even without the changes in #23 for AU.

However, after you change in https://chromium-review.googlesource.com/798773, the firmware updater now only takes bin/ folder into path.

These ebuilds currently add extra programs by

CROS_FIRMWARE_EXTRA_LIST=\
"${SYSROOT}/bin/xxd;${SYSROOT}/usr/sbin/cros_spi_descriptor"

Now, my question is, how do we add programs into the new bin/ folder in ebuild file?

I think we need a quick fix like https://chromium-review.googlesource.com/#/c/chromiumos/platform/firmware/+/952725/.


In future we should add one setting for putting files into bin/ folder.

let's merge 952725 first, which "should" solve most cases...

This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-asuka-private/+/c7c01608f91135575fb8ded430be903f57a4b0ee

commit c7c01608f91135575fb8ded430be903f57a4b0ee
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Mar 07 18:50:40 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-lars-private/+/4daf9142c2dd708c98352160be5c0c56fc170bdb

commit 4daf9142c2dd708c98352160be5c0c56fc170bdb
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Mar 07 18:50:39 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-sentry-private/+/16eca315426be1a237e44eb8cde1af36cb6a9c33

commit 16eca315426be1a237e44eb8cde1af36cb6a9c33
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Mar 07 18:50:36 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/afee678ecc49609f4b6e4b1f1e48b62da658769e

commit afee678ecc49609f4b6e4b1f1e48b62da658769e
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 00:55:02 2018

pack_stub: Always append updater content top folder to PATH.

Many ebuild files are still adding scripts or binary programs into
updater by CROS_FIRMWARE_EXTRA_LIST, which collects files into top
folder. Starting from CL:798773, the additional programs must be added
into bin/ folder.

For files downloaded from BCS, we can create additional folder structure
but for files picked from chroot (i.e., /build/$BOARD/usr/bin/some_file),
there is no easy way to specify the files.

The binaries and data should still live in separate places.  The lddtree
conversion may make it a moot point as it may auto relocate ELF
binaries.

So a long term solution would be adding some other eclass settings to
specify "additional programs to add".

But for now, in order to prevent fixing all existing ebuild files, we
are adding the top folder to PATH for execution as a short term
solution.

BUG= chromium:810541 
TEST=chromeos-firmwareupdate --mode=autoupdate

Change-Id: I6845f8bf9e44769fef0aaaf4c084a3f93e1505aa
Reviewed-on: https://chromium-review.googlesource.com/952725
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/afee678ecc49609f4b6e4b1f1e48b62da658769e/pack_stub

ToT has all changes needed merged.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/4d71ebfe2d269e2e1e31e70d7f5aa3666bc7b4f9

commit 4d71ebfe2d269e2e1e31e70d7f5aa3666bc7b4f9
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 02:53:01 2018

pack_stub: Always append updater content top folder to PATH.

Many ebuild files are still adding scripts or binary programs into
updater by CROS_FIRMWARE_EXTRA_LIST, which collects files into top
folder. Starting from CL:798773, the additional programs must be added
into bin/ folder.

For files downloaded from BCS, we can create additional folder structure
but for files picked from chroot (i.e., /build/$BOARD/usr/bin/some_file),
there is no easy way to specify the files.

The binaries and data should still live in separate places.  The lddtree
conversion may make it a moot point as it may auto relocate ELF
binaries.

So a long term solution would be adding some other eclass settings to
specify "additional programs to add".

But for now, in order to prevent fixing all existing ebuild files, we
are adding the top folder to PATH for execution as a short term
solution.

BUG= chromium:810541 
TEST=chromeos-firmwareupdate --mode=autoupdate

Change-Id: I6845f8bf9e44769fef0aaaf4c084a3f93e1505aa
Reviewed-on: https://chromium-review.googlesource.com/952725
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit afee678ecc49609f4b6e4b1f1e48b62da658769e)
Reviewed-on: https://chromium-review.googlesource.com/954382
Trybot-Ready: Hung-Te Lin <hungte@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>

[modify] https://crrev.com/4d71ebfe2d269e2e1e31e70d7f5aa3666bc7b4f9/pack_stub

Your change meets the bar and is auto-approved for M66. Please go ahead and merge the CL to branch 3359 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), josafat@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-asuka-private/+/462f8ea28c69fd4b0d8c39ef2d4e1848a44dce91

commit 462f8ea28c69fd4b0d8c39ef2d4e1848a44dce91
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 06:26:29 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-sentry-private/+/42fe8deb78ec1d2d7796dc089d00f333b705ae40

commit 42fe8deb78ec1d2d7796dc089d00f333b705ae40
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 06:26:29 2018

Verified build 10470 that the file does exist in rootfs for sentry and asuka. Waiting for lars to finish building.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/0967b8bd0b0598edeb87cc95b3f6411014c3f17f

commit 0967b8bd0b0598edeb87cc95b3f6411014c3f17f
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 08:59:28 2018

pack_stub: Always append updater content top folder to PATH.

Many ebuild files are still adding scripts or binary programs into
updater by CROS_FIRMWARE_EXTRA_LIST, which collects files into top
folder. Starting from CL:798773, the additional programs must be added
into bin/ folder.

For files downloaded from BCS, we can create additional folder structure
but for files picked from chroot (i.e., /build/$BOARD/usr/bin/some_file),
there is no easy way to specify the files.

The binaries and data should still live in separate places.  The lddtree
conversion may make it a moot point as it may auto relocate ELF
binaries.

So a long term solution would be adding some other eclass settings to
specify "additional programs to add".

But for now, in order to prevent fixing all existing ebuild files, we
are adding the top folder to PATH for execution as a short term
solution.

BUG= chromium:810541 
TEST=chromeos-firmwareupdate --mode=autoupdate

Change-Id: I6845f8bf9e44769fef0aaaf4c084a3f93e1505aa
Reviewed-on: https://chromium-review.googlesource.com/952725
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
(cherry picked from commit afee678ecc49609f4b6e4b1f1e48b62da658769e)
Reviewed-on: https://chromium-review.googlesource.com/954962
Trybot-Ready: Hung-Te Lin <hungte@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>

[modify] https://crrev.com/0967b8bd0b0598edeb87cc95b3f6411014c3f17f/pack_stub

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-sentry-private/+/2fed5acf74981ba4b8f8573e59451623cdd1401c

commit 2fed5acf74981ba4b8f8573e59451623cdd1401c
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 09:38:47 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-asuka-private/+/211816f8d39b3dce42aa92aa4b0977322bf202dc

commit 211816f8d39b3dce42aa92aa4b0977322bf202dc
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Mar 08 09:38:47 2018

last lars build failed, so I've merged patches of sentry, asuka to ToT, R65, R66.

Probably need one more day before I can validate lars build contents.

But with 952725 (already merged), recovery and AU should both work.

The lars-specific CL is for fixing "board-setgoodfirmware" (after AU).

ToT lars builder has been not generating output since 10470 so I can't verify that, but asuka and sentry looks good so I'll push lars changes for R65 now and see if I can verify the output there.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-lars-private/+/e64692a11d26ad452255cfe98b3c40b0f31a11d2

commit e64692a11d26ad452255cfe98b3c40b0f31a11d2
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Mar 09 07:49:43 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-lars-private/+/c52aff67e39f7de96cc7caab3510373d6599fadb

commit c52aff67e39f7de96cc7caab3510373d6599fadb
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Mar 09 07:49:50 2018

verified the fix on Chrome 65.0.3325.150/CrOS 10323.53.0 - Lars