Do you have recovery log?

recovery log: https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cr/664232/

From recovery logs:
==================================
+ tee -a /run/frecon/vt1 /run/frecon/vt2
 .+ cgpt find -1 -u 586f5ae0-ecd2-794a-95bf-2ac9426ac836
ERROR: Can't read enough: -1, not 512
ERROR: Cannot read primary GPT header
WARNING: Primary GPT header is invalid
WARNING: Secondary GPT header is invalid
WARNING: Primary GPT header is invalid
WARNING: Secondary GPT header is invalid
flashrom v0.9.4  : 65be03a : Nov 08 2016 10:51:57 UTC on Linux 4.4.21-06015-ge301df2 (aarch64)
flashrom v0.9.4  : 65be03a : Nov 08 2016 10:51:57 UTC on Linux 4.4.21-06015-ge301df2 (aarch64)
Invalid entry specified: RW_GPT
ERROR: Cannot exec flashrom to read from RW_GPT section.
ReadNorFlash: Is a directory
+ kern=
+ newroot=3
+ [ -b 3 ]
+ sleep 1
+ dlogf  .
+ + printftee  .
 -a /run/frecon/vt1 /run/frecon/vt2
 .+ cgpt find -1 -u 586f5ae0-ecd2-794a-95bf-2ac9426ac836
ERROR: Can't read enough: -1, not 512
ERROR: Cannot read primary GPT header
===================================

Recovery still failed in 8978.0.0/56.0.2915.0. Tested kevin/peppy. 

Assigning to yunlian@ as you fixed a similar issue: crbug/650227
Please pass along to right owner if not you, thanks

Is this happening on ARM boards? If not, it's unlikely a toolchain problem and we have better get experts of recovery involved as early as possible.

Sorry for the typo...

Is this also happening on ALL ARM boards? If YES, it's unlikely a toolchain problem and we have better get experts of recovery involved as early as possible.

We only rolled all amd64, veyron_minnie and nyan_big to use clang/llvm. Most arm boards are built by gcc currently.

Looks like this happen on ARM (kevin) and Intel (peppy)

I will try to create a recover image built by gcc to see whether this is related to toolchain changes.

To #11, it should not be related to toolchain. We did not touch arm toolchain recently.

Issue 664407 has been merged into this issue.

This is a serious regression. Can we get more eyes on this one? Someone needs to bisect changes from 8975 to 8977.

The samus_recovery.log shows it failed here:

+ chroot /usb /usr/sbin/chromeos-recovery /dev/sdb
.: 843: Can't open /tmp/tmp.PLBBO99HQC/usr/sbin/write_gpt.sh
+ local install_status=127
+ [ 127 -ne 0 ]
+ dlog WARNING!!! Installation of software failed. Displaying hw diagnostics


This change is suspicious: https://chromium.googlesource.com/chromiumos/platform2/+/4c5a7cb1c287275d72604ff33b7a5093c00d795f%5E%21/#F0

if it's recent, prob want to revert:
  https://chromium-review.googlesource.com/405031

Looks good. Reverting.

Revert here : https://chromium-review.googlesource.com/#/c/410166/

I've confirmed using a release trybot recovery image that the revert above is effective. Sending it through the CQ again now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/495cfbc122c9f3f14324195ca03a749c38ec54b0

commit 495cfbc122c9f3f14324195ca03a749c38ec54b0
Author: Benson Leung <bleung@chromium.org>
Date: Fri Nov 11 18:45:01 2016

Revert "installer: Load GPT helper functions before using DEFAULT_ROOTDEV"

This reverts commit 4c5a7cb1c287275d72604ff33b7a5093c00d795f.

BUG= chromium:664232 
TEST=Recovery works again.

Change-Id: I5f278248873bf91491841477380d22146487a5b5
Reviewed-on: https://chromium-review.googlesource.com/410166
Commit-Ready: Benson Leung <bleung@chromium.org>
Tested-by: Benson Leung <bleung@chromium.org>
Reviewed-by: Benson Leung <bleung@chromium.org>
Reviewed-by: Alexis Savery <asavery@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/495cfbc122c9f3f14324195ca03a749c38ec54b0/installer/chromeos-install

8983.0 and later have the revert. Recovery should be working after that point on all boards again.

verified in 8989.0.0 / 56.0.2919.0.