If autoupdate is successful, this should be marked as Stable blocker.

successful auto updated to 8838.0.0 on falco.

Do you have the logs handy from the failed recovery attempt?

We had some other recovery failures recently involving Flashrom, and I am wondering if this might be the same thing, they should be fixed though, the logs can tell us.

Recovery stick does not contain recovery logs.
After inserting recovery stick:
- The device turned black, & then return with dev screen
- Recovery process never start 

This issue is similar to https://code.google.com/p/chrome-os-partner/issues/detail?id=57345

Issue is still seen in M55: 55.0.2874.0/8848.0.0 (Official Build) dev-channel Falco

Note: Auto update to latest Dev channel is also not working now

Still able to reproduce this issue on M55 build 8872.5.0

Is this related to the recovery app ?

Nope. This is pure recovery mechanism. No Recovery tool needed to re-image the stick.

Still reproducible in ChromeOS: 8872.15.0/55.0.2883.17 on Falco-li

Issue is still reproducible with the build M56: 56.0.2895.0/8918.0.0 dev-channel Falco 

Issue is still seen in 56.0.2890./8928.0.0 dev -channel Wolf, Gnawty, Glimmer, Zako as well.

@Can some one from Dev team please look into this issue.

+reinauer

Stefan, it sounds like per comment 5 we may not be making it past the firmware portion of the recovery process, whom is the right person to dig into this?

If we have no logs otherwise, this might require a serial enabled firmware :-/. 

It could be firmware or early kernel. In my experience it's usually the latter... once the firmware turns off the screen there's very little left that could go wrong.

Next step would be to run on Servo with both serial-enabled firmware and earlycon kernel. Command line arguments can either be injected with a small firmware patch (just a strcpy() at the end of depthcharge/src/boot/commandline.c) or by reciting the correct futility incantations over the recovery stick's kernel partition.

 Issue 658649  has been merged into this issue.

Shelley is debugging right now with one falco device from FW team.

Shelley, assigning this to you as you're looking into it. Please pass along to right owner if need be.

Tracking M56 dev blocker version of this bug in crbug/658649

If this is only in recovery, we can punt to stable for now.

THis looks like a duplicate of https://code.google.com/p/chrome-os-partner/issues/detail?id=58976

but that is in the partner bug database, so I cannot duplicate..

We are working on this pretty hard. We need at most a couple of days to find the culprit.

According to the labels here, I see this does not block dev releases. 
So, is it ok to take a couple of days to fix the issue before we have to revert the change for LLVM migration?

Issue is still seen in M56: 56.0.2899.0/8934.0.0 dev-channel Gnawty, Paine, Falco, Butterfly, Samus and Link as well

bleung@ says we are blocking dogfooding. 
So, we will revert the LLVM AMD64 roll for now.
revert at: https://chromium-review.googlesource.com/#/c/403288/

Updating it to ReleaseBlock-Dev as we are unable to work on many devices on ToT.
Feel free to update if not appropriate.

After chatting offline with Kalin, agreed that this is not release block dev for M56 as recovery sticks isn't the usual way to get build. 

Re #28 - Thanks for pushing the priority high. It seems shchen@ and team are working on it.

I want to propose ReleaseBlock-Beta, so to reflect on the dogfooding being blocked, and the variety of devices where recovery is failing.

Updating ownership to the proper people.

note that by #27, we have reverted the change. 
So, at the moment this is not blocking anything.
We are working on identifying the culprit.

Recovery images after and including 8935.0 should no longer have this issue. Please take a look.

Note that LLVM was rolled early on Falco on M-55, so this may still be a releaseblock-beta... Busybox is the culprit right now, so inorder to unbreak the falco recovery images on 55, this needs to be picked back : https://chromium-review.googlesource.com/#/c/404148/

The change in comment 38 appears to cherry-pick cleanly, if we are confident this will resolve the problem in 55 let us go ahead with it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/b97dbb3b367295d7638b40bc9b60e0801a037c77

commit b97dbb3b367295d7638b40bc9b60e0801a037c77
Author: Yunlian Jiang <yunlian@chromium.org>
Date: Wed Oct 26 23:04:11 2016

use gcc to build busybox.

Building busybox with llvm breaks recovery image.
So we use gcc to build it instead.

BUG= chromium:650227 
TEST=the recovery image works.

Change-Id: Ibd37ab79ad315451ed0863fe7f7b2974930c7e7e
Reviewed-on: https://chromium-review.googlesource.com/403437
Reviewed-by: Benson Leung <bleung@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[add] https://crrev.com/b97dbb3b367295d7638b40bc9b60e0801a037c77/chromeos/config/env/sys-apps/busybox

thanks Bernie!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/d3c29bcdb5370e0b76eeee20a1040b4605cf300a

commit d3c29bcdb5370e0b76eeee20a1040b4605cf300a
Author: Yunlian Jiang <yunlian@chromium.org>
Date: Wed Oct 26 23:04:11 2016

use gcc to build busybox.

Building busybox with llvm breaks recovery image.
So we use gcc to build it instead.

BUG= chromium:650227 
TEST=the recovery image works.

Change-Id: Ibd37ab79ad315451ed0863fe7f7b2974930c7e7e
Reviewed-on: https://chromium-review.googlesource.com/404148
Reviewed-by: Benson Leung <bleung@chromium.org>
Commit-Queue: Yunlian Jiang <yunlian@chromium.org>
Tested-by: Yunlian Jiang <yunlian@chromium.org>

[add] https://crrev.com/d3c29bcdb5370e0b76eeee20a1040b4605cf300a/chromeos/config/env/sys-apps/busybox

Since we landed the change, assuming this no longer blocks 55 beta.

Thank you all to look into this on priority, will update it once we have latest M56 ToT and R55-8872.B available with the Fix landed.

Able to recover falco-li/samus/link/peppy with USB stick in 8872.27.0/55.0.2883.29

Recovery is successful with the below devices
1. M55: 55.0.2883.29/8872.27.0 dev-channel Falco, Gnawty,  Butterfly, Peppy, Paine, Link, Glimmer, Samus
2. M56: 56.0.2900.0/8940.0.0 dev-channel auron_paine, Peach-Pit, Jerry, Falco, Samus, Peppy

Thanks!!!

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

yunlian@ : Could you please update the Status to Fixed as its already verified.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This was merged as
https://chromium-review.googlesource.com/#/c/403437/

#CBC-RS/TC-watchlist