New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 17 users

Issue metadata

Status: Verified
Closed: Jun 2017
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug-Regression

issue 736000
issue 736812
issue 736871

Sign in to add a comment

Black screen after device AutoUpdate (device unusable) and initiating recovery

Project Member Reported by, Jun 26 2017 Back to list

Issue description

Build: 	9686.0.0, 61.0.3138.0 caroline

What steps will reproduce the problem?
1. AU device to 9686.0.0

What happens instead?
Black screen after device boot.
Labels: ReleaseBlock-Dev
observed issue on guado and samus.
Components: Internals>Installer
Components: -Internals>Installer
Observed the same issue on cave device as well
Components: OS>Kernel Internals>Installer
observed same issue on Caroline, Samus, and Cyan.

Comment 6 by, Jun 26 2017

Components: OS>Firmware
Labels: -Pri-1 Pri-0
Recovery is also broken, having USB stick plugged, boards are hanging on maximized backlight black screen. USB stick LLED inactive.

Comment 7 by, Jun 26 2017


Comment 8 by, Jun 26 2017

*LED inactive.
Not all devices affected but lots of boards - auron, glados, strago, veyron, daisy. 
(Successful recovery done on reef and kevin)

Comment 9 by, Jun 26 2017

Summary: Black screen after device AutoUpdate (device unusable) and initiating recovery (was: Black screen after device AutoUpdate (device unusable))
guado moblab is also affected - all moblabs in Stlg Ct are down

Comment 12 by, Jun 26 2017

works fine on Kevin, Minnie, Elm, and Eve.
Recovery broken seen in Paine device.

Comment 16 by, Jun 26 2017

Seems related to  ?

Comment 17 by, Jun 26 2017

Other boards with good recovery - Falco, Peach-pit, Daisy, Banjo, Nyan_blaze.
(In c#8 I posted the board family names. With 'daisy' I meant daisy-spring).

Comment 18 by, Jun 26 2017

Re #16 - I do not think they are related, but some observations in  issue 736812  might be.
> Seems related to  ?

I expect so.   Bug 736812  is showing that during at least one update
test, DUTs go offline can only be fixed by reinstalling from USB
(i.e. recovery).  This bug seems to describe pretty much the same
symptom, but from a different perspective.

The time frame and affected boards seems to match up, too.  The
paygen failures started at around 9679.0.0, and this bug is for
9686.0.0.  And cave and caroline both seem affected.

Issue 736871 has been merged into this issue.
Blocking: 736871
same issue on kefka, lulu.

Comment 23 by, Jun 26 2017

Maybe related to  (teravest@ pinged me)

I've kicked off a trybot with a revert of the only CL that looks plausible

Thanks, assigning to you since this is P0 and needs an owner 
ok great. how soon can we get it in ToT? We need a good solid green upreved build by tomorrow morning.

Comment 26 by, Jun 26 2017

Well I'm not even sure it is the problem.  seems similar but is marked fixed

9678 is bootable while 9679 starts to fail.

Comment 28 by, Jun 26 2017

Summary so far from my understanding (please chime in with other info):

- Lots of discussion on go/crosoncall
- List of CLs that are in 9679.0.0 but 9678.0.0 seems innocuous
   - cernekee tried a revert of dbus-glib to no avail
   - chrome and arc++ versions are the same
   - seems to die very early (black screen)
   - kirtika pointed to
   - assuming we can rely on the tools, this was not new in this release
- This toolchain CL almost corresponds:
   - It landed in 9677.0.0
   - See
   - That is a little early, but perhaps there is something funny going on? Did the kernel perhaps not get rebuilt?

- eve seems to fail but soraka (same SoC) is happy
   - But that appears to just be because of missing hardware:

Comment 29 by, Jun 26 2017

Blocking: 736812

Comment 30 by, Jun 26 2017

I believe we can rule out firmware update? Was there a firmware update in the lab?

Comment 31 by, Jun 26 2017

Components: -OS>Firmware

Comment 32 by, Jun 26 2017

+Luis for toolchain question

Revert is here:

Propose to push this in the hope of a fix (since we presumably have nothing to lose)

Comment 33 by, Jun 26 2017


Comment 34 by, Jun 26 2017

preparing revert.

Comment 35 by, Jun 26 2017

Blocking: 736000
Hmmm, the latest eve-release (this morning) passed.  as mentioned above.

But many other builds still fail.

> I believe we can rule out firmware update? Was there a firmware update in the lab?

The lab only runs firmware that's been released to customers.
Generally, if there's an update, it will happen Tuesday morning
after the Beta build with the update is released to users.

Last Tuesday was too early for the failures we're seeing, and
although there were updates, there's not much correspondence
with affected boards:
     snappy                 Google_Snappy.9042.87.1 -> Google_Snappy.9042.110.0
     sand                   Google_Sand.9042.87.0 -> Google_Sand.9042.110.0
     asuka                  Google_Asuka.7820.257.0 -> Google_Asuka.7820.306.0
     elm                    Google_Elm.8438.73.0 -> Google_Elm.8438.85.0
     ultima                 Google_Ultima.7287.131.63 -> Google_Ultima.7287.131.85
     pyro                   Google_Pyro.9042.87.1 -> Google_Pyro.9042.110.0

Comment 37 by, Jun 27 2017


but there is another CL needed to revert the SDK. we are working on it.

Comment 38 by, Jun 27 2017

PLEASE note that we reverting binutils upgrade just to be conservative. We have not verified this is the cause of the problem. 

The one data point we have is:

The 9678 cyan kernel is good
The 9679 cyan kernel does not boot
There were no commits in the kernel tree between 9678 and 9679
There are differences in the generated kernel code between 9678 and 9679
If this is related to  eve

Good: R61-9678.0.0, Bad: R61-9679.0.0
FWIW canary update of eve from 9669 -> 9686 did NOT fail (1 of 1 tested)
Reports on CBC with Acer CB5-311 - big board.!topic/chromebook-central/I0Xo_BvUFWs


Comment 43 by, Jun 27 2017

It looks like the live dev-channel build has this problem on many boards. I don't know if it possible to roll back to 9679.0.0?

Comment 45 by, Jun 27 2017

..but the 2am PT build 9690.0.0 still seems to be broken.

Comment 46 by, Jun 27 2017

Status: Started
Actually 9690.0.0 seems better now, auron and strago boards seems to be working.

So perhaps the toolchain was the problem. I don't know of any other action taken.

Comment 47 by, Jun 27 2017

Waiting on release team before deciding whether this is fixed, or needs more action. Builds look pretty green in GoldenEye.

Comment 48 by, Jun 27 2017

ketakid - things are looking pretty good now.

Comment 49 by, Jun 27 2017

Status: Fixed
Marking this fixed since the toolchain revert.
Status: Verified
AU works fine on Caroline and Cyan.
Verified on M61-9690.0.0

Comment 51 by, Jun 27 2017

TestEng team verified on several boards the AU and recovery is successful.

Comment 52 by, Jun 28 2017

I updated from 61.0.3138.0 (canary) on Kevin today and seeing what seems to be a similar black screen issue. 

I'm fortunately in dev mode, was able to open the dev console and change my channel.
On kevin device:

cros flash usb:// xbuddy://remote/kevin/latest-canary and install from usb.

Black screen seems to be an ongoing issue. A slight difference is that chrome logo is up before screen goes to black.

Comment 54 by, Jun 28 2017

I believe this is a different bug since you get a display up. Can you file a bug for that?
Issue 738242 has been merged into this issue.

Sign in to add a comment