Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 17 users
Status: Verified
Owner:
Closed: Jun 27
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug-Regression

Blocking:
issue 736000
issue 736812
issue 736871



Sign in to add a comment
Black screen after device AutoUpdate (device unusable) and initiating recovery
Project Member Reported by sdantul...@chromium.org, Jun 26 Back to list
Build: 	9686.0.0, 61.0.3138.0 caroline

What steps will reproduce the problem?
1. AU device to 9686.0.0

What happens instead?
Black screen after device boot.
 
Cc: vsu...@chromium.org
Labels: ReleaseBlock-Dev
observed issue on guado and samus.
Cc: kmshelton@chromium.org
Components: Internals>Installer
Cc: rjahagir@chromium.org
Components: -Internals>Installer
Observed the same issue on cave device as well
Cc: ka...@chromium.org sontis@chromium.org
Components: OS>Kernel Internals>Installer
observed same issue on Caroline, Samus, and Cyan.
Cc: puthik@chromium.org tbroch@chromium.org
Components: OS>Firmware
Labels: -Pri-1 Pri-0
Recovery is also broken, having USB stick plugged, boards are hanging on maximized backlight black screen. USB stick LLED inactive.
Cc: keta...@chromium.org
*LED inactive.
Not all devices affected but lots of boards - auron, glados, strago, veyron, daisy. 
(Successful recovery done on reef and kevin)
Summary: Black screen after device AutoUpdate (device unusable) and initiating recovery (was: Black screen after device AutoUpdate (device unusable))
Cc: -ka...@chromium.org -abodenha@chromium.org haddowk@chromium.org krk@chromium.org
guado moblab is also affected - all moblabs in Stlg Ct are down
Cc: ka...@chromium.org abodenha@chromium.org
Cc: gkihumba@chromium.org
works fine on Kevin, Minnie, Elm, and Eve.
Cc: allendam@chromium.org
Recovery broken seen in Paine device.
Seems related to  crbug.com/736812  ?
Other boards with good recovery - Falco, Peach-pit, Daisy, Banjo, Nyan_blaze.
(In c#8 I posted the board family names. With 'daisy' I meant daisy-spring).
Re #16 - I do not think they are related, but some observations in  issue 736812  might be.
> Seems related to  crbug.com/736812  ?

I expect so.   Bug 736812  is showing that during at least one update
test, DUTs go offline can only be fixed by reinstalling from USB
(i.e. recovery).  This bug seems to describe pretty much the same
symptom, but from a different perspective.

The time frame and affected boards seems to match up, too.  The
paygen failures started at around 9679.0.0, and this bug is for
9686.0.0.  And cave and caroline both seem affected.

Issue 736871 has been merged into this issue.
Blocking: 736871
same issue on kefka, lulu.
Maybe related to  crbug.com/736000  (teravest@ pinged me)

I've kicked off a trybot with a revert of the only CL that looks plausible

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=sjg@chromium.org&builder=release

Owner: sjg@chromium.org
Thanks, assigning to you since this is P0 and needs an owner 
ok great. how soon can we get it in ToT? We need a good solid green upreved build by tomorrow morning.
Well I'm not even sure it is the problem.

 crbug.com/736000  seems similar but is marked fixed

9678 is bootable while 9679 starts to fail.
Summary so far from my understanding (please chime in with other info):

- Lots of discussion on go/crosoncall
- List of CLs that are in 9679.0.0 but 9678.0.0 seems innocuous
   - https://crosland.corp.google.com/log/9678.0.0..9679.0.0
   - cernekee tried a revert of dbus-glib to no avail
   - chrome and arc++ versions are the same
   - seems to die very early (black screen)
   - kirtika pointed to https://chromium-review.googlesource.com/c/545104/7/drivers/gpu/drm/i915/intel_dp.c
   - assuming we can rely on the tools, this was not new in this release
- This toolchain CL almost corresponds:
   - https://chromium-review.googlesource.com/c/486144
   - It landed in 9677.0.0
   - See https://crosland.corp.google.com/cl?q=chromium:486144
   - That is a little early, but perhaps there is something funny going on? Did the kernel perhaps not get rebuilt?

- eve seems to fail but soraka (same SoC) is happy
   - But that appears to just be because of missing hardware:
   - https://uberchromegw.corp.google.com/i/chromeos/builders/soraka-release/builds/199/steps/PaygenTestDev/logs/stdio

Blocking: 736812
I believe we can rule out firmware update? Was there a firmware update in the lab?
Components: -OS>Firmware
Cc: llozano@chromium.org
+Luis for toolchain question

Revert is here:

https://chromium-review.googlesource.com/c/547575

Propose to push this in the hope of a fix (since we presumably have nothing to lose)
Cc: cernekee@chromium.org
preparing revert.

Blocking: 736000
Hmmm, the latest eve-release (this morning) passed.  crbug.com/736000  as mentioned above.

But many other builds still fail.

> I believe we can rule out firmware update? Was there a firmware update in the lab?

The lab only runs firmware that's been released to customers.
Generally, if there's an update, it will happen Tuesday morning
after the Beta build with the update is released to users.

Last Tuesday was too early for the failures we're seeing, and
although there were updates, there's not much correspondence
with affected boards:
     snappy                 Google_Snappy.9042.87.1 -> Google_Snappy.9042.110.0
     sand                   Google_Sand.9042.87.0 -> Google_Sand.9042.110.0
     asuka                  Google_Asuka.7820.257.0 -> Google_Asuka.7820.306.0
     elm                    Google_Elm.8438.73.0 -> Google_Elm.8438.85.0
     ultima                 Google_Ultima.7287.131.63 -> Google_Ultima.7287.131.85
     pyro                   Google_Pyro.9042.87.1 -> Google_Pyro.9042.110.0

chumped https://chromium-review.googlesource.com/c/547575/

but there is another CL needed to revert the SDK. we are working on it.

PLEASE note that we reverting binutils upgrade just to be conservative. We have not verified this is the cause of the problem. 

The one data point we have is:

The 9678 cyan kernel is good
The 9679 cyan kernel does not boot
There were no commits in the kernel tree between 9678 and 9679
There are differences in the generated kernel code between 9678 and 9679
If this is related to  crbug.com/736000  eve

Good: R61-9678.0.0, Bad: R61-9679.0.0
FWIW canary update of eve from 9669 -> 9686 did NOT fail (1 of 1 tested)
Reports on CBC with Acer CB5-311 - big board.
https://productforums.google.com/forum/#!topic/chromebook-central/I0Xo_BvUFWs

#CBC-RS/TC-watchlist
It looks like the live dev-channel build has this problem on many boards. I don't know if it possible to roll back to 9679.0.0?
..but the 2am PT build 9690.0.0 still seems to be broken.

Status: Started
Actually 9690.0.0 seems better now, auron and strago boards seems to be working.

So perhaps the toolchain was the problem. I don't know of any other action taken.

Waiting on release team before deciding whether this is fixed, or needs more action. Builds look pretty green in GoldenEye.
ketakid - things are looking pretty good now.
Status: Fixed
Marking this fixed since the toolchain revert.
Status: Verified
AU works fine on Caroline and Cyan.
Verified on M61-9690.0.0
TestEng team verified on several boards the AU and recovery is successful.
I updated from 61.0.3138.0 (canary) on Kevin today and seeing what seems to be a similar black screen issue. 

I'm fortunately in dev mode, was able to open the dev console and change my channel.
On kevin device:

cros flash usb:// xbuddy://remote/kevin/latest-canary and install from usb.

Black screen seems to be an ongoing issue. A slight difference is that chrome logo is up before screen goes to black.
I believe this is a different bug since you get a display up. Can you file a bug for that?
Issue 738242 has been merged into this issue.
Sign in to add a comment