observed issue on guado and samus.

Observed the same issue on cave device as well

observed same issue on Caroline, Samus, and Cyan.

Recovery is also broken, having USB stick plugged, boards are hanging on maximized backlight black screen. USB stick LLED inactive.

*LED inactive.
Not all devices affected but lots of boards - auron, glados, strago, veyron, daisy. 
(Successful recovery done on reef and kevin)

guado moblab is also affected - all moblabs in Stlg Ct are down

works fine on Kevin, Minnie, Elm, and Eve.

Recovery broken seen in Paine device.

Seems related to  crbug.com/736812  ?

Other boards with good recovery - Falco, Peach-pit, Daisy, Banjo, Nyan_blaze.
(In c#8 I posted the board family names. With 'daisy' I meant daisy-spring).

Re #16 - I do not think they are related, but some observations in  issue 736812  might be.

> Seems related to  crbug.com/736812  ?

I expect so.   Bug 736812  is showing that during at least one update
test, DUTs go offline can only be fixed by reinstalling from USB
(i.e. recovery).  This bug seems to describe pretty much the same
symptom, but from a different perspective.

The time frame and affected boards seems to match up, too.  The
paygen failures started at around 9679.0.0, and this bug is for
9686.0.0.  And cave and caroline both seem affected.

Issue 736871 has been merged into this issue.

same issue on kefka, lulu.

Maybe related to  crbug.com/736000  (teravest@ pinged me)

I've kicked off a trybot with a revert of the only CL that looks plausible

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=sjg@chromium.org&builder=release

Thanks, assigning to you since this is P0 and needs an owner 

ok great. how soon can we get it in ToT? We need a good solid green upreved build by tomorrow morning.

Well I'm not even sure it is the problem.

 crbug.com/736000  seems similar but is marked fixed

9678 is bootable while 9679 starts to fail.

Summary so far from my understanding (please chime in with other info):

- Lots of discussion on go/crosoncall
- List of CLs that are in 9679.0.0 but 9678.0.0 seems innocuous
   - https://crosland.corp.google.com/log/9678.0.0..9679.0.0
   - cernekee tried a revert of dbus-glib to no avail
   - chrome and arc++ versions are the same
   - seems to die very early (black screen)
   - kirtika pointed to https://chromium-review.googlesource.com/c/545104/7/drivers/gpu/drm/i915/intel_dp.c
   - assuming we can rely on the tools, this was not new in this release
- This toolchain CL almost corresponds:
   - https://chromium-review.googlesource.com/c/486144
   - It landed in 9677.0.0
   - See https://crosland.corp.google.com/cl?q=chromium:486144
   - That is a little early, but perhaps there is something funny going on? Did the kernel perhaps not get rebuilt?

- eve seems to fail but soraka (same SoC) is happy
   - But that appears to just be because of missing hardware:
   - https://uberchromegw.corp.google.com/i/chromeos/builders/soraka-release/builds/199/steps/PaygenTestDev/logs/stdio

I believe we can rule out firmware update? Was there a firmware update in the lab?

+Luis for toolchain question

Revert is here:

https://chromium-review.googlesource.com/c/547575

Propose to push this in the hope of a fix (since we presumably have nothing to lose)

preparing revert.

Hmmm, the latest eve-release (this morning) passed.  crbug.com/736000  as mentioned above.

But many other builds still fail.

> I believe we can rule out firmware update? Was there a firmware update in the lab?

The lab only runs firmware that's been released to customers.
Generally, if there's an update, it will happen Tuesday morning
after the Beta build with the update is released to users.

Last Tuesday was too early for the failures we're seeing, and
although there were updates, there's not much correspondence
with affected boards:
     snappy                 Google_Snappy.9042.87.1 -> Google_Snappy.9042.110.0
     sand                   Google_Sand.9042.87.0 -> Google_Sand.9042.110.0
     asuka                  Google_Asuka.7820.257.0 -> Google_Asuka.7820.306.0
     elm                    Google_Elm.8438.73.0 -> Google_Elm.8438.85.0
     ultima                 Google_Ultima.7287.131.63 -> Google_Ultima.7287.131.85
     pyro                   Google_Pyro.9042.87.1 -> Google_Pyro.9042.110.0

chumped https://chromium-review.googlesource.com/c/547575/

but there is another CL needed to revert the SDK. we are working on it.

PLEASE note that we reverting binutils upgrade just to be conservative. We have not verified this is the cause of the problem. 

The one data point we have is:

The 9678 cyan kernel is good
The 9679 cyan kernel does not boot
There were no commits in the kernel tree between 9678 and 9679
There are differences in the generated kernel code between 9678 and 9679

If this is related to  crbug.com/736000  eve

Good: R61-9678.0.0, Bad: R61-9679.0.0

FWIW canary update of eve from 9669 -> 9686 did NOT fail (1 of 1 tested)

Reports on CBC with Acer CB5-311 - big board.
https://productforums.google.com/forum/#!topic/chromebook-central/I0Xo_BvUFWs

#CBC-RS/TC-watchlist

It looks like the live dev-channel build has this problem on many boards. I don't know if it possible to roll back to 9679.0.0?

binutils rollback completed on crbug.com/736534 :

https://chromium-review.googlesource.com/c/549577/ 
https://chromium-review.googlesource.com/c/549093/
https://chromium-review.googlesource.com/c/547575/

It appears to have had an effect on lakitu 

https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-paladin/builds/6915

but the 2am 

..but the 2am PT build 9690.0.0 still seems to be broken.

Actually 9690.0.0 seems better now, auron and strago boards seems to be working.

So perhaps the toolchain was the problem. I don't know of any other action taken.

Waiting on release team before deciding whether this is fixed, or needs more action. Builds look pretty green in GoldenEye.

ketakid - things are looking pretty good now.

Marking this fixed since the toolchain revert.

AU works fine on Caroline and Cyan.
Verified on M61-9690.0.0

TestEng team verified on several boards the AU and recovery is successful.

I updated from 61.0.3138.0 (canary) on Kevin today and seeing what seems to be a similar black screen issue. 

I'm fortunately in dev mode, was able to open the dev console and change my channel.

On kevin device:

cros flash usb:// xbuddy://remote/kevin/latest-canary and install from usb.

Black screen seems to be an ongoing issue. A slight difference is that chrome logo is up before screen goes to black.

I believe this is a different bug since you get a display up. Can you file a bug for that?

Filed here: https://bugs.chromium.org/p/chromium/issues/detail?id=737707

Issue 738242 has been merged into this issue.