Greg, this looks like Chrome isn't able to reliably open a guest session... So the test is good, but Chrome is the issue.

I took a look at graphics_Sanity. It reliably fails on PFQ (9 out of 3*3 retries latest runs) but passes on CQ and canary. So clearly a bad Chrome coming in. Setting up a device for Chrome bisection.

Last good Chrome 67.0.3369.0, first bad Chrome 67.0.3370.0, i.e. one of
https://chromium.googlesource.com/chromium/src/+log/67.0.3369.0..67.0.3370.0?n=10000
(There are a bunch of "cros"/"chromeos" changes that might be candidates.)

I have installed tot build (with good Chrome) on peppy. Running tests now locally to setup repro.

Thanks for all the help setting this up, ihf@!  Bisect in progress...

Bisect revealed the problem CL to be
  https://chromium-review.googlesource.com/c/chromium/src/+/801974
To reproduce on peppy device:
  cd /usr/local/autotest
  bin/autotest tests/graphics_Sanity/control
The test doesn't actually report failure on device.  On a good revision, the screen will flash through a login flow and a few test screens before returning to login screen.  On a bad revision, only the test screens (white and blue ovals) are seen; other screens are black.  (Thanks again to ihf@ for help with this!)

As far as I've seen, of all the main pfq builders, this only breaks peppy.

hoegsberg@ - Could you either revert this change, or make necessary changes to peppy config so that this CL doesn't break this device?  This issue is (partially) responsible for a red pfq, so I'll go ahead and revert it in a few hours if there's no activity here.

I'll take a look now, and we can certainly revert in a few hours if we don't make any progress.

Lets make sure to revert by 4pm to allow the PFQ a chance to uprev tonight.

I'm testing with

CHROMEOS_RELEASE_DESCRIPTION=10491.0.0 (Official Build) dev-channel peppy test

and can't reproduce the failure. Steps described in #4 gives:

localhost /usr/local/autotest # ./bin/autotest tests/graphics_Sanity/control
11:29:33 INFO | Writing results to /usr/local/autotest/results/default
11:29:33 INFO | START   ----    ----    timestamp=1521138573    localtime=Mar 15 11:29:33
11:29:33 INFO |         START   graphics_Sanity graphics_Sanity timestamp=1521138573    localtime=Mar 15 11:29:33

...

11:30:02 INFO | wflinfo: OpenGL renderer string: Mesa DRI Intel(R) Haswell Mobile
11:30:03 INFO | Starting after_hook for graphics_Sanity
11:30:03 INFO | after_hook completed
11:30:03 INFO |                 GOOD    graphics_Sanity graphics_Sanity timestamp=1521138603    localtime=Mar 15 11:30:03       completed successfully
11:30:03 INFO |         END GOOD        graphics_Sanity graphics_Sanity timestamp=1521138603    localtime=Mar 15 11:30:03
11:30:03 INFO | END GOOD        ----    ----    timestamp=1521138603    localtime=Mar 15 11:30:03

And that was with chrome 67.0.3369.0 so it didn't have the commit in question...

The culprit CL in #4 also caused the consistent crash ad slowness on login screen on cave and caroline. Can we revert it first to avoid affecting more people? 

Ok, reproducing the issue now with ToT chrome and should be able to fix it within the hour.

#10: What kind of slowness do you on caroline?

Re#11: the device is very janky. It probably take a second or two to move the mouse or to respond to a mouse/touch event. It's seen on both cave and caroline.

Is there anything in /var/log/ui/ui.LATEST?  I'll take a look myself as soon as I get chrome built for caroline.

Sorry I just swiped my device. I'll retest again and paste the log here.

Attached logs.

Deleted: ui.LATEST 417 KB

ui.LATEST 417 KB Download

Here's the fix we're looking at:

https://chromium-review.googlesource.com/c/chromium/src/+/964984

Looks like it should fix caroline too from the log, trying now.

Yup, I reproduced the problem on caroline and verified that the CL fixes it. Now going through CQ.

Hey, my device is a link and I scrambled all day yesterday figuring out why my deploy was crashing. glevin@ pointed me to this bug. I confirm that my link will not crash on startup with b7c8d1ce9f10ff2cd18f0b9d13849bfac484d88c but does crash on startup with 48d1ecafe170a41e0e1eee736370f0f56f9e3e93.

I'd say next time if it's clear a CL is impacting ToT so badly a revert should be immediate. Having the master branch broken for this long slows down other developers.

It looks like the proposed fix is not passing ozone_unittests, so reverting the culprit CL seems appropriate now. I'll revert within the next hour if no one else does.

Yes, thanks. I wasn't expecting the unit test to fall apart like this :(

Revert landed
https://chromium-review.googlesource.com/c/chromium/src/+/965721

Looks like Chrome has upreved over the weekend.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/258ff091cc20abbd1b34b6833684ccfd5b1024d4

commit 258ff091cc20abbd1b34b6833684ccfd5b1024d4
Author: Kristian H. Kristensen <hoegsberg@chromium.org>
Date: Thu Mar 22 17:15:07 2018

ozone/drm: Don't use RGBA framebuffers with legacy KMS

With CL:801974 we started using RGBA for primary framebuffers so that
the alpha channel will be well-defined. Boards without atomic
modesetting should not use overlays and scanout the primary
framebuffer as RGB, but this didn't quite work. We tried to modeset
and pageflip to RGBA buffers on boards with RGB-only overlays (peppy).

This commit makes sure that we always pick the opaque framebuffer for
the lowest plane.

Bug:  821944 , b/74997524
Change-Id: I79929d59bc8a568d7a6090a3ab5a4b31fad658b6
Reviewed-on: https://chromium-review.googlesource.com/964984
Commit-Queue: Kristian H. Kristensen <hoegsberg@chromium.org>
Reviewed-by: Daniel Nicoara <dnicoara@chromium.org>
Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#545124}
[modify] https://crrev.com/258ff091cc20abbd1b34b6833684ccfd5b1024d4/ui/ozone/platform/drm/gpu/crtc_controller.cc
[modify] https://crrev.com/258ff091cc20abbd1b34b6833684ccfd5b1024d4/ui/ozone/platform/drm/gpu/hardware_display_plane_manager.cc
[modify] https://crrev.com/258ff091cc20abbd1b34b6833684ccfd5b1024d4/ui/ozone/platform/drm/gpu/hardware_display_plane_manager_legacy.cc
[modify] https://crrev.com/258ff091cc20abbd1b34b6833684ccfd5b1024d4/ui/ozone/platform/drm/gpu/screen_manager.cc
[modify] https://crrev.com/258ff091cc20abbd1b34b6833684ccfd5b1024d4/ui/ozone/platform/drm/gpu/screen_manager_unittest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/65ae686a967db072631212c5afe7ce95c12e9547

commit 65ae686a967db072631212c5afe7ce95c12e9547
Author: Daniele Castagna <dcastagna@chromium.org>
Date: Fri Mar 23 16:29:29 2018

viz: Make primary plane transparent

This CL enables alpha blending at scanout for the primary plane.
In this way chromecast will be able to punch a hole in the primary
plane and show content underneath.

Note that on platform using legacy pageflip we'll still be
scanning out without alpha blending (crrev.com/c/964984).

Bug:  821944 , b/74997524
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Change-Id: I654935dc630f7a9f6cb4717cd574a21a2749c442
Reviewed-on: https://chromium-review.googlesource.com/976961
Reviewed-by: Daniel Nicoara <dnicoara@chromium.org>
Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#545478}
[modify] https://crrev.com/65ae686a967db072631212c5afe7ce95c12e9547/components/viz/service/display/direct_renderer.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/79a0b39380971cfcd7ddee9e0da2305da15a743c

commit 79a0b39380971cfcd7ddee9e0da2305da15a743c
Author: Daniele Castagna <dcastagna@chromium.org>
Date: Fri Mar 23 17:59:08 2018

Revert "viz: Make primary plane transparent"

This reverts commit 65ae686a967db072631212c5afe7ce95c12e9547.

Reason for revert: breaks legacy page flips, since HardwareDisplayPlaneManager::IsCompatible tests with RGBA and fails.

Original change's description:
> viz: Make primary plane transparent
> 
> This CL enables alpha blending at scanout for the primary plane.
> In this way chromecast will be able to punch a hole in the primary
> plane and show content underneath.
> 
> Note that on platform using legacy pageflip we'll still be
> scanning out without alpha blending (crrev.com/c/964984).
> 
> Bug:  821944 , b/74997524
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
> Change-Id: I654935dc630f7a9f6cb4717cd574a21a2749c442
> Reviewed-on: https://chromium-review.googlesource.com/976961
> Reviewed-by: Daniel Nicoara <dnicoara@chromium.org>
> Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#545478}

TBR=dnicoara@chromium.org,dcastagna@chromium.org,hoegsberg@chromium.org

Change-Id: If2d49fb3d9b1042653f7d4359b699d784c757d14
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  821944 , b/74997524
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/978581
Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#545513}
[modify] https://crrev.com/79a0b39380971cfcd7ddee9e0da2305da15a743c/components/viz/service/display/direct_renderer.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b2ad224eb1c7b3c831f28efa56a45ea8db4093b4

commit b2ad224eb1c7b3c831f28efa56a45ea8db4093b4
Author: Daniele Castagna <dcastagna@chromium.org>
Date: Tue Mar 27 03:16:46 2018

ozone: Check opaque fb format with legacy KMS

With crrev.com/c/964984 we always scanout an opaque fb when
we're using legacy KMS.

HardwareDisplayPlaneManager::IsCompatible was still testing
that scanout buffer was compatible using a non opaque format.
This caused issues when we landed crrev.com/c/976961 and
would cause pageflip on samus (RGBA not supported) to fail.

This patch overrides HardwareDisplayPlaneManager::IsCompatible
for the Legacy HardwareDisplayPlaneManager and checks
using the opaque format.

Bug:  821944 , b/74997524
Test: samus ui with crrev.com/c/976961 works.
Change-Id: I6fd77ba55832a2d77d1652e81a77f1f0555cffa1
Reviewed-on: https://chromium-review.googlesource.com/981239
Reviewed-by: Alex Sakhartchouk <alexst@chromium.org>
Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#545947}
[modify] https://crrev.com/b2ad224eb1c7b3c831f28efa56a45ea8db4093b4/ui/ozone/platform/drm/gpu/hardware_display_plane_manager.h
[modify] https://crrev.com/b2ad224eb1c7b3c831f28efa56a45ea8db4093b4/ui/ozone/platform/drm/gpu/hardware_display_plane_manager_legacy.cc
[modify] https://crrev.com/b2ad224eb1c7b3c831f28efa56a45ea8db4093b4/ui/ozone/platform/drm/gpu/hardware_display_plane_manager_legacy.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a91d4ae3dcca7bc6c212ea520aa5ff362f58ca04

commit a91d4ae3dcca7bc6c212ea520aa5ff362f58ca04
Author: Daniele Castagna <dcastagna@chromium.org>
Date: Tue Mar 27 18:41:28 2018

Reland "viz: Make primary plane transparent"

This is a reland of 65ae686a967db072631212c5afe7ce95c12e9547

The issue it caused when we originally landed it was fixed
in crrev.com/c/981239

Original change's description:
> viz: Make primary plane transparent
>
> This CL enables alpha blending at scanout for the primary plane.
> In this way chromecast will be able to punch a hole in the primary
> plane and show content underneath.
>
> Note that on platform using legacy pageflip we'll still be
> scanning out without alpha blending (crrev.com/c/964984).
>
> Bug:  821944 , b/74997524
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
> Change-Id: I654935dc630f7a9f6cb4717cd574a21a2749c442
> Reviewed-on: https://chromium-review.googlesource.com/976961
> Reviewed-by: Daniel Nicoara <dnicoara@chromium.org>
> Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#545478}

Bug:  821944 , b/74997524
Change-Id: Icfff83119231cd9a1e486719dbb0a25786ea71f2
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Tbr: dnicoara@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/980956
Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#546182}
[modify] https://crrev.com/a91d4ae3dcca7bc6c212ea520aa5ff362f58ca04/components/viz/service/display/direct_renderer.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19c4399b6ab4ab6a9d400b2229b5effdb182790d

commit 19c4399b6ab4ab6a9d400b2229b5effdb182790d
Author: Daniele Castagna <dcastagna@chromium.org>
Date: Fri Mar 30 21:56:11 2018

Revert "Reland "viz: Make primary plane transparent""

This reverts commit a91d4ae3dcca7bc6c212ea520aa5ff362f58ca04.

Reason for revert: rockchip display is all black because of crbug.com/826967

Original change's description:
> Reland "viz: Make primary plane transparent"
> 
> This is a reland of 65ae686a967db072631212c5afe7ce95c12e9547
> 
> The issue it caused when we originally landed it was fixed
> in crrev.com/c/981239
> 
> Original change's description:
> > viz: Make primary plane transparent
> >
> > This CL enables alpha blending at scanout for the primary plane.
> > In this way chromecast will be able to punch a hole in the primary
> > plane and show content underneath.
> >
> > Note that on platform using legacy pageflip we'll still be
> > scanning out without alpha blending (crrev.com/c/964984).
> >
> > Bug:  821944 , b/74997524
> > Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
> > Change-Id: I654935dc630f7a9f6cb4717cd574a21a2749c442
> > Reviewed-on: https://chromium-review.googlesource.com/976961
> > Reviewed-by: Daniel Nicoara <dnicoara@chromium.org>
> > Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#545478}
> 
> Bug:  821944 , b/74997524
> Change-Id: Icfff83119231cd9a1e486719dbb0a25786ea71f2
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
> Tbr: dnicoara@chromium.org
> Reviewed-on: https://chromium-review.googlesource.com/980956
> Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
> Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#546182}

TBR=dnicoara@chromium.org,dcastagna@chromium.org,hoegsberg@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  821944 , b/74997524
Change-Id: Ide67215812a19b2a3bff93ba7c67cfbe31f62565
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/985033
Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#547281}
[modify] https://crrev.com/19c4399b6ab4ab6a9d400b2229b5effdb182790d/components/viz/service/display/direct_renderer.cc