I was able to get the system to boot from recovery usb disk and would walk through bisecting what update caused this, or any other troubleshooting y'all might find useful.

Good news. I was able to reproduce this from a fresh stable image.

1) refresh to a stable TERRA image: chromeos_10176.76.0_terra_recovery_stable-channel_mp-v2.bin  ( model TERRA D25-E4C-A2I-A6A-A6L )

2) go into settings, select developer channel

3) let the update process update 

4) reboot.

5) when system boots, it's stuck in reboot loop just as before.  

(also I didn't notice before, after every about 10 attempts to restart the login it goes back to a white screen.  )

Hey all,

We're seeing additional reports of this on the community forum.

- https://productforums.google.com/forum/#!topic/chromebook-central/9XwYt7WAf9w
- https://productforums.google.com/forum/#!topic/chromebook-central/nt7ARxnTvkk

Thanks!

Reports are about Terra & Terra13.

Per Goldeneye, it looks like the Dev Channel would be 67.0.3376.0.

cc'ing myself as the M67 lead.   10176 is M64, however,   Thus per #2 this doesn't appear to be a M67 regression.

Adding two reddit reports that appear to be related in case they offer any insight:

- https://www.reddit.com/r/chromeos/comments/87l24a/chromebook_c202sa_unable_to_login/
- https://www.reddit.com/r/chromeos/comments/87m52a/update_broke_my_cb/

Reprod on terra AU from M64 stable to M67(10510.0.0) but not reproduced on minnie. 

Its could be one of duplicate issue of below ones.
https://bugs.chromium.org/p/chromium/issues/detail?id=825425
https://bugs.chromium.org/p/chromium/issues/detail?id=824630

attached log file.

Deleted: debug-logs_20180328-112609.tgz 41.9 KB

debug-logs_20180328-112609.tgz 41.9 KB Download

Terra appears to have not been running tests on ToT for some time now, it looks like the last time we had tests on 67 was 10504 on 3/20.

https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-03-15&last_date=2018-03-28&build=R67-*&board=%5Eterra%24&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true

First release build failing shows we failed sanity then http://cautotest-prod/afe/#tab_id=view_job&object_id=185161206 and it shows the device failed to make it to Chrome.
...
03/20 14:59:17.014 DEBUG|     dynamic_suite:0612| Waiting on suite.
03/20 15:17:53.015 INFO |        server_job:0218| START	185161256-chromeos-test/chromeos4-row8-rack6-host3/provision_AutoUpdate	provision	timestamp=1521583232	localtime=Mar 20 15:00:32	
03/20 15:17:53.016 INFO |        server_job:0218| 	FAIL	185161256-chromeos-test/chromeos4-row8-rack6-host3/provision_AutoUpdate	provision	timestamp=1521584151	localtime=Mar 20 15:15:51	Unhandled DevServerException: CrOS auto-update failed for host chromeos4-row8-rack6-host3: 0) RootfsUpdateError: After update and reboot, Chrome failed to reach login screen within 180 seconds, 1) RootfsUpdateError: After update and reboot, Chrome failed to reach login screen within 180 seconds
03/20 15:17:53.016 INFO |        server_job:0218| END FAIL	185161256-chromeos-test/chromeos4-row8-rack6-host3/provision_AutoUpdate	provision	timestamp=1521584151	localtime=Mar 20 15:15:51
...

I don't see any CLs at that point that should have any direct impact on Terra like this though. https://crosland.corp.google.com/log/10504.0.0..10505.0.0

Infra wise terra appears to have a substantial number of devices available in the various pools.
...
bhthompson@ragnarok:/usr/local/ssd960gb/chromiumos-1/src/third_party/autotest/files/contrib$ atest host list -b board:terra | ./count_labels -p

      6 bvt
     39 cts
     14 cts-perbuild
      1 performance
     17 suites
      1 wificell
...

The DUTs all look healthy.
https://viceroy.corp.google.com/chromeos/dut_health?model=terra&board=terra&build_config=kevin-paladin&build_number=&duration=8d&experimental=False&mdb_role=chrome-infra&milestone_version=&pool=suites&refresh=-1&status=Running&topstreams=35&waterfall=chromeos

I don't believe this is an infrastructure failure at this point, we probably need to reproduce this and dig into what may be going on.

Logs from comment 11:

...
[1008:1008:0328/112516.229366:FATAL:login_display_host_webui.cc(840)] Renderer crash on login window
[1289:1289:0328/112517.447513:ERROR:input_method_manager_impl.cc(1068)] IMEEngine for "jkghodnilhceideoidjikpgommlajknk" is not registered
...

This could be one of those bugs, but the weird thing is the same chrome version worked ok on the 10504 build, so I would not be suspecting a Chrome side problem.

Albert, do you know whom might be able to dig into the crash suggested here?

Not reproduced on M67(10510.0.0) Cyan and kefka(strago family) 

My initial thought was a dupe of bug 824630, but the symptoms are different (crash before logging in rather than after)

+hejq@ just in case they're related tho. 2 crash loops in the same week is unusual.

log entry from c#11
var/log/chrome/chrome_20180328-112544
[1:1:0328/112545.196591:VERBOSE1:zygote_main_linux.cc(336)] ZygoteMain: initializing 2 fork delegates
[2300:2300:0328/112545.254896:WARNING:install_attributes.cc(94)] Install attributes missing, first sign in
[2300:2300:0328/112545.263176:VERBOSE1:drm_device_handle.cc(83)] Succeeded authenticating /dev/dri/card0 in 0 ms with 1 attempt(s)
[2300:2325:0328/112545.500616:WARNING:name_value_pairs_parser.cc(55)] Key block_devmode already has value (error), ignoring new value: 0
[2300:2325:0328/112545.500708:WARNING:name_value_pairs_parser.cc(55)] Key customization_id already has value ASUS-TERRA2, ignoring new value: ASUS-TERRA2
[2300:2325:0328/112545.500745:WARNING:name_value_pairs_parser.cc(55)] Key rlz_brand_code already has value ASUJ, ignoring new value: ASUJ
[2300:2325:0328/112545.500824:WARNING:name_value_pairs_parser.cc(55)] Key gbind_attribute already has value =CisKIEQgnuT9con9Q-0Nw26XXXO93QUk9Adzispq9rh4FIEiEAAaBXRlcnJhEK6thYsI, ignoring new value: =CisKIEQgnuT9con9Q-0Nw26XXXO93QUk9Adzispq9rh4FIEiEAAaBXRlcnJhEK6thYsI
[2300:2325:0328/112545.500857:WARNING:name_value_pairs_parser.cc(55)] Key ubind_attribute already has value =CisKIKkvnUtF1GK1PPVqutqCxf84dHT7Xk8xy1BhfCtxm1RxEAEaBXRlcnJhELSH8o4P, ignoring new value: =CisKIKkvnUtF1GK1PPVqutqCxf84dHT7Xk8xy1BhfCtxm1RxEAEaBXRlcnJhELSH8o4P
[2300:2325:0328/112545.500924:WARNING:name_value_pairs_parser.cc(55)] Key customization_id already has value ASUS-TERRA2, ignoring new value: ASUS-TERRA2
[2300:2325:0328/112545.500955:WARNING:name_value_pairs_parser.cc(55)] Key initial_locale already has value en-US, ignoring new value: en-US
[2300:2325:0328/112545.500985:WARNING:name_value_pairs_parser.cc(55)] Key initial_timezone already has value America/Los_Angeles, ignoring new value: America/Los_Angeles
[2300:2325:0328/112545.501023:WARNING:name_value_pairs_parser.cc(55)] Key keyboard_layout already has value xkb:us::eng, ignoring new value: xkb:us::eng
[2300:2325:0328/112545.501052:WARNING:name_value_pairs_parser.cc(55)] Key region already has value us, ignoring new value: us
[2300:2325:0328/112545.501082:WARNING:name_value_pairs_parser.cc(55)] Key rlz_brand_code already has value ASUJ, ignoring new value: ASUJ
[2300:2325:0328/112545.501111:WARNING:name_value_pairs_parser.cc(55)] Key ActivateDate already has value 2018-13, ignoring new value: 2018-13
[2300:2325:0328/112545.501141:WARNING:name_value_pairs_parser.cc(55)] Key block_devmode already has value (error), ignoring new value: 0
[2300:2325:0328/112545.501173:WARNING:name_value_pairs_parser.cc(55)] Key check_enrollment already has value 0, ignoring new value: 0
[2300:2325:0328/112545.501231:WARNING:statistics_provider.cc(587)] Unable to load OEM Manifest file: /usr/share/oem/oobe/manifest.json
[2300:2300:0328/112545.503631:WARNING:statistics_provider.cc(285)] Statistics loaded after waiting 31ms.
[2300:2325:0328/112545.513282:WARNING:accelerometer_reader.cc(246)] Accelerometer device directory is empty at /dev/cros-ec-accel
[2300:2300:0328/112545.515532:VERBOSE1:update_display_configuration_task.cc(68)] OnDisplaysUpdated: new_display_state=SINGLE new_power_state=ALL_ON flags=1 force_configure=1 display_count=1
[2300:2300:0328/112545.515658:VERBOSE1:display_configurator.cc(210)] EnterState: display=SINGLE power=ALL_ON
[2300:2300:0328/112545.515726:VERBOSE1:display_configurator.cc(1060)] OnConfigured: success=1 new_display_state=SINGLE new_power_state=ALL_ON
[2300:2300:0328/112545.584615:WARNING:statistics_provider.cc(370)] Requested statistic not found: Product_S/N
[2300:2300:0328/112545.630394:ERROR:input_method_manager_impl.cc(1068)] IMEEngine for "jkghodnilhceideoidjikpgommlajknk" is not registered
[2300:2300:0328/112545.631850:VERBOSE1:chrome_session_manager.cc(203)] Starting Chrome with login/oobe screen.
[2300:2300:0328/112545.632033:VERBOSE1:login_display_host_webui.cc(1167)] Showing OOBE screen: unknown
[2300:2300:0328/112545.642581:VERBOSE1:login_display_host_webui.cc(1237)] Current locale: en-US
[2300:2300:0328/112545.642761:VERBOSE1:login_display_host_webui.cc(1262)] Initial locale: en-USkeyboard layout xkb:us::eng
[2300:2300:0328/112545.646218:VERBOSE1:input_events_blocker.cc(19)] InputEventsBlocker 0x11054e7a3f90 created.
[2300:2300:0328/112545.646388:VERBOSE1:login_display_host_webui.cc(482)] Login WebUI >> zero_delay: 0 wait_for_wp_load_: 1 init_webui_hidden_: 0
[2300:2300:0328/112545.646442:VERBOSE1:login_display_host_webui.cc(591)] Login WebUI >> wizard postponed
[2300:2300:0328/112545.646473:VERBOSE1:login_display_host_webui.cc(226)] Initial time zone: America/Los_Angeles
[2300:2300:0328/112545.646629:VERBOSE1:input_events_blocker.cc(28)] InputEventsBlocker 0x11054e7a3f90 destroyed.
[2300:2300:0328/112545.669347:VERBOSE1:display_configurator.cc(906)] SetDisplayPower: power_state=ALL_ON flags=0, configure timer=Stopped
[2300:2300:0328/112545.699277:VERBOSE1:drm_display_host_manager.cc(247)] Got display event ADD for /dev/dri/card0
[2300:2300:0328/112545.704115:VERBOSE1:drm_display_host_manager.cc(247)] Got display event ADD for /dev/dri/card1
[2300:2328:0328/112545.704371:VERBOSE1:drm_device_handle.cc(83)] Succeeded authenticating /dev/dri/card1 in 0 ms with 1 attempt(s)
[2300:2300:0328/112545.752174:VERBOSE1:automatic_reboot_manager.cc(310)] Rescheduling reboot
[2340:2345:0328/112545.763676:VERBOSE1:drm_device_manager.cc(53)] Could not initialize DRM device for /sys/devices/platform/vgem/drm/card1
[2300:2300:0328/112546.224777:VERBOSE1:login_display_host_webui.cc(801)] Login WebUI >> wp animation done
[2300:2300:0328/112546.224864:VERBOSE1:login_display_host_webui.cc(976)] Login WebUI >> Init postponed WebUI
[2300:2300:0328/112546.224904:VERBOSE1:login_display_host_webui.cc(594)] Login WebUI >> wizard
[2300:2300:0328/112546.226740:VERBOSE1:login_display_host_webui.cc(1064)] Login WebUI >> show login wnd on create
[2300:2300:0328/112546.249900:VERBOSE1:wizard_controller.cc(302)] Starting OOBE wizard with screen: unknown
[2300:2300:0328/112546.250142:VERBOSE1:wizard_controller.cc(472)] Showing network screen.
[2300:2300:0328/112546.250680:VERBOSE1:wizard_controller.cc(1102)] SetCurrentScreenSmooth: connect
[2300:2300:0328/112546.905870:VERBOSE1:display_configurator.cc(954)] Display snapshots invalidated.
[2300:2300:0328/112546.906039:VERBOSE1:update_display_configuration_task.cc(68)] OnDisplaysUpdated: new_display_state=SINGLE new_power_state=ALL_ON flags=0 force_configure=1 display_count=1
[2300:2300:0328/112546.906104:VERBOSE1:display_configurator.cc(210)] EnterState: display=SINGLE power=ALL_ON
[2340:2345:0328/112546.908039:VERBOSE1:drm_display.cc(105)] DRM configuring: device=/sys/devices/pci0000:00/0000:00:02.0/drm/card0 crtc=31 connector=47 origin=0,0 size=1366x768
[2300:2300:0328/112546.911698:VERBOSE1:display_configurator.cc(1060)] OnConfigured: success=1 new_display_state=SINGLE new_power_state=ALL_ON
[2300:2300:0328/112547.140275:WARNING:CONSOLE(0)] "Styling master document from stylesheets defined in HTML Imports is deprecated, and is planned to be removed in M67, around May 2018. Please refer to https://goo.gl/EGXzpw for possible migration paths.", source:  (0)
[2340:2345:0328/112547.359350:WARNING:screen_manager.cc(115)] Display controller (crtc=31) already present.
[2300:2300:0328/112547.359723:VERBOSE1:display_configurator.cc(954)] Display snapshots invalidated.
[2300:2300:0328/112547.359792:VERBOSE1:update_display_configuration_task.cc(68)] OnDisplaysUpdated: new_display_state=SINGLE new_power_state=ALL_OFF flags=0 force_configure=0 display_count=1
[2300:2300:0328/112547.359854:VERBOSE1:display_configurator.cc(210)] EnterState: display=SINGLE power=ALL_OFF
[2340:2345:0328/112547.360107:VERBOSE1:drm_display.cc(105)] DRM configuring: device=/sys/devices/pci0000:00/0000:00:02.0/drm/card0 crtc=31 connector=47 origin=0,0 size=0x0
[2300:2300:0328/112547.756207:VERBOSE1:display_configurator.cc(1060)] OnConfigured: success=1 new_display_state=SINGLE new_power_state=ALL_OFF
[2300:2300:0328/112549.890292:WARNING:statistics_provider.cc(370)] Requested statistic not found: Product_S/N
[2300:2300:0328/112549.963269:FATAL:login_display_host_webui.cc(840)] Renderer crash on login window


+albert need your help to assign to the right owner.

> [2300:2300:0328/112549.963269:FATAL:login_display_host_webui.cc(840)] Renderer crash on login window

This is probably a blink issue.

Very strange that this only happen on Terra and not other strago

dchan@ why did this come back to me? 

The login code owners are the right starting point, which is why I assigned to jdufault@ who narrowed it down to a renderer crash. We'll want to try to get a crash dump from an affected device to narrow it further and route to someone on Chrome.

FYI, I set the terra push to 0% to keep the issue from propagating further.

Did #20 confirm that this issue is restricted to terra?

+abodenha, sorry about reassign to you, wasn't meant to.

+kbleicher, yes, +ansar has try a few other strago and seems to be fine.

This issue reproduces on ToT 10525.0.0/67.0.3381.0 with Bob
(from #c14 of Issue 824630)

Bob issue reproduce on both dev mode and non-dev mode

add log file from Bob

Deleted: debug-logs_20180329-102356.tgz 194 KB

debug-logs_20180329-102356.tgz 194 KB Download

Re #26-29 I don't think this is a dupe of bug 824630.  That crash happens on multiple devices after signing in. This one actually crashes before the sign-in page comes up.

bob might also be a different issue, jsiangc@ filed https://bugs.chromium.org/p/chromium/issues/detail?id=827249
the device boots and seems to stuck.

https://crbug.com/826973#13 : It affects all RK3399 boards: kevin, bob, scarlet.

I borrowed the terra device in comment #11 and was able to reproduce. Deploying the latest Chrome build from master fixed the issue, so I'm guessing the issue was found and fixed elsewhere.

FYI: old bug Sign in bootloop after auto update to V56
https://bugs.chromium.org/p/chromium/issues/detail?id=692802

Comment 51 by et...@we3chapmans.com, Today (103 minutes ago)
⚐
Has reoccurred for me on 66.0.3359.67 beta 64-bit on Acer 14

+jdufault, any idea what went wrong ?  terra seems to be only one effected for the family of strago.

dymphep: That seems like a different issue.
dchan: Not sure; I wasn't able to get a stack trace on the device when it was failing, and when I deployed an updated chrome the failure stopped.

To add a bit: it was caused by a renderer crash.

Do you know what version of chrome that fix this problem ?

M67 just got 67.0.3390.0 and terra is still having problem.

re opening as per c#39

(feel free to close/open a new bug if different issue)

> (feel free to close/open a new bug if different issue)

Without going into a long monologue on why, I recommend we re-close
this bug, and focus attention on the cluster of bugs around  bug 830321 .

...just pointing out 830321 is a locked one for us mere users - if you're going to close this one it'd be nice to be able to subscribe to it or something that represents it or have some idea if it ever gets fixed/if it's appropriate to try upgrading to 67 again for further testing from out here in terra userland.

> ...just pointing out 830321 is a locked [ ... ]

I've opened the bug.  Short summary is simply that there's a
software bug from a recent change (possibly/probably in the
graphics libraries).  Once that's fixed (and the fix is
published), it should be safe to return to the dev channel.

Removing Blink in the absence of a stack trace.

closing (dup) this as per c#41