1) Can you add screenshot ?
2) Can you ask customers to press "tab" and see if it adds any error codes to screens ?

 royans@, thanks for your comments.
1) Can you add screenshot ?> it is available in the shared folder.
https://drive.google.com/drive/folders/0B01YYztUbOCuaDU2RVpzTEZKclU?usp=sharing
2) Can you ask customers to press "tab" and see if it adds any error codes to screens ?
>I will ask the customer.

The customer provided more information about the issue.
2) Can you ask customers to press "tab" and see if it adds any error codes to screens ?
>Pushing "tab" did nothing, just stayed on the blue screen. 

Reproduce step:
1.Upgrade from  Chrome OS 58 (334.72.0 58.0.3029.14) to Chrome OS59(9460.73.0 59.0.3071.134)
2.After completing the OS upgrade, leave the device and move it to the sleep mode.
3.Resume the device.(Can be happen with user logged on or off.)
4.The Blue Screen appears and have to reboot the device to resolve the issue.
(Pushing "tab" did nothing, just stayed on the blue screen. )

-Waiting to hear back on device updated to beta v61. 

Known workaround
rebuild 1 device with a clean v59 image which updated to v60, as the restriction policy to v59 had not applied yet.
(Currently know workaround is rebuild from OS image)

I uploaded few other log file to share folder.
Had issue - debug-logs_20170921-160205.tgz
Updated & working - debug-logs_20170921-160411.tgz
debug-logs_20170921-162904.tgz (It happened at approx 16:27)

From message log file in debug-logs_20170921-162904.tgz, noticed below message.

017-09-21T16:26:42.039690+09:30 INFO avahi-daemon[1014]: Interface eth0.IPv4 no longer relevant for mDNS.
2017-09-21T16:26:42.040213+09:30 DEBUG kernel: [ 8047.319612] wlan0: deauthenticating from 00:a2:89:e8:4e:e4 by local choice (reason=3  [Reason: Warning:Wifi AP disconnected]   [Reason: info:ieee80211_reasoncode] )
2017-09-21T16:26:42.041034+09:30 ERR chrome[11992]: [11992:11992:0921/162642.040996:INFO:remote_commands_invalidator.cc(45)] Shutdown RemoteCommandsInvalidator.
2017-09-21T16:26:42.041088+09:30 ERR chrome[11992]: [11992:11992:0921/162642.041073:INFO:remote_commands_invalidator.cc(68)] Stopping RemoteCommandsInvalidator.
2017-09-21T16:26:42.041111+09:30 ERR chrome[11992]: [11992:11992:0921/162642.041101:INFO:remote_commands_invalidator.cc(167)] Unregister RemoteCommandsInvalidator.
2017-09-21T16:26:42.042081+09:30 DEBUG kernel: [ 8047.321260] wlan0: moving STA 00:a2:89:e8:4e:e4 to state 3
2017-09-21T16:26:42.042092+09:30 DEBUG kernel: [ 8047.321265] wlan0: moving STA 00:a2:89:e8:4e:e4 to state 2
2017-09-21T16:26:42.042094+09:30 DEBUG kernel: [ 8047.321270] wlan0: moving STA 00:a2:89:e8:4e:e4 to state 1
2017-09-21T16:26:42.047559+09:30 DEBUG kernel: [ 8047.325983] ieee80211 phy0: Removed STA 00:a2:89:e8:4e:e4
2017-09-21T16:26:42.047575+09:30 DEBUG kernel: [ 8047.326061] ieee80211 phy0: Destroyed STA 00:a2:89:e8:4e:e4
2017-09-21T16:26:42.058097+09:30 DEBUG kernel: [ 8047.337148] ieee80211 phy0: device now idle
2017-09-21T16:26:42.058107+09:30 ERR kernel: [ 8047.337494] ath: phy0: TX while HW is in FULL_SLEEP mode
2017-09-21T16:26:42.058108+09:30 ERR kernel: [ 8047.337510] ath: phy0: TX while HW is in FULL_SLEEP mode
2017-09-21T16:26:42.058110+09:30 ERR kernel: [ 8047.337521] ath: phy0: TX while HW is in FULL_SLEEP mode
2017-09-21T16:26:42.058111+09:30 ERR kernel: [ 8047.337533] ath: phy0: TX while HW is in FULL_SLEEP mode
2017-09-21T16:26:42.061285+09:30 INFO kernel: [ 8047.340598] cfg80211: Calling CRDA to update world regulatory domain
2017-09-21T16:26:42.069133+09:30 INFO kernel: [ 8047.348509] cfg80211: World regulatory domain updated:
2017-09-21T16:26:42.069140+09:30 INFO kernel: [ 8047.348522] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
2017-09-21T16:26:42.069142+09:30 INFO kernel: [ 8047.348535] cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069143+09:30 INFO kernel: [ 8047.348546] cfg80211:   (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069145+09:30 INFO kernel: [ 8047.348557] cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069851+09:30 INFO kernel: [ 8047.348568] cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069855+09:30 INFO kernel: [ 8047.348579] cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069856+09:30 INFO kernel: [ 8047.348590] cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069858+09:30 INFO kernel: [ 8047.348601] cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm)
2017-09-21T16:26:42.069859+09:30 INFO kernel: [ 8047.348612] cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm)
2017-09-21T16:26:42.078378+09:30 INFO avahi-daemon[1014]: Withdrawing address record for 172.20.20.15 on wlan0.
2017-09-21T16:26:42.078408+09:30 INFO avahi-daemon[1014]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 172.20.20.15.
2017-09-21T16:26:42.078433+09:30 INFO avahi-daemon[1014]: Interface wlan0.IPv4 no longer relevant for mDNS.
2017-09-21T16:26:42.143853+09:30 ERR shill[839]: [ERROR:object_proxy.cc(582)] Failed to call method: fi.w1.wpa_supplicant1.Interface.Disconnect: object_path= /fi/w1/wpa_supplicant1/Interfaces/1: fi.w1.wpa_supplicant1.NotConnected: This interface is not connected
2017-09-21T16:26:42.143958+09:30 ERR shill[839]: [ERROR:dbus_method_invoker.h(111)] CallMethodAndBlockWithTimeout(...): Domain=dbus, Code=fi.w1.wpa_supplicant1.NotConnected, Message=This interface is not connected
2017-09-21T16:26:42.144026+09:30 ERR shill[839]: [ERROR:chromeos_supplicant_interface_proxy.cc(198)] Failed to disconnect: fi.w1.wpa_supplicant1.NotConnected This interface is not connected
2017-09-21T16:26:42.167644+09:30 ERR wpa_supplicant[407]: rmdir[ctrl_interface=/run/wpa_supplicant]: Permission denied
2017-09-21T16:26:42.220429+09:30 NOTICE shill_logout_user[12587]: Failed to pop shill user profiles
2017-09-21T16:26:42.224596+09:30 INFO session_manager[11964]: [INFO:browser_job.cc(157)] Terminating process: 
2017-09-21T16:26:42.224612+09:30 INFO session_manager[11964]: [INFO:system_utils_impl.cc(110)] Sending 15 to 11992 as 1000
2017-09-21T16:26:42.499359+09:30 INFO session_manager[11964]: [INFO:session_manager_service.cc(488)] SessionManagerService quitting run loop
2017-09-21T16:26:42.499429+09:30 INFO session_manager[11964]: [INFO:session_manager_service.cc(214)] SessionManagerService exiting
2017-09-21T16:26:42.517716+09:30 INFO session_manager[11964]: [INFO:policy_service.cc(202)] Persisted policy to disk.
2017-09-21T16:26:58.425640+09:30 INFO kernel: [    0.000000] Initializing cgroup subsys cpu  [Reason: info:Reboot] 
2017-09-21T16:26:58.425720+09:30 NOTICE kernel: [    0.000000] Linux version 3.8.11 (chrome-bot@cros-beefy227-c2) (gcc version 4.9.x 20150123 (prerelease) (4.9.2_cos_gg_4.9.2-r152-32c89c19b042a12b5a1bf0153299154ea5435c03_4.9.2-r152) ) #1 SMP Thu Jul 13 11:52:14 PDT 2017  [Reason: info:ChromeOS Build Date] 
2017-09-21T16:26:58.425723+09:30 INFO kernel: [    0.000000] Command line: cros_secure console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 2539520 verity payload=PARTUUID=7f26bcbb-4096-a344-bd39-06ba3b18ed5b/PARTNROFF=1 hashtree=PARTUUID=7f26bcbb-4096-a344-bd39-06ba3b18ed5b/PARTNROFF=1 hashstart=2539520 alg=sha1 root_hexdigest=756df766e2cc4c16580f85424eddda77f218c557 salt=8ddc3aa7c32f52fe7cce8780a5d59186b240e36fb36c24a6ed079f5295d58853" noinitrd vt.global_cursor_default=0 kern_guid=7f26bcbb-4096-a344-bd39-06ba3b18ed5b add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic  
2017-09-21T16:26:58.425724+09:30 INFO kernel: [    0.000000] e820: BIOS-provided physical RAM map:

Bernie: Can you help us route this bug ?

bhthompson@
Could you please help us?
Please let me know if you need extra info from the customer.

Does this happen reliably across multiple devices?

It is hard to tell from the video/picture, but is the blue the same sort of blue you get when the device is not getting any video input (this one has a special display controller to allow it to work as a monitor, the blue might indicate no output from the Chrome OS board at all). 

If a USB display device is available (like a docking station or USB monitor) can we try connecting that up to a system in this state to see if we can see any output?

the number of affected devices are more than 40 but potentially more (suspecting 100 over).

The customer has reported about “when the blue screen is there, they can still conduct a screen capture from the login screen and on rebooting & loggin,” so I have asked them to take another video capture. I will ask to them for the extend monitor.

I also uploaded another log file(debug-logs_20170927-110519.tgz) that customer has provided.

Bernie: Do we have these devices with the test team and can have this run through reboot cycles and see if we can repro it in anyway. Based on customer input, reboot fixes it which makes me believe this is not a purely hardware issue.

+dchan

I do believe these are around, though they are getting kind of old and are bulky, if this happened all that often I would have expected us to see it as you suggest. 

What we probably want to see is the output of `modetest` while a device is in this state, but that can be awkward to capture. If they can do a screen capture while the device is in this state that is a good sign, at least the OS underneath is there. 

So to clarify, this may or may not happen on a reboot (e.g. it is intermittent)?

I uploaded another video file (Screen Capture during Blue Screen.mp4) to below share drive.
https://drive.google.com/drive/folders/0B01YYztUbOCuaDU2RVpzTEZKclU?usp=sharing

It looks like OS is still running during the Blue Screen.

bhthompson@
Please let me know if we still need to attach another display to confirm the condition of OS.

It sounds like the multiplexing of the display isn't always working. Who is the hw team contact here?

+vineeths, alexpau, dyeh

I don't know that we have anyone at Google with deep expertise on that monitor SoC, I don't believe we have any control over the firmware for it either if it is really the monitor SoC getting wedged. We may want to engage with LG here.

We should see if we can get one of these into this state locally preferably in dev mode for further debug, I am curious if it is still reporting a proper EDID to Chrome OS. 

I also wonder if you plug in an external HDMI device if it pops it back into displaying something.

+kalin and team to replicate this problem

We'll try to reproduce in non-CfM setup, but I am puzzled by the fact it is CfM unit, that is not supposed to suspend(sleep) AFAIK. After the restart from update the device should start the kiosk app and enter the CFM app. No?

I am the Google Cloud TAM for large customer in Australia who is affected by this. 

The customer is very concerned that this fault is still open with no solution to provide Chromebase stability whilst the root cause is identified and fixed. 

The customer currently has more than 1,550 of the Chromebox 22CB25S models that are blue screening after sleep wakeup 

Customer can provide a Chromebox to Google support to allow Google support to test to reproduce device 22CB25S 

Customer performed a "clean build" on an affected Chromebox and the problem has not returned. However, these 1,550 Chromeboxes are spread at stores all over Australia. 

I'm a little confused , is this a Chromebox issue or a Chromebase issue ?
Please try the step here to see if you could get the display back
1) plug HDMI to a laptop
2) touch the CHROME printing next to the power button , it should be
switching the input to HDMI.
3)check if the screen recovers .

Have we had any luck reproducing this locally?

On the user's side are we only seeing this with one enterprise user or do we have reports from multiple? If this is just one, are all the devices reporting it loading the same kiosk app, or do we know more about how the devices are being used (ephemeral, desktop, etc)? If clean build means the system was recovered by USB stick, that implies some local state may be implicated, and if these are being used in a kiosk mode it may be related to the app being loaded. 

I don't see anything obvious in the logs, but if it is at the monitor SoC level we may be a bit blind, as far as I know we don't have any logging or debug information coming to Chrome OS from the monitor SoC. These early Chromebase systems are essentially a regular monitor with a Chromebox board bolted inside, which is how it has the ability to work as a regular HDMI monitor. 

Chrome OS Board ------> Monitor SoC ------> LCD Panel
HDMI Input --------------/

As such, this failure mode appears as if the monitor itself stopped working, Chrome OS still thinks it is running and talking to the monitor, but the monitor is in a state where it is dumping blue to the panel. 

Vineeth, can we start an inquiry with LG to help debug the monitor SoC if we have not already?

Can I get a bit more information here:

1. Does this problem happen *only* on 9592.96.0 and 9460.73.0, how about other Chrome OS versions or are we sure that there is no OS dependency here and the problem happens irrespective of Chrome OS version?
2. Please try steps recommended by dyeh@ in comments 17 and let us know what the result is
3. Per comment 16, there are 1550 devices that are showing this problem. Did this problem start instantaneously on all devices or have these issues been accumulating over a period of time?

Also can someone confirm if this behavior is after the Chromebase is put into CFM mode or is the issue happening irrespective of CFM?

Spoke to smcewan@ over chat. He mentioned its not setup as CFM.

Attempting to recreate the blue-screen problem on a LG Chromebase (Monroe) in kiosk mode.  Details:

M-60 Stable build 9592.96.0, chrome 60.0.3112.114: Test image
Kiosk app:  Chrome Sign Builder displaying a mix of still images and 5-minute videos.

I put the device to sleep using the root command "powerd_dbus_suspend".

First try:  I woke the device up after 3 seconds and the app resumed nicely.

Second try:   The device slept for 5 minutes.  I had to hit the <enter> key and do several mouse left-clicks to wake the Monroe up.  There was a brief flash of blue filling the upper half of the screen, then the app played normally.

Third try:  1 hour of sleep time.  Typing <enter> and mouse clicks did not seem to have any effect.  Pressing the power button briefly finally woke the device up.  There was no observable blue flash this time, and the app resumed playing.

I will let the device sleep over the weekend and see what happens.

After the weekend asleep, the Monroe in kiosk mode woke up after one press of the <enter> key and resumed its Chrome Sign Builder program.   I have not been able to reproduce the problem situation where the device continuously shows a blue screen.

Hi,

Are you testing with a model 22CB25S ?

The problem only exists on a 22CB25S

Customer can ship you a 22CB25S for you to test

Can someone respond to questions in comment #17 and #20?

mlight tested on 22CV241.  We located a 22CCB25S, will retest.

From the hwid decode, the spec between  22CB25S and 22CV241 have the same panel but a different storage.

we need to know the answer to #17, #28 and what firmware do you have ?

The hwid doesn't always tell you everything. Can you dump the EDID for both and see if it's the same?

asking the customer about #17 and #20.
I will update the thread once I got answer.

Update from customer on #20

Spoke to store, they have not seen the issue since on v61 beta, but we only have 1. device running this. In another store, a clean rebuild from USB to v59 then updated to v60 has also not had the issue re-occue. Updating from v59 to v60, the issue still occured

2. I have to get back to stores to try this, maybe tomorrow

3.Did this problem start instantaneously on all affecting devices or have these issues been accumulating over a period of time? (in you have information)

Not sure, it took some weeks for it to be reported to me. Speaking to stores, it all started with update from v58 to v59.

Borrowed a Monroe model 22CB25S to try to reproduce the blue-screen problem in kiosk mode.  My schedule was a bit hectic yesterday so I would re-awaken the device at random intervals between 5 minutes and 1 hour, plus an overnight sleep last night.  The Monroe was a bit eccentric about what it takes to wake it up (sometimes a mouse click, sometimes <enter>, sometimes brief power button press), but it would wake up eventually and I did not see any blue screens.

Update from customer on #17

Test result for HMDI-In.

During Blue Screen,
- Plugging in laptop and changed to HDMI, this worked fine and laptop was displayed on the chromebase. 
- Changing back to chromebase, screen was black and not blue, could still not wake up device
- I also has a displaylink usb adapter which I brought with me. I plugged this in to an external screen, and it came up as a second monitor but the chromebase screen was still black
- I could move the mouse onto the 2nd monitor, click the menu on the bottom right and select sign out. 
- On signing out, device rebooted and then displayed 2 screens.


The customer has provided screen shots and new device log so I saved it in the shared drive link below.

https://drive.google.com/drive/folders/0B01YYztUbOCuaDU2RVpzTEZKclU?usp=sharing

new files name:HDMI-IN test result.zip(screen shot) and debug-logs_20171005-15444_during_HDMI-INtest.tgz

c#33 is a different test case.  The original bug is wake problem with chromeOS from chromebase, in c#33, you are using monroe as external display monitor.  

Please confirm which one is the customer use case ?

The wake is the failure, the external display is just a debug step. 

The laptop driving the Chromebase display working implies the monitor SoC is at least alive. 

The Displaylink USB adapter working proves Chrome OS is up and running properly, and it potentially gives us a window into the state of the system for further debug. 

So we need to determine why the Chrome OS board and the monitor SoC are no longer working together, even though they are working independently. 

I have not seen a modetest output yet here, we should get that from a feedback report if we can locate one from a system in this state, or it could be pulled from a displaylink display attached to a device in this state in about://system. The modetest utility would tell us what Chrome OS thinks the monitor is doing, but given Chrome OS appears to still be displaying to it (e.g. the mouse is able to be on the blank internal panel), I am afraid modetest will indicate everything is fine.

Stephane, any other ideas for debug of this?

@35: I would start with the output of modetest which the screen is blue. However at the end of the day, this is  likely to be related to the display input switching logic, of which I know nothing about.

This is really something for hw folks. Who is the hw eng contact for this device?

Please ship one of these failing devices to me (a specific unit that is showing this problem). We'll take a look.

Ok, we will work to provide you a failing unit from the customer or from LG.

I'm stuck on this page (see attached picture). Can't really get into dev mode. Any options for me to log into this device?

Deleted: IMG_9502.JPG 100 KB

	IMG_9502.JPG 100 KB View Download

Can you please reach out to me privately. 

Sent you an email.

srcrew@ reports that this is no longer an issue after the M62 update. Closing.