Waiting to hear back from the customer on the repro steps, and more details on the login screen.  I see these lines in the /var/log/chrome:

[790:790:0101/190004.494992:ERROR:xkb_keyboard_layout_engine.cc(865)] No current XKB state
[790:790:0101/190004.497434:ERROR:xkb_keyboard_layout_engine.cc(865)] No current XKB state
[790:790:0101/190004.497589:ERROR:xkb_keyboard_layout_engine.cc(865)] No current XKB state
[790:790:0101/190004.497967:ERROR:xkb_keyboard_layout_engine.cc(865)] No current XKB state

Could this be related?

I'm verifying with the customer, but I may have found the issue.

If you have another input device plugged in at the same time (in this case, a usb keyboard) as well as the touchscreen, on the login page only the keyboard will work as input.  You can still select items with the touchscreen, but it will not call the on-screen keyboard.  Once you login to the device, both methods work as input.

Testing this with a kiosk app, the same issue occurs.  Although since the kiosk app is loaded instead of logging into the actual os, it will only take the keyboard as acceptable input.  I checked the logs that were provided, and there was a keyboard attached to the device.  This is on Chrome version 70.0.3538.76.

I think the a11y keyboard should show up regardless of external keyboards, so I filed bug 906131 to look into this.  

I just got off the call with the customer, and the issue does happen without the keyboard attached.  It is not reproducible all the time, but when the issue does happen it persists through a few reboots.

One way they have found that fixes the issue when it occurs, is to plug in a usb keyboard and then unplug the device which resolves the on-screen keyboard issue.  

Keep in mind, although the on-screen keyboard doesn't come up, the touchscreen is fully functional, so I don't think this is an input issue.

Any thoughts as to why this would happen?

I may have been able to reproduce the issue.  In the video, the times that the text box was pressed was about 2-3 times and the keyboard never came up.  I was able to reproduce this issue.  It seems that the field isn't recognizing the input/call to bring up the keyboard?  Although the field is selected as the cursor is inside of it.

If I keep tapping that same area of the screen, eventually the keyboard will show (About 10+ times).  If I plug in a usb keyboard and remove it as soon as I see this issue, the next time I try to tap on the field, it comes up as expected.

Keep in mind that this is using a mimo touchscreen, and not the same ELO touchscreen the customer is using.  We are having one shipped to us by EOW.

@pbathini,

Can you attempt to repro this issue with the same Chrome OS version? (70.0.3538.76).  It will take multiple reboots to eventually show up.

I checked with 70.0.3538.76 / 11021.56.0 build on fievel device connected to ELO touchscreen. The virtual keyboard was showing up fine .

I tried to reboot the machine couple of times and check the virtual keyboard in guest mode and sign screen. It was showing up as expected.

Received an ELO touchscreen this morning and attempted to reproduce on 70.0.3538.76, and was unable to repro.  I also tested on 69.0.3497.95 and was also unable to reproduce as well.

For both methods, I rebooted the device 20+ times to see if the issue occurred.

Had a meeting with the customer, and they were able to reproduce the issue on their units.  After speaking with them, they also stated that their QA team was seeing this issue in other releases, but it has gotten better over time.

One thing to note is this only affects their Aopen devices which are arm-based.  Their intel based CXi3 chromeboxes don't appear to be affected.  I'm wondering if on some level the touch screen presents itself as a keyboard to chrome sporadically?

So I think the issue is that the touchscreen is being detected as a keyboard on some instances of boot, which causes this issue.  Looking at the logs again, there are 2 devices addresses listed:

ff540000 - Should be for Touch Digitizer

Atmel Atmel maXTouch Digitizer as /devices/ff540000.usb/usb2/2-1/2-1.1/2-1.1.1/2-1.1.1:1.1/0003:03EB:8A6E.0002/input/input3
2017-01-01T19:00:05.427185-05:00 INFO kernel: [    7.953934] hid-multitouch 0003:03EB:8A6E.0002: input,hiddev0,hidraw1: USB HID v1.11 Device [Atmel Atmel maXTouch Digitizer] on usb-ff540000.usb-1.1.1/input1

ff580000 - Should be for a Keyboard

2018-11-07T08:34:54.848442-05:00 INFO kernel: [63049.481254] input: USB USB Keyboard as /devices/ff580000.usb/usb3/3-1/3-1:1.0/0003:1A2C:2D23.0003/input/input5
2018-11-07T08:34:54.848691-05:00 INFO kernel: [63049.486158] hid-generic 0003:1A2C:2D23.0003: input,hidraw2: USB HID v1.10 Keyboard [USB USB Keyboard] on usb-ff580000.usb-1/input0
2018-11-07T08:34:54.867965-05:00 INFO kernel: [63049.501100] input: USB USB Keyboard as /devices/ff580000.usb/usb3/3-1/3-1:1.1/0003:1A2C:2D23.0004/input/input6
2018-11-07T08:34:54.868077-05:00 INFO kernel: [63049.503575] hid-generic 0003:1A2C:2D23.0004: input,hidraw3: USB HID v1.10 Device [USB USB Keyboard] on usb-ff580000.usb-1/input1


Now notice in these logs, the address that should be for the touchscreen seems to be assigned to a keyboard.  Also note the error -71 that occurs as well when this happens:


2018-11-07T08:34:54.848442-05:00 INFO kernel: [63049.481254] input: USB USB Keyboard as /devices/ff580000.usb/usb3/3-1/3-1:1.0/0003:1A2C:2D23.0003/input/input5
2018-11-07T08:34:54.848691-05:00 INFO kernel: [63049.486158] hid-generic 0003:1A2C:2D23.0003: input,hidraw2: USB HID v1.10 Keyboard [USB USB Keyboard] on usb-ff580000.usb-1/input0
2018-11-07T08:34:54.867965-05:00 INFO kernel: [63049.501100] input: USB USB Keyboard as /devices/ff580000.usb/usb3/3-1/3-1:1.1/0003:1A2C:2D23.0004/input/input6
2018-11-07T08:34:54.868077-05:00 INFO kernel: [63049.503575] hid-generic 0003:1A2C:2D23.0004: input,hidraw3: USB HID v1.10 Device [USB USB Keyboard] on usb-ff580000.usb-1/input1
2018-11-07T08:34:58.077960-05:00 ERR kernel: [63052.714551] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.078081-05:00 ERR kernel: [63052.715352] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.078108-05:00 ERR kernel: [63052.716361] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.099092-05:00 ERR kernel: [63052.729345] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.099203-05:00 ERR kernel: [63052.730346] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.099229-05:00 ERR kernel: [63052.731583] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.127745-05:00 INFO kernel: [63052.758607] usb 3-1: USB disconnect, device number 2
2018-11-07T08:34:58.127773-05:00 ERR kernel: [63052.759342] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.127777-05:00 ERR kernel: [63052.760392] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.127801-05:00 ERR kernel: [63052.761190] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.187780-05:00 ERR kernel: [63052.819211] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.187819-05:00 ERR kernel: [63052.820203] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.187824-05:00 ERR kernel: [63052.821208] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.297742-05:00 ERR kernel: [63052.929202] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.297772-05:00 ERR kernel: [63052.930187] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.297777-05:00 ERR kernel: [63052.931186] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.407805-05:00 ERR kernel: [63053.039261] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.407861-05:00 ERR kernel: [63053.040247] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.407875-05:00 ERR kernel: [63053.041247] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.517759-05:00 ERR kernel: [63053.149565] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.517788-05:00 ERR kernel: [63053.150203] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.517793-05:00 ERR kernel: [63053.151200] usb 2-1.1: clear tt 1 (9043) error -71
2018-11-07T08:34:58.538669-05:00 INFO kernel: [63053.169297] usb 2-1.1: USB disconnect, device number 3
2018-11-07T08:34:58.538703-05:00 INFO kernel: [63053.169315] usb 2-1.1.1: USB disconnect, device number 4
2018-11-07T08:35:13.657781-05:00 INFO kernel: [63068.288154] usb 3-1: new low-speed USB device number 3 using dwc2
2018-11-07T08:35:13.878093-05:00 INFO kernel: [63068.510723] usb 3-1: New USB device found, idVendor=0461, idProduct=4e17, bcdDevice= 1.32
2018-11-07T08:35:13.878258-05:00 INFO kernel: [63068.510896] usb 3-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
2018-11-07T08:35:13.878304-05:00 INFO kernel: [63068.511050] usb 3-1: Product: USB Keyboard
2018-11-07T08:35:13.897757-05:00 INFO kernel: [63068.530120] input: USB Keyboard as /devices/ff580000.usb/usb3/3-1/3-1:1.0/0003:0461:4E17.0005/input/input7
2018-11-07T08:35:13.897789-05:00 INFO kernel: [63068.530607] hid-generic 0003:0461:4E17.0005: input,hidraw0: USB HID v1.11 Keyboard [USB Keyboard] on usb-ff580000.usb-1/input0
2018-11-07T08:35:17.818273-05:00 INFO kernel: [63072.449017] usb 2-1.2: new low-speed USB device number 5 using dwc2
2018-11-07T08:35:17.918213-05:00 INFO kernel: [63072.555895] usb 2-1.2: New USB device found, idVendor=1a2c, idProduct=2d23, bcdDevice= 1.10
2018-11-07T08:35:17.918346-05:00 INFO kernel: [63072.556067] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
2018-11-07T08:35:17.918391-05:00 INFO kernel: [63072.556256] usb 2-1.2: Product: USB Keyboard
2018-11-07T08:35:17.918427-05:00 INFO kernel: [63072.556371] usb 2-1.2: Manufacturer: USB
2018-11-07T08:35:17.947779-05:00 INFO kernel: [63072.580711] input: USB USB Keyboard as /devices/ff540000.usb/usb2/2-1/2-1.2/2-1.2:1.0/0003:1A2C:2D23.0006/input/input8
2018-11-07T08:35:17.947812-05:00 INFO kernel: [63072.581278] hid-generic 0003:1A2C:2D23.0006: input,hidraw1: USB HID v1.10 Keyboard [USB USB Keyboard] on usb-ff540000.usb-1.2/input0
2018-11-07T08:35:17.947817-05:00 INFO kernel: [63072.585608] input: USB USB Keyboard as /devices/ff540000.usb/usb2/2-1/2-1.2/2-1.2:1.1/0003:1A2C:2D23.0007/input/input9
2018-11-07T08:35:17.947822-05:00 INFO kernel: [63072.586141] hid-generic 0003:1A2C:2D23.0007: input,hidraw2: USB HID v1.10 Device [USB USB Keyboard] on usb-ff540000.usb-1.2/input1


I"m going to request some more recent logs on the issue to see if i see the same pattern.

Was able to repro the exact issue.  The key is when restarting device, you need to just pull the power and plug it back in.  Before when I was testing, i would either gracefully shut down using the menu, or holding the power button.

I was able to reproduce this issue in 2 separate reboots, back to back.

Attaching logs from my device.  Time of issue was around 11:03am.  After rebooting the device by pulling the power, I noticed that the screen wasn't turned on.  I unplugged the hdmi, and then plugged it back in and the screen came back up.  I then tried to call the on-screen keyboard and it would not pop-up.  

Also in this state, if I try to manually call the on-screen keyboard by enabling it in the accessibility menu, it does not enable.  The icon changes to white as if it was enabled, but nothing appears on the screen.

Deleted: ofrancis_Repro_Aopen.tgz 825 KB

ofrancis_Repro_Aopen.tgz 825 KB Download

@pbathini, can you try with the following repro steps:

1. Boot the device with just the ELO touchscreen attached
2. Attempt to tap on a text field to call the keyboard
3. If it works, reboot the device by pulling the power

Within a few reboots the issue seems to occur.  It will also persist on another reboot if you pull the power to do so.

Also using a script from Jay, i was able to view what ChromeOS thinks is plugged into the unit by calling lsusb every second after boot.  It does not appear that the OS thinks that a keyboard is attached, so the issue may lie somewhere else.

Curiously I wasn't able to reproduce if I shut down the unit via the UI or by holding the power button. 

dianders@ and dtor@ - this feels very similar to  crbug.com/850222  - same hardware and seems to be Veyron USB related. Could Chrome be getting weird USB signals that cause it to think a physical keyboard is present and not show onscreen keyboard?

@29: It's really hard to follow all the above conversation.  I'm also not an expert on the rules that the Chrome browser uses to determine whether to show the on-screen keyboard.  However:

1. Some quick testing on a machine in front of me makes me believe that plugging in a keyboard makes the on-screen keyboard stop showing up.

2. Since the touchscreen is still working OK, it's plausible to guess that the system thinks a keyboard is plugged in and thus won't show the on-screen keyboard.

3. The comment about "you need to just pull the power and plug it back in" makes me believe that somehow this is causing the touchscreen to get itself into a bad state.  Maybe the VBUS to the device is dropping somewhat below 5 V but not discharging all the way?  Then the device crashes but doesn't get fully reset and it doesn't talk very well on the next bootup?  It would be interesting to know if you unplugged the device for a longer period of time if it fixed the issue.

4. My guess is that the USB touchscreen has some sort of fallback config where if it doesn't properly enumerate then it goes into a mode where it enumerates as both a keyboard and a touchscreen.  ...and this is confusing us.

---

Does that seem sane?  If it's right then maybe the easiest solution is to blacklist and refuse the enumerate the keyboard part of this touchscreen?  What do you think Dmitry?

Ah, I should clarify that the device that i'm pulling the power on is the Aopen Chromebox, not the touchscreen itself.  The touchscreen always has power through another plug.

Also in this state, even attempting to manually bring up the on-screen keyboard via the accessibility menu doesn't work.

Running Jay's script (https://github.com/jay0lee/cros-scripts/blob/master/log-hids.sh) in dev mode reported the following output from lsusb:


--When the issue is experienced--
Bus 003 Device 003: ID 03eb:8a6e Atmel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The same output is shown when the issue doesn't appear as well, as in there is no difference in the lsusb output in bad and good states.

Re: #30, yes, the virtual keyboard will not open if there's an external keyboard attached. If you want the virtual keyboard to always show, there's a policy you can set (see b/117556810).

I don't know much about the hardware side unfortunately...

It is Chrome that decides whether to show virtual keyboard or not... In the comment #31 I do not see a keyboard attached, just a touchscreen.

Correct, which may rule out the theory that the device is being detected as a keyboard.

But the main issue still remains, as the on-screen keyboard doesn't show up in this state either by tapping on the screen, or calling the keyboard manually. 

@35: If the touch screen works but the virtual keyboard won't come up then the problem sure seems to be above the level of the kernel and somewhere in UI.

I could sorta believe that we might do USB enumeration in a different order or at a different speed or maybe we have trouble enumerating and have to re-enumerate once.  Perhaps whatever rules Chrome uses has a race condition and can't handle it?

Can you tell if the device is in tablet mode? The VK listens for tablet mode changes in order to decide if it should show or not.

If it's not in tablet mode, then it's not a VK issue.
If it's in tablet mode, then it's a VK / app / kiosk issue.

Sorry, that's "does the OS think it's in tablet mode or not" (can you see UI that is only shown in tablet mode?)

The issue only happens at the login screen to the device, or when a kiosk app is force loaded.  I presume that when Chrome OS loads a kiosk app, it is in a similar environment to the login page as it hasn't fully loaded into the OS.

The UI looks exactly the same whether or not the issue occurs.

Silly question. Would a setting in the Chrome Device Management console be the solution to go?

We could have a setting that dictates whether the on-screen keyboard is displayed if an HID keyboard is detected or not. The setting could be:
1 - Default (current behavior)
2 - Always hide the on-screen keyboard
3 - Always show the on-screen keyboard

The issue here would be that when the bug is triggered (by restarting due to power loss), it isn't respecting the settings for the on-screen keyboard. 

Something related to shutting down the device in an unclean state triggers this issue.  This wouldn't be something that would be solved by a policy setting in Chrome Device Management, I think the issue is more lower level than that.

Has anyone been able to reproduce the issue on the eng side with my steps listed in #27?  In those steps "device" refers to the Chromebox.

I totally agree with you. This would be more like a workaround than anything else. 

By implementing this setting, the OS itself would not even worry about whether there's a keyboard or not and would render the keyboard in the textbox fields.

From the video that was posted in the bug description, there was no 'Auto-rotate' icon/setting in the system tray, which indicated that the device was not in tablet mode when this issue happened. So I think it's either a virtual keyboard issue, or a kiosk issue. 

Assign to keyboard owner for investigation. 

Hmm, yes, the original video had no auto-rotate icon in the system tray, so the device was not in tablet mode. In that case the VK is working as intended.

Re-assigning back to bigo@ for hardware investigation. There must be something weird going on where some device is being recognized as an external keyboard or something. Or this is a problem with kiosk mode.

Regarding a quick workaround (#40), please see b/117556810. We already have an existing device policy for this.

The video above shows devices in kisok mode, and a login screen that appears after you exit the auto launch of an app in kiosk.  The only way to get the screen to rotate 90 degrees with these devices is to put them in kiosk mode which exposes the policy in admin console to rotate the screen.

The touchscreen by itself has no ability to rotate the screen.  When reproducing this, the only device plugged in is the touchscreen device as stated.  Also stated above, we ran a script that ran lsusb every second to rule out the fact that a keyboard was being detected.

I can also reproduce the issue outside of kiosk mode, with the screen orientation set to normal so this is not limited to just kiosk mode.

Regarding b/117556810, the device functions normally when you start it up.  I don't believe the issue is with policy.  If i shut down the device with the power button, or shut down the device via the UI I will never run into this issue.  The issue only occurs of the device is shut down via power loss (pulling the cord).  Seeing that these devices are primarily being used by the customer for ordering kiosks, I would imagine that these units arent necessarily located in ideal locations to power down gracefully most of the time. 

Hmm alright, so the problem is that when you pull the power from a Chromebox, the VK doesn't show upon start up?

Unfortunately I don't have access to a Chromebox, so I'm not sure how I can try to repro this.

Correct, the Aopen ME4100 is a Chromebox device, and the issue is just as you described.  

shend: you can request a AOpen "fievel" Chromebook via crosdistros@ - feel free to call me on the request, I'm happy to relay how urgent the need is for the customer issue.

Okay, I managed to find a dusty Chromebox in the office. I plugged it into a touchscreen device (displayport + USB) and upgraded it to latest stable. I pulled the power, but after booting up, the VK shows up fine (on login screen and in kiosk app). I did this several times.

The Chromebox is a HP Chromebox. The touchscreen device is some Dell display.

It needs to be a fievel unit, the issue seems specific to it. Again, please reach out to crosdistros@ and cc me.

Ah okay, I've sent a request. Thanks!

Hi bigo@, sorry I'm just rereading this thread in more detail now:

> If I keep tapping that same area of the screen, eventually the keyboard will show (About 10+ times). 

Is this still true with the power plug repro? What if you tap once, and then wait a while. Does the VK show up eventually?

> One way they have found that fixes the issue when it occurs, is to plug in a usb keyboard and then unplug the device which resolves the on-screen keyboard issue. 

Are you able to repro this too?

No, with the power plug repro the keyboard never shows up no matter how many times I tap.  That step was with a Mimo touch screen, and was before I gathered more information about pulling the power plug.

I have tried plugging in a keyboard and then unplugging the device, but that didn't work for me.  The only thing that worked was to login to guest mode, and then log back out.  This restored the functionality of the VK.  According to the customer, after a number of restarts in this fashion it will eventually come back to normal.  I believe that if you shut down the device cleanly, it will return to normal, but I will verify tomorrow.

Interesting, thanks!

So after this issue occurs, I have confirmed that if the device is restarted afterwards by either holding the power button or using the UI to shut down, the functionality returns to normal.

As these units are primarily used as kiosks, the ability to remotely restart the device is available in the admin console.  More information about this can be found here under "Reboot Device". (https://support.google.com/chrome/a/answer/6179663?hl=en)

This could be a workaround while we narrow down the root cause of the issue.

The customer is requesting an ETA on a fix for this issue, as the last issue that they experienced took months to resolve for them, and they would like to avoid this as this is a blocker to their migration in their production environment.

+googleo@ who is taking a look today or tomorrow in MTV. If we can't figure it out, we'll have to ship a fievel unit to SYD, unless someone else from MTV can help out.

How certain are we that the power plug issue is the same issue that the customer is experiencing?

bigo@ can you please arrange to ship a Fievel to Sydney as a proactive
measure in case they can't reproduce?  Thanks

@59 - already in progress.

@58 - It was verified by the customer that everytime that they encounter the issue, they restart it in this fashion.  I have seen the issue over video conference, so im pretty confident that this is the case.

Fantastic, thank you!

Monday morning update:

I was able to reproduce the same issue on a Chromebit(Minnie) device, so it may be a chipset issue as both the Aopen Chromebox as well as the Chromebit device are ARM based.

The customer has intel based Chromeboxes as well where this issue does not occur.  I will test with an intel based Chromebox to verify on my end.

With an Intel based Chromebox, i'm unable to reproduce the issue.  The VK comes up everytime without failure.

Thanks bigo@.

googleo@, could it be an error in the NaCL build for ARM?

The device has been delivered according to the tracking.

@shend,

Let us know when you have the device in hand, and have started working on reproducing/diagnosing the issue.

Yes I have received the device, thanks! Unfortunately I was not able to reproduce the issue. I'll note my repro steps; could you please check if I've missed anything?

1. The set up
I plugged in a touchscreen monitor to the device (HDMI + USB) and connected it to the power. See attachment.

2. Clean boot
I booted the device normally and tapped a text field on the initial screen. VK shows as expected. See https://drive.google.com/file/d/1_s8kh5GWcaYo3hJHZfFJBvU1ICRmJvYb/view?usp=sharing

3. Pull power
While the device is on, I pulled the power from the device. Waited a few seconds and plugged it back in. Machine boots up automatically by itself. I tapped the same text field. VK shows as expected. See https://drive.google.com/file/d/1KXhZ7iSPadqd5bGNDOewsk64mBXQNV9q/view?usp=sharing

I also repeated #3 a couple of times, but still couldn't repro.

Deleted: IMG_20181217_094022_1.jpg 3.9 MB

	IMG_20181217_094022_1.jpg 3.9 MB View Download

Hmm, yes those would be the exact repro steps.  What version is the device currently on?

Can you test by wiping the device, and joining a test domain besides google.com?  Just to be more in line with the customer's setup.

I'm wondering if the issue is sporadic between devices?  I can reproduce it, as well as the customer.  Do you happen to have another Arm based device (Chromebit) that you would also be able to test as I was able to reproduce it on that device as well?

Thanks, by wiping, I can just do a powerwash? How do I join a test domain besides google.com? Sorry I'm not very familiar with managed devices

Instructions for wiping a Chromebox can be found here:
https://support.google.com/chrome/a/answer/1360642?hl=en

I will send you test credentials where you can enterprise enroll it to my test domain.  Instructions for this can be found here:

https://support.google.com/chrome/a/answer/1360534?hl=en

Just to update: I can repro the issue. We're looking into finding the root cause by putting in a test image. I've been quite busy so we haven't made much progress yet. Will update again tomorrow.

Just got back from vacation. Flashed device with 71.0.3578.94 (beta) recovery image. Can repro the issue.

Will try to repro on the test image with same version.

Can repro on test image as well after a few tries (enable a11y keyboard and click on the text field; no touch screen required).

Will add some additional logging to see what's happening.

Can repro with custom logging. Basically, what usually should happen is:

1. VK container loads the VK UI as a web page
2. VK UI javascript resizes the virtual keyboard window
3. VK container changes visibility

However, it appears that we are missing step #2. That is, the VK is being shown, but with 0 height because the JavaScript code has not told the container the correct height. This probably points to a JavaScript error in the keyboard extension, so I'll see how I can get those logs from the login screen.

Ok, there looks like a JavaScript error happening in the VK extension upon load. For some reason, some performance tracking code is erroring because some timestamps are incorrect. Still looking into this. Hopefully this is not a red herring :P

Well, not sure if it's the issue, but when part of the chrome extension loads...it calls "new Date.getTime()", which returns 1483315205371 (or "Mon 2 January 2017
11:00:05"). Later on, it gets the correct timestamp. When it tries to log the difference, it calculates a duration of more than a year, which errors out because that doesn't fit in a 32 bit integer...

Could it be that when Chrome OS boots in a certain way, it gives new Date.getTime() returns incorrect values?

I commented out the line that does the logging and it seems to have fixed it (altho you can never be sure if it's flaky...)

Meanwhile I'll file a bug about this.

bigo@, how urgently is a fix required? Ideally we'd get to the bottom of this, but we can also opt for a quick hacky fix if necessary.

After talking with some Chrome OS folks, it might be related to hardware rtc clocks or something. Anyway, the root issue is probably so obscure that it'll never be fixed. Let's just fix it on the JavaScript side.

I'll try and land a fix tmr and we can verify on Canary.

Fix is on the way.

Fun fact: you can actually see the issue happening if you look at the clock on the bottom right. When booting normally, the clock is correct from the beginning. When booting after a power outage, the time is way off.

Thanks for the update shend@.  I have updated the customer that a fix is in the works.  Once the fix is deployed and confirmed working, I can request a merge down to 71.

Yes, Chromebox devices do not have a clock at all nor a battery backup so when they boot after full power loss they have no idea what time it is. The best guess they can make is that it's at least the time of the OS build date so they use that until they can get online and use tlsdate to pull the real time from Google servers.

+atwilson@chromium.org - any idea why we're assuming 2017 here when 71 would have been compiled in late 2018? I'm concerned we're jumping dates a lot on these cold boots and it's confusing to pieces further up the stack.

It really should not matter what time we boot with, or what time we get form an NTP server. Our stacks should be prepared to handle arbitrary jumps in time.

store: agreed but Chrome OS has a long history of time sync issues triggering problems further up the stack. We can track the time sync weirdness in https://crbug.com/918000

@shend, 

Can you let me know what Canary version this will land in?  I would like to update the customer so that they can test on a subset of their devices.

Hi bigo@, we're still trying to get our fix approved (https://chrome-internal-review.googlesource.com/c/chromeos/overlays/chromeos-overlay/+/749953). Some delays right now because most of our team is away. I'm trying to find someone who can approve the CL.

Ok we got approval, so just waiting for it land now. Might take 2 - 3 days to land, and a few more days to hit Canary?

Chrome OS CQ is in a bad state right now :/, so landing might take longer than expected.

Got it, thanks for the update.  I will keep the customer informed of the progress and set correct expectations.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/chromeos-overlay/+/334244524e4798f7837f3e8965b2b23bd6020010

commit 334244524e4798f7837f3e8965b2b23bd6020010
Author: Chen Gong <chengong@google.com>
Date: Thu Jan 03 23:04:37 2019

Fix landed, will check Monday next week to see if hits Canary.

Fix landed in 11520.0.0 and newer (thanks shend@) - https://crosland.corp.google.com/cl?q=334244524e4798f7837f3e8965b2b23bd6020010

@Omar: can you validate the fix on canary channel? You'll need to grab the image and USB recover since this canary build isn't live on canary channel just yet.

I'll test this today.

I flashed the device to version 11522.0.0 and was unable to reproduce the issue after 5+ attempts of rebooting the device by pulling the power.

I will update the customer with the information.

This bug requires manual review: M72 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: govind@(Android), kariahda@(iOS), djmm@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

re #92: Fantastic! I'll get a cherry pick CL ready for merge by EOD.

Thanks!  Tests were also confirmed by the customer to have fixed the issue.  I will keep them updated on the progress of the merge.

> any idea why we're assuming 2017 here when 71 would have been compiled in 
> late 2018? I'm concerned we're jumping dates a lot on these cold boots 
> and it's confusing to pieces further up the stack.

The PMIC (Power Management IC) is used for the clock on rk3288-based boards.  ...so it's possible that the 2017 comes from there.  At least some version of the RTC datasheet claims that the RTC will return "13" for the year at reset, which would make be think it would default to 2013.  ...but maybe ones that were manufactured in newer years have a different default?

It's also possible that the BIOS (which accesses the RTC) inits it to something sane if it doesn't make sense.

If it's important we could track this down, but it seems likely we wouldn't be able to fix all old hardware without big hacks / assumptions about what the BIOS and RTC is doing.

Thanks bigo@, that's good to hear. Our M72 cherry pick is going through internal QA testing right now. Will update when that's done.

Thank you for the updates. I met with the customer today and they confirmed the fix worked. They are looking for confirmation if and when the fix might hit m71. 

Hmm, are we planning to merge to M71? I think we're rolling out M72 to stable at the end of the month right?

Yes.  I believe there are talks of another 71 version, but nothing has been confirmed.  I have requested a merge to 71 for this scenarios in order to try to get the fix to the customer sooner.

+ kbliecher to comment on plans for 71 respin or not.

Yes, we're planning a M71 respin but any merges would need to go in asap to make it.  Is there a verified CL ready for review?  Is this a critical fix?

Yes, this is a critical fix for our largest enterprise customer who has been stuck on 63 for some time due to regressions and bugs in every release since.

We have a CL ready for M71: https://chrome-internal-review.googlesource.com/c/chromeos/overlays/chromeos-overlay/+/781350

We're still doing QA for M72. Will keep you updated.

this needs to be merged asap or miss the respin; looks like we won't have time to get this in.

Wait sorry I thought there was no merge approval for M71 yet. I can submit it now.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/chromeos-overlay/+/61d690c2f9b51f0ea22a5e1577087482e93dabd3

commit 61d690c2f9b51f0ea22a5e1577087482e93dabd3
Author: Darren Shen <shend@google.com>
Date: Fri Jan 11 05:14:16 2019

Looks like the merge landed in 11151.114.0.

@kbleicher, is that version <= to the 71 respin that is going out?  I want confirm whether it made it or not.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/chromeos-overlay/+/5be967bce4a18c7e9fd9c776108da4d80ac9e59e

commit 5be967bce4a18c7e9fd9c776108da4d80ac9e59e
Author: Darren Shen <shend@google.com>
Date: Wed Jan 16 02:23:54 2019

It looks like this change didn't make it in time for the respin to 71.  This should be available in version 72 which the current ETA has it landing towards the beginning of next month: https://chromiumdash.appspot.com/schedule