All the powerd logs that I looked at from https://drive.google.com/open?id=1ZV4NABn2i3MscM8N0w3tNNoIhcFDZPkv show the system shutting down due to the power button being pressed:

[0131/084626:INFO:daemon.cc(525)] Power button down
[0131/084626:INFO:main.cc(218)] Launching "sync"
[0131/084626:INFO:activity_logger.cc(20)] User activity reported
[0131/084627:INFO:daemon.cc(1163)] Got RequestShutdown message from :1.8 with reason user-request (UI request from ash)
[0131/084627:INFO:display_power_setter.cc(81)] Asking DisplayService to turn all displays off
[0131/084627:INFO:internal_backlight_controller.cc(677)] Setting brightness to 0 (0%) over 0 ms
[0131/084627:INFO:daemon.cc(1585)] Shutting down, reason: user-request

[0131/084710:INFO:daemon.cc(525)] Power button down
[0131/084710:INFO:main.cc(218)] Launching "sync"
[0131/084712:INFO:daemon.cc(1163)] Got RequestShutdown message from :1.8 with reason user-request (UI request from ash)
[0131/084712:INFO:display_power_setter.cc(81)] Asking DisplayService to turn all displays off
[0131/084712:INFO:internal_backlight_controller.cc(677)] Setting brightness to 0 (0%) over 0 ms
[0131/084712:INFO:daemon.cc(1585)] Shutting down, reason: user-request

[0131/090406:INFO:daemon.cc(525)] Power button down
[0131/090406:INFO:main.cc(218)] Launching "sync"
[0131/090407:INFO:daemon.cc(1163)] Got RequestShutdown message from :1.8 with reason user-request (UI request from ash)
[0131/090407:INFO:display_power_setter.cc(81)] Asking DisplayService to turn all displays off
[0131/090407:INFO:internal_backlight_controller.cc(677)] Setting brightness to 0 (0%) over 0 ms
[0131/090407:INFO:daemon.cc(1585)] Shutting down, reason: user-request

[0131/090435:INFO:daemon.cc(525)] Power button down
[0131/090435:INFO:main.cc(218)] Launching "sync"
[0131/090435:INFO:activity_logger.cc(20)] User activity reported
[0131/090435:INFO:daemon.cc(525)] Power button up
[0131/090446:INFO:daemon.cc(525)] Power button down
[0131/090446:INFO:main.cc(218)] Launching "sync"
[0131/090447:INFO:activity_logger.cc(20)] Video activity stopped; last reported 20 sec ago
[0131/090447:INFO:daemon.cc(1163)] Got RequestShutdown message from :1.8 with reason user-request (UI request from ash)
[0131/090447:INFO:display_power_setter.cc(81)] Asking DisplayService to turn all displays off
[0131/090447:INFO:internal_backlight_controller.cc(677)] Setting brightness to 0 (0%) over 0 ms
[0131/090447:INFO:daemon.cc(1585)] Shutting down, reason: user-request

The system goes to sleep at one point, but I don't know if that's what's being reported:

[0131/090521:INFO:activity_logger.cc(20)] User activity stopped; last reported 20 sec ago
[0131/091201:INFO:state_controller.cc(89)] Dimming screen after 7m
[0131/091201:INFO:internal_backlight_controller.cc(677)] Setting brightness to 102 (34.9032%) over 200 ms
[0131/091301:INFO:state_controller.cc(89)] Turning screen off after 8m
[0131/091301:INFO:internal_backlight_controller.cc(677)] Setting brightness to 0 (0%) over 200 ms
[0131/091301:INFO:display_power_setter.cc(81)] Asking DisplayService to turn all displays off
[0131/091301:INFO:daemon.cc(1412)] Chrome is using normal display mode
[0131/091355:INFO:daemon.cc(1525)] Update operation is UPDATE_STATUS_CHECKING_FOR_UPDATE
[0131/091355:INFO:daemon.cc(1525)] Update operation is UPDATE_STATUS_IDLE
[0131/093501:INFO:state_controller.cc(953)] Ready to perform idle action (suspend) after 30m

I don't think anyone can do anything more without more information about the time at which the reboot(s) occurred. If you open the corresponding log file from /var/log/power_manager, it should end with information about why the system is rebooting.

 derat@
Is there any potential problem if the customer does not touch the power button during the meeting?

#2: Sorry, I don't understand the question. I didn't see any reboots in the logs that I looked at. It's expected that the device shuts down if the user is pressing the power button. I'll need more information about when the issue occurred and/or the correct logs in order to diagnose this further.

We're seeing this behavior with some devices here on our domain. Do not think this is an isolated incident. 

To clarify; we're seeing random reboots of our Chromebase units within out fleet. Can confirm users are not holding down the power button.

I'm not entirely sure how to provide these logs, but in the past few days I had an Acer chromebase 24 suddenly get Error 002 - cannot connect to hangouts/meet. This occurs every few mins in a meeting and even at the calendar view.

I pulled out another chromebase 24 yesterday and for about 24 hours it was fine, but it too is now exhibiting the same Error 002 issue. The power button is not being held during meetings or at the calendar view.

More than happy to help figure out this issue.

I'd like to add a secondary observation.

Between the hours of 9AM CST-11:45AM CST (7AM-9:45AM PST), these same chromebases exhibit no issues. It appears that once people on the west coast "wake up", or in other words, once the entire US is using Meet/Hangouts, these chromebases begin exhibiting issues.

The network that these chromebases are installed has no increase in traffic.

You can save the logs by visiting chrome://net-internals/#chromeos

and hitting the 'Store debug logs' button which should leave them in your Downloads directory

Is that only for desktop mode? These devices are in the hangout mode, where the chrome browser isn't shown.

egomez@pinterest.com, 
You can restart Cfm and enter ChromeOS mode by pressing Ctrl + Alt + S during boot, then get ChromeOS debug logs from chrome://net-internals/#chromeos. Please note the date/time of issue when you share logs with us. You can also contact Google Cloud support if you would like additional help in getting logs.

By rebooting the CFM, it looks like I lose all logs prior to the reboot and/or debug logs only appear when on this page?

If so, that will make this borderline impossible to debug/capture as the logs will be wiped during the issue.

This issue just happened three times in an hour meeting and is highly disruptive. Out of 5 meetings, only the early morning one worked without Error 002

No, just a reboot doesn't delete logs.

We have got a fresh logs from the customer with a timestamp.
https://drive.google.com/open?id=1rLfVqkQCPg7Gzzmj7zvfy2habd-8PhPA

Device rebooted 2018-02-14 at 10:51 -10:54
 
2018-02-14T10:51:09.010837+01:00 ERR chrome[1061]: [1061:1061:0214/105109.010619:INFO:remote_commands_service.cc(36)] Fetching remote commands.
2018-02-14T10:51:09.622003+01:00 ERR chrome[1061]: [1061:1061:0214/105109.621916:INFO:remote_commands_service.cc(161)] Remote commands fetched.
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2018-02-14T10:54:50.454
206+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset
2018-02-14T10:54:50.454278+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpu
2018-02-14T10:54:50.454282+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuacct

I can also see another reboot on 2018-02-05 at 14:01 - 14:04

2018-02-05T14:01:02.460556+01:00 INFO kernel: [  132.516655] haswell-pcm-audio haswell-pcm-audio: FW loaded, mailbox readback FW info: type 01, - version: 00.00, build 77, source commit id: 876ac6906f31a43b6772b
23c7c983ce9dcb18a19
2018-02-05T14:01:09.173955+01:00 ERR cras_server[2414]: Fail to create conversion matrix due to invalid channel layout
2018-02-05T14:01:10.191557+01:00 INFO kernel: [  140.250548] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
2018-02-05T14:01:33.705549+01:00 ERR kernel: [  163.774883] go2001 0000:03:00.0: go2001_dec_g_fmt_cap:1579 Frame info not available yet
2018-02-05T14:04:20.461814+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset
2018-02-05T14:04:20.461883+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpu
2018-02-05T14:04:20.461888+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuacct

we have got another debug logs from customer:
https://drive.google.com/open?id=1iGL8Mulvf32UDQiMkt4O15VfqkR-p5hh

time stamps: 10:22 AM and 10:27 AM 

2018-02-16T10:20:43.662155+01:00 INFO kernel: [  173.031028] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
2018-02-16T10:21:25.053278+01:00 ERR cras_server[2293]: Fail to create conversion matrix due to invalid channel layout
2018-02-16T10:22:32.639107+01:00 ERR kernel: [  282018-02-16T10:23:36.524155+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset

--
2018-02-16T10:26:25.172104+01:00 INFO kernel: [  170.903799] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
2018-02-16T10:27:26.488836+01:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpus

Thanks for the timestamps! eventlog.txt:

...
223 | 2018-02-16 10:17:49 | Kernel Event | Clean Shutdown
224 | 2018-02-16 10:17:50 | System boot | 183
225 | 2018-02-16 10:17:50 | System Reset
226 | 2018-02-16 10:17:50 | Wake Source | GPIO | 27
227 | 2018-02-16 10:23:34 | System boot | 184
228 | 2018-02-16 10:23:34 | SUS Power Fail
229 | 2018-02-16 10:23:34 | System Reset
230 | 2018-02-16 10:23:34 | ACPI Wake | S5
231 | 2018-02-16 10:23:34 | Wake Source | Power Button | 0
232 | 2018-02-16 10:23:34 | Wake Source | GPIO | 27
233 | 2018-02-16 10:27:24 | System boot | 185
234 | 2018-02-16 10:27:24 | SUS Power Fail
235 | 2018-02-16 10:27:24 | System Reset
236 | 2018-02-16 10:27:24 | ACPI Wake | S5
237 | 2018-02-16 10:27:24 | Wake Source | Power Button | 0
238 | 2018-02-16 10:27:24 | Wake Source | GPIO | 27
239 | 2018-02-16 10:28:21 | Kernel Event | Clean Shutdown
240 | 2018-02-16 10:28:21 | ACPI Enter | S5
241 | 2018-02-16 10:28:26 | System boot | 186
242 | 2018-02-16 10:28:26 | SUS Power Fail
243 | 2018-02-16 10:28:26 | System Reset
244 | 2018-02-16 10:28:26 | ACPI Wake | S5
245 | 2018-02-16 10:28:26 | Wake Source | Power Button | 0
246 | 2018-02-16 10:28:26 | Wake Source | GPIO | 27
EOF

powerd.20180216-101753 ends abruptly after 10:22:22.
powerd.20180216-102337 ends after 10:24:44 with some junk suggesting an unclean shutdown.
powerd.20180216-102727 ends at 10:28:19 with a clean shutdown caused by the power button being pressed.

In the messages file, I see the same two abrupt boots quoted in #16. There's 40-60 seconds of time between the last message and the cgroup message logged after the system booted again.

There are a few more messages in net.log with timestamps shortly after the 10:21:25 message from cras_server:

1116:2018-02-16T10:21:33.395348+01:00 INFO shill[2050]: [INFO:service.cc(296)] Service 5 destroyed.
1117:2018-02-16T10:21:33.404934+01:00 INFO shill[2050]: [INFO:service.cc(296)] Service 4 destroyed.
1118:2018-02-16T10:21:33.405134+01:00 INFO shill[2050]: [INFO:wifi_service.cc(783)] Rep ep updated for 3. [SSID=MCB], bssid: 00:27:e3:a5:bb:4d, sig: -83, sec: 802_1x, freq: 5660
1119:2018-02-16T10:21:33.405691+01:00 INFO shill[2050]: [INFO:service.cc(296)] Service 6 destroyed.
1120:2018-02-16T10:21:33.406144+01:00 INFO shill[2050]: [INFO:service.cc(296)] Service 3 destroyed.
1121:2018-02-16T10:21:33.406570+01:00 INFO shill[2050]: [INFO:service.cc(296)] Service 7 destroyed.
1122:2018-02-16T10:21:33.407274+01:00 INFO shill[2050]: [INFO:service.cc(296)] Service 8 destroyed.
1123:2018-02-16T10:21:33.407867+01:00 INFO shill[2050]: [INFO:service.cc(312)] Suppressed autoconnect to service 2 (no carrier)

I don't see any clues in the logs about what went wrong, though. I think that they rule out any decisions from userspace to reboot the system, and may also suggest that the problem is somewhere below the kernel.

I definitely think the issue is low level and is a regression from some
update, as my boxes continually crash even when not connected to a
hangout/team meeting.

Hi all, are there any updates I can share with the customer or any additional information we can provide? Thanks

I hardwired my acer today and my crashing issues instantly went away, both at the calendar view and in the meetings.

I'm now assuming something is wrong with the wireless driver/controller.

Thanks for the debug and status update.

Just want to confirm by 'hardwired' you connected built-in LAN (RJ45 connector) via network cable or did you use a different method (USB -> ethernet dongle)?

Plausible roots could be as you mentioned something with wifi (iwlwifi) 
          Subsystem: Intel Corporation Dual Band Wireless-AC 7260

Or perhaps just that using a hardwired connection inhibits some deeper low power state entry which was causing the power loss/reboot.

Could try plugging usbkey into one of the chromebox ports which should inhibit deeper LP state and  see if that makes wifi more stable.

Also does anyone have more details about date/channel when this was first seen?

When I look through commits for R62 (mentioned in reporting)

  https://crosland.corp.google.com/log/9901.1.0..9901.77.0

Nothing jumps out to me ATM.

Correct - built-in LAN/RJ45.

As far as dates, I believe I first started seeing this issue around December 14th, or at the very least this was the first time I started discussing it internally.

I don't have a USB dongle for network, but will report back if I see any issues come back. So far I have been in six successful meetings without any crashes.

Oh actually I see you just wanted me to plugin a USB key in general.

I actually have a USB microphone plugged in at all times (and yes I did try removing it for a while to see if that was the issue) which requires power and that didn't help, so I don't think it's the presence of USB itself.

Hi! Is there any update on this?

For the customer originally reported this issue:

- I see customer moved Cfm devices to Beta and confirmed the issue persists, or even became worse. Let's thank them for cooperation and ask them to put back to Stable (+ USB recovery to M64 unfortunately, since changing channels doesn't downgrade from ChromeOS M65)

- Is it possible to try wired network and see if the issue persists? We can confirm whether the original report and #c20 are the same issue. My guess is that multiple issues are discussed here, because the original report mentions "No issue outside of meetings" but #c18 states otherwise.

- It would be good to have more recent samples. Can we ask them to submit feedback after they see the issue, if it takes time to manually gather ChromeOS debug logs?

Ryuta, are you able to work w/ customer to work through comments in #c25?

In logfiles I see following:

2018-01-31T02:10:37.811993+01:00 ERR kernel: [   35.778039] ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
2018-01-31T02:10:37.812016+01:00 ERR kernel: [   35.778064] ata1.00: failed command: READ FPDMA QUEUED
2018-01-31T02:10:37.812018+01:00 ERR kernel: [   35.778085] ata1.00: cmd 60/e0:00:78:9c:1e/00:00:00:00:00/40 tag 0 ncq 114688 in
2018-01-31T02:10:37.812021+01:00 ERR kernel: [   35.778085]          res 40/00:00:e0:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
2018-01-31T02:10:37.812023+01:00 ERR kernel: [   35.778115] ata1.00: status: { DRDY }
2018-01-31T02:10:37.812024+01:00 INFO kernel: [   35.778135] ata1: hard resetting link
2018-01-31T02:10:38.076950+01:00 INFO kernel: [   36.043123] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
2018-01-31T02:10:38.076972+01:00 INFO kernel: [   36.043834] ata1.00: configured for UDMA/133
2018-01-31T02:10:38.076975+01:00 INFO kernel: [   36.043890] ata1: EH complete
2018-01-31T02:10:38.133002+01:00 INFO kernel: [   36.099171] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
2018-01-31T02:10:38.163420+01:00 INFO kernel: [   36.129837] tpm_tis tpm_tis: command 0xa (size 10) returned code 0x0
2018-01-31T02:10:38.178738+01:00 INFO crash_reporter[2571]: Enabling kernel crash handling

Also they have LiteOn SSD
So how about merging to  crbug.com/803716 

Re: #25
No more Lite-on SSD-related errors for original customer, but issue is still present.

25.1: It's happening when USB-recovered to stable 62,63,64,65

25.2: Checking if it's happening on wired network.

25.3:
Fresh logfiles:
https://drive.google.com/open?id=1FcbMnD8xXnnP0I-gE9M7V3JDL5afo-pW

25.2: working fine on wired connection, but often reboots during the meeting on wifi.

The fresh logs look similar to comment #c1 in that its a user-requested shutdown.

From powerd.PREVIOUS:
[0413/041419:INFO:main.cc(293)] vcsid 0.0.2-r2590-555a2945fe2a4402ad5e17a108565f643de399d9
[0413/041419:INFO:main.cc(299)] System uptime: 3s
...
[0413/041701:INFO:daemon.cc(530)] Power button down
...
[0413/041702:INFO:daemon.cc(1606)] Shutting down, reason: user-request
...
[0413/041703:INFO:daemon.cc(530)] Power button up

So after being up for ~3min device thinks someone has pressed power button (for ~2sec) to shutdown.  


Now there is something interesting in syslog,

2018-04-13T11:09:55.421159+00:00 INFO p2p-server[5703]: p2p-server starting [../../../../../../../../../mnt/host/source/src/platform2/p2p/server/main.cc:59]
2018-04-13T11:10:15.724315+00:00 WARNING kernel: [  456.835222] perf samples too long (2569 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
2018-04-13T04:14:18.576305-07:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset 

So between 2018-04-13T11:10:15 & 2018-04-13T04:14:18 I can't explain why there's not at least a log line like:

2018-04-13T04:17:03.736133-07:00 NOTICE pre-shutdown[4256]: Shutting down for halt: user-request

Unfortunately there's nothing in logs that I can see that exposes the issue beyond what I'd expect to see in syslog

Perhaps the powerd shutdown we're seeing is the user returning to the hung device and holding/tapping power button while trying to make sense of the unresponsiveness.  Not long enough to trigger cold reboot but in a manner that shuts down before the display can come up.

If thats the case and there's anyway to avoid it we might find something in 'console-ramoops' to help identify what went wrong.

#32: It seems unlikely (but not impossible) that the Chrome was hanging but still able to process power button events and ask powerd to shut the system down.

I think that further diagnosis requires logs to be provided along with information about precisely what was observed just before they were collected. This bug is about devices spontaneously rebooting, but the logs from #30 show a device shutting down (not rebooting) in response to the power button being held for two seconds.

Latest logs we have
https://drive.google.com/open?id=1FcbMnD8xXnnP0I-gE9M7V3JDL5afo-pW

and feedback 
https://listnr.corp.google.com/product/208/report/85309333821?dateRange=All

I especially asked, if it's rebooting or somehow else, so 
Yes, its just rebooting. It drops out of the meeting, screen turns black and
then you will see the kiosk mode startup after a few seconds.

I asked to disconnect all peripherals (to avoid accidental reboot caused by long-press of "Hang UP" button on remote or Jabra) and disable automated reboots, but it still happening.

And it never happens on wired connection.

Latest logs we have
https://drive.google.com/open?id=1FcbMnD8xXnnP0I-gE9M7V3JDL5afo-pW

and feedback 
https://listnr.corp.google.com/product/208/report/85309333821?dateRange=All

I especially asked, if it's rebooting or somehow else, so 
Yes, its just rebooting. It drops out of the meeting, screen turns black and
then you will see the kiosk mode startup after a few seconds.

I asked to disconnect all peripherals (to avoid accidental reboot caused by long-press of "Hang UP" button on remote or Jabra) and disable automated reboots, but it still happening.

And it never happens on wired connection.

#c34/c35: Issue description does sounds like a Chrome crash.

However that feedback and log is from April, and is not even from Cfm. It's an anonymous feedback from ChromeOS. Can you confirm you attached the right link?

What are the next steps here? 
Igor: did you follow-up with the customers? what was their update?

Igor, can we get this bug into a good state where it can be handed to eng (or closed if this is cleaned up) ?

Per comment #25 "put back to Stable (+ USB recovery to M64 unfortunately, since changing channels doesn't downgrade from ChromeOS M65)" -- Don't forget the now (seemingly) forgotten M65 EOL devices have also started to repeatedly hang, freeze, and/or reboot -- (here) even in GUEST mode.  Don't know about M64 though, since don't know when M65 upgrade happened.

https://bugs.chromium.org/p/chromium/issues/detail?id=821607

#40: I don't see anything suggesting that the issue that you linked to has anything to do with this one. Please file a new issue instead of commenting on closed or unrelated ones.

@41: "I don't see anything suggesting that the issue that you linked to has anything to do with this one. Please file a new issue instead of commenting on closed or unrelated ones."

#33: "This bug is about devices spontaneously rebooting"

 Bug #821607  is about devices hanging, especially M65 devices as title says, but as other comments show (#114, my #167, maybe #6, #7, #20), some devices (also?) reboot (the 2 here after the hang) -- and despite #116 and #161 there was no separate bug found for such yet, though my comments still belong there and may be useful here as these both hang AND reboot, and importantly, even in GUEST mode, sometimes with just ONE website opened right after a previous reboot.