+ Drew, please triage this issue.

+jin/ivan

Ivan, do you see any errors in the log for establishing a GCM channel? In general, it sounds like maybe GCM channel isn't working, so we're getting neither server-side pushes nor heartbeats sent up.

Jin, anyone on the GCM team you think we should loop in on this?

Why is this R-V-G?

[930:930:0425/064816.359648:ERROR:gcm_channel_status_request.cc(127)] GCM channel request failed.


Good guess Drew.

+zea, the channel request uses a sync endpoint
+pozlen FYI

mlight@, could you create a screenshot of your chrome://gcm-internals page?

By default the channel status will be tested through a request to

  [Beta/Stable] https://clients4.google.com/chrome-sync/experimentstatus
  [Canary/Dev] https://clients4.google.com/chrome-sync/dev/experimentstatus

Is looks like the request fails. (Canceled or failed.)

I have the chrome://gcm-internals page as a pdf file (attached).

Deleted: GCM_Internals.pdf 54.7 KB

GCM_Internals.pdf 54.7 KB Download

re #2, I think beverloo@ owns the GCM chrome client (according to fgorski@).

I thought client has logic to re-establish GCM connections if it failed initially? Do we have proof to see that subsequent connection attempts all failed?

+zhanlu@ who now owns the Kiosk stuff on the server-side.

I'm beverloo@.

The heartbeat logic uses the GCM Driver that's scoped to the BrowserProcessImpl, so chrome://gcm-internals/ doesn't display anything of use. That's really not helpful in this case, I filed Issue 715190 to track that. (The attachment in #7 is therefore not relevant.)

zea@ may be aware of changes on the server side that could've caused this.

There haven't been any changes on our side w.r.t. the channel endpoint AFAIK (+yiinho to confirm)

Is it possible to get a chrome://net-internals dump for the BrowserProcessImpl? I wonder if perhaps there's some connection problem or firewall/proxy issue here.

Peter, the gcm-internals dump from #7 should theoretically show a connection though right, even if it's not the same connection used by Cpanel? The fact that it doesn't is suspicious. Perhaps a chrome://net-internals dump from the main profile would be useful too (you might need to configure chrome://net-internals to start at startup to capture any request attempts before backoff becomes too large)

Oops, didn't mean to drop Drew from the cc list.

I think we have the wrong owner here - who can diagnose GCM connection errors? The problems with kiosk heartbeats are just a symptom of a general GCM failure so isandrk@ is the wrong owner.

#10 - No, because there's a separate GCM Driver for system-level features on Chrome OS. chrome://gcm-internals is only populated with the data of the Profile-bound GCM driver, the BrowserProcessImpl::gcm_driver() one is ignored. See Issue 715190, we can probably add a checkbox or something to switch between the two.

#12 - The log message indicates a failure in a request to an endpoint that's periodically checked to see if we should connect to GCM at all. That defaults to Yes so it shouldn't be the only cause, but it's still very suspicious that the request fails.

I do echo what zea@ says: having an export of chrome://net-internals/ would be super helpful in figuring out what's going on.

Mike, can you please get chrome://net-internals/ dump from any of the affected units?

Right, gcm-internals is the profile-bound GCM driver. My point is that the the dump there implies the profile-bound GCM driver isn't connecting, when it definitely should be (this is CrOS after all, so the user has to be syncing).

So if the profile-bound GCM driver isn't connecting, I wonder if both GCM drivers are hitting the same issue.

re: #14  Is there some way to collect net internals in VT2?  While in kiosk mode (necessary for the heartbeat to be sent) I don't have access to a chrome browser.

re: #16: sent config details for capturing netlogs to @mlight

Thank you!  Added some flags to /etc/chrome_dev.conf, rebooted, went back to the System Activity and Troubleshooting page for the device, and...  Green heartbeat.

I'll let it run for an hour and see if it goes gray at any point.

Tried three more times to reproduce the problem, but everything is behaving as it should.  If I ever bump into this again I'll know how to collect the net-internals log.  For now, closing this as "WontFix" (not reproducible).

@Mike, which flags did you add exactly to /etc/chrome_dev.conf?

The flags I added to /etc/chrome_dev.conf were:

--enable-logging
--v=1
--log-net-log=/tmp/netlog
--net-log-level=0