Thank you.

~Dymphe~

I also hope you push out an updated recovery image ASAP. 

Tested on cyan 52.0.2743.120/8350.79.0. Disabling the wifi the crashes stops

Possible dupe of issue 641268

xdai@ sorry to keep hitting you with the weird ones, but can you dig into this?

It should be fixed with the upcoming 53 push, but we still need to understand what went wrong so we can make sure it doesn't happen again.

I have 2 somewhat rambling Docs with all the steps to reproduce several times.

https://docs.google.com/document/d/1ktxnTmSRVXRSm_zGtGBu4FtxCkAoo_oyfHuDnw_GCts/edit?usp=sharing
This one is when it first happened on the Acer 15 on Stable and the Flip on Dev.

To rule out Arc++ and a specific powerwash from Dev or Beta to Stable, I powerwashed the Flip to Stable while the Acer 15 was on Stable also.

https://docs.google.com/document/d/1N1BwJ9ILI6O2HN12p8kBvXekI_l-Y7azJWcjAX7iXDw/edit?usp=sharing

@edoan issue 641268 is different from this bug. Issue 641268 is still reproducible on 53. Also there is another workaround available for this bug which is to reset the sync process from the sync dashboard: https://www.google.com/settings/chrome/sync

Note that resetting sync via the dashboard will also wipe your server data for sync. If you don't have existing client devices that are up to date, this could result in data loss.

Do we have any crash reports?

A user on CBC found another easier workaround. https://goo.gl/PuZaC3

So I did all of that (and I tried it many times), but for some reason it did not work for me. What I ended up doing instead was powerwashing and signing in as a different account to make it admin, switching to beta channel, powerwashing again, and making my normal account admin. And it is working fine! 

Issue 644049 has been merged into this issue.

With the same user account, here are logs of the issue present on 51 and not on 53:

Issue on 51: 
https://drive.google.com/open?id=0B9a1yrkxazgMRl9fWFNKMzhaNmM

No issue on 53:
https://drive.google.com/open?id=0B9a1yrkxazgMY3N1Zkp5ejdPSzQ

Issue 641268 has been merged into this issue.

@abodenha This is not a duplicate.

issue 641268:

- It cannot be solved by resting the Chrome Sync process.
- It can be reproduced in 53.
- It can be reproduced while Chrome Sync is completely disabled.

From the document https://docs.google.com/document/d/1ktxnTmSRVXRSm_zGtGBu4FtxCkAoo_oyfHuDnw_GCts/edit that dymphep@ provided in #7, I found a crash report: https://crash.corp.google.com/browse?stbtiq=3b6cc1a500000000. And from the crash stack, I think I found the CL (yes, it's in M53) https://codereview.chromium.org/2055553004 that introduced the crash when the user switches from the dev channel (M53) back to stable channel (M52).

In short words, the CL above refactored the shelf pin model in M53. It introduced sync-based pin model while in M52 the pin model was based on prefs. So when the user switches from M53 back to M52, the sync item for the shelf item might not exist, which then caused the crash. 

xdai@ that makes sense. Thanks. 

It would make sense that our tests missed this since you'd need to take certain actions on 53 and then shift to a 52 device for the problem to appear.

Let's sync up tomorrow. I want to try to figure out what sort of testing we need to add to prevent something like this from happening again. At a minimum I'm thinking we should have more tests for invalid sync responses. A bad sync server response should not be able to trigger a browser crash.

Given this will be fixed in the M53 rollout, I'm going to call this "fixed".  I've filed bug 645137 to track efforts at preventing this sort of fail in the future.

I did not reset my device or downgrade it, yet am stuck in this state. I've already attempted the 'reset chrome sync' approach with no luck. t/23152977 has more info for Googlers.

Ah, I know how this happened. I am running M53 on my Chromebook Pixel which chrome syncs to my earlier version Chromebox under my desk.

(and the 'solution' for me was resetting chrome sync, then deleting my account from the chromebox and re-adding my account on the chromebox)

Re lizf@: Yes, that explains the crash. Thanks for the workaround which I think might benefit other people who stuck in the same issue. 

I feel like my sync is broken now. :< 

I ran into the issue where my M52 samus would boot loop right after logging in last week. I tried powerwashing the device and re-enrolling, but that seemed to not help at all.

Following the suggestion made in c#20, I can now use my samus. However, none of my settings are syncing down to my samus.

My other Chrome OS devices are on M53 and M54, but my samus had been (and still is) on M52.

Just providing an update, I got my samus to sync again, however now I'm back where I started with the looping issue.

If you have a device on 52 and a device on 53 and try to sync between them the crash loop is possible.

53 is now stable for most devices (not samus, flip, or R11 tho)

The only fixes that I know of.

1. After the reset of sync, restart sync without app syncing. All other sync, like bookmarks, is OK.

2. Move to Beta 53. As soon as you do, everything is synced including apps without issues.

There are users in the forum who are powerwashing/USB recovering their chromebooks.

That takes them back to Version 52. One example
https://productforums.google.com/d/msg/chromebook-central/ytItte7CI1M/1U7xzv9mAwAJ

Marking this as fixed since recovery images have been updated to M53 which should not have this issue. 

Issue 677647 has been merged into this issue.

Are there any workarounds for this bug for the customers who try to use the freshly unboxed device at M-52?
Is it possible to do something on the server side to suppress this issue for those devices?

As for a workaround, the user can use the guest account and navigate to chrome://chrome and click the check for updates button. As I understand it, once you're past M-52, you should be fine.

I just had to go through this with a family member this past holiday.

I'm pushing for a config change to force AU during OOBE if a device is on 52 or prior.  We've historically avoided doing that as it makes OOBE a MUCH more lengthy process but I feel it's warranted for this.

Re #34: Are you talking about a solution for the existing devices, or only about an improvement for the future?

Because probably something has to be done to resolve the current issues.

Note that during December, at least ~165'000 unique clients faced this issue (see [1]). All, AFAIU, were likely newly bought devices (unless something is wrong with auto update, as all crash reports are with <=53.0.2773.3).
Also note that a single day yesterday brought about 8'000 unique clients with this crash (see [2]).
And, I guess, those numbers not include enterprises who disable data reporting by a policy.

[1] https://crash.corp.google.com/dremel_query_ui?q=SELECT%20count(distinct%20clientid)%20FROM%20crash.prod.latest%20WHERE%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BAssert%5D%20syncer%3A%3AOrdinal%3Csyncer%3A%3AStringOrdinalTraits%3E%3A%3AEquals%27%20AND%20product.name%3D%27Chrome_ChromeOS%27%20AND%20STRFTIME_UTC_USEC(reporttime*1000%2C%27%25Y%2F%25m%27)%3D%272016%2F12%27
[2] https://crash.corp.google.com/dremel_query_ui?q=SELECT%20count(distinct%20clientid)%20FROM%20crash.prod.latest%20WHERE%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BAssert%5D%20syncer%3A%3AOrdinal%3Csyncer%3A%3AStringOrdinalTraits%3E%3A%3AEquals%27%20AND%20product.name%3D%27Chrome_ChromeOS%27%20AND%20STRFTIME_UTC_USEC(reporttime*1000%2C%27%25Y%2F%25m%2F%25d%27)%3D%272017%2F01%2F03%27

Existing devices.

The solution is to force an update during initial setup so that when the user signs in the device has already updated to latest stable.

That's something we've resisted in the past since a forced update like that is a crappy first-run experience.

+ Royans as FYI

We DO NOT want to update these devices to latest before OOBE completes because that completely breaks enterprise version pinning. How many reports of these kind of enrollment issues are we getting? I'm not sure it's enough to justify breaking pinning entirely.

Comment #35 claims 165,000 clients reported the crash in December. That's substantial, especially considering the user's first experience will be a crash-loop when it occurs.

What is our cap on how far we support version pinning?

1) Version pinning is allowed for last 3 major versions, and we keep payloads for last 4 (n-3 to n). (Currently: 55,54,53,52).
2) Based on latest statistics, about 8% of devices are currently enrolling with version 52 on a daily basis.
3) Its ok to get these devices to upgrade, but if they have pinning set to 53, it should not go beyond 53.
4) Forcing them to do upgrade before enrollment would move these devices to 55 since the device doesn't know any better at that point. And that would put device in a state the admin didn't expect it to go to.

For enterprises, we can just put out a KI and have customers ask users to upgrade. My personal opinion is that forced actions at this point may be counter productive.

Alternatively, if 52 is completely broken we could potentially discuss the option of forcing devices on M52 to go to M53 (instead of M55) before enrollment.

52 is completely broken if you try to sign into an account that's used elsewhere on 53+.

I like the idea of forcing an update from 52 and prior to 53. That would solve the issue without breaking pinning.  We need input from josafat@ on the feasibility of that.

If you want to forceupdate all pre-53 to 53, thats about roughly about 40% of the devices enrolling on a daily basis (based on DM logs)

This is a good thing from security and stability view point, but we should take into account that it would also increase the onboarding hurdle for customers with smaller network bandwidth.

Yep, it's a tradeoff and far from ideal, but given the severity of the problem seems justified.

Is there any bug tracking the update forcing that was discussed here in comments 34..42?

#CBC-RS/TC-watchlist

We are still getting reports on CBC from users experiencing this issue with their new Chromebooks.

Albert, reopening this to bring up again the question of update forcing (see the comments 34..43).
I don't know how severe this issue looks on the radars of our support line, but looking at the crash statistics, the number of corresponding crash reports doesn't decrease significantly - still 500K reports from 140K different clients in Jan, 2017 (see crbug.com/677647#c9 for the crash statistics link).

Over to josafat@ who is working on pushing the forced update.

josafat@ any eta we can share?

I have forced au rules setup on staging server now ...

These should be live tomorrow morning 

Awesome! Thanks josafat@.  Please close this once it's pushed.

OOBE Forced AU to M53 (for all pre-M52) is live now.. 

A very BIG Thank You from the CBC team!

Clarification question - if a user is on a wireless connection when they set up their device, will the update be forced? Or will they still have to enter the crosh command to enable update over wireless?

I am trying to understand what should be the OOBE so that we can accurately explain to users on CBC and anticipate any remaining problems.

@52 - the auto-update will happen automatically over Wi-Fi. Unless you're talking about setting up a 3G/4G Chromebook with no Wi-Fi available?

I am talking about setting up a conventional Chromebook (not 3G/4G equipped) but using a Verison hotspot or cellphone hotspot.

Based on the number of users that we have to tell about update_over_wireless, these types of connections are fairly common, especially with users who have unlimited data plans. They do not have cable/DSL internet.

I think #c52 meant to say a 'cellular' connection and will they still have to enter the crosh command: 'update_over_cellular enable' ?

Sorry for the confusing typo. Thank you #55 for the correction.

Re questions in 52-56: Short answer is "I don't know for sure" 

I'm testing now.

Or I will when I can get my hands on an M52 image...

Just confirmed that update will NOT happen while connected over cellular.  Best bet there is to go into guest mode and update from there.