New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 870265 link

Starred by 6 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Moving a device from dev/beta to beta/stable in enterprise may delay critical updates for up to 12 weeks

Project Member Reported by tnagel@chromium.org, Aug 2

Issue description

Chrome Version: 69
OS: Chrome

What steps will reproduce the problem?
(1) Enroll device for enterprise management
(2) Move to dev/beta channel --> get updated to version aa.bb.cc.dd
(3) Move back to beta/stable channel ("target channel")

What is the expected result?
The device should pin to the branch (i.e. aa.bb.cc.*) until the target channel has overtaken it, and then release the pin. That way, the device still gets fixes and improvements while it's waiting for the target channel to catch up.

What happens instead?
The device is stuck on version aa.bb.cc.dd until the target channel has caught up, at which point it updates to the latest release on the target channel. This is not great because it may mean that a device doesn't receive updates for up to ~12 weeks (dev --> stable) or up to ~6 weeks (dev --> beta, beta --> stable). This may cause the device to miss high-risk security fixes which are pushed out across dev/beta/stable during that time.
 
Labels: Enterprise-Triaged
Owner: hunyadym@chromium.org
Assigning to Marton for triage
Cc: moisesos...@chromium.org
I think we have 2 options here:

1) Roll back the device immediately to the target channel, even if it's an enterprise user (like it happens for non-enrolled users). Enterprise rollback will support this case (without losing management/policy), we just have to enable user-initiated rollbacks too.

2) We do server side changes, so Omaha serves images from different channels too.
I think the recommended pinning to branch is not ideal (it won't work for dev, since we have always a new branch number for every build).
I think it's quite hard to determine the correct image in this case, probably we could add rules like "if the newest image on the current channel (stable/beta) is older than the currently installed image, then install the latest image from any channel which has the same Chrome version as the current image has", but this is complicated, because the device only sends up the Chrome OS version, not the Chrome version. CC'ing Moises for his opinion on this change.


1) I think it would be nice to let users choose: "Do you want to change channels immediately and powerwash, or do prefer to wait until the release channel catches up." -- I think such choice would be nice for the consumer case as well and might increase the size of our beta/dev populations. (It has always deterred me from running dev that I couldn't go back to stable without powerwashing. ARC++ and Crostini make that even worse.)

2) My proposal of pinning aa.bb.cc.* is based on the assumption that by going back to stable/beta the user expresses the intent to run less unstable software. If we don't pin the branch, the next dev release may even be less stable. But I guess that's a drawback that we need to accept, because otherwise during a dev --> stable transition the user would miss beta completely. I guess instead of pinning to aa.bb.cc.* I should propose to pin to aa.bb.*.* instead. We should be able to do that mostly by using existing code.
2) Currently pinning is actually to the Chrome OS version (like 10575.64.0) and not the Chrome version (e.g. 67.0.3396.127). So pinning to M67 is actually pinning to 10575.*

While we're in dev, we don't know yet the branch number beta and stable will have.  According to your idea, if you're on dev 10539.0.0 and go back to stable, the next you should update do is 10575.22.0 (this was the first beta of 67) - I don't really see how could we figure this out client-side.

Confirming some stuff Omaha-wise:
  a) You're correct, seems we don't send the Chrome version to Omaha, so checking for the client to have certain Chrome version is a no-go (unless we add this, of course).
  b) Yes, we cannot pin the branch, just the "milestone" represented by 10575.*. This is already how enterprise pinning works, it's just users don't have this option. This is also friendly with rollbacks, as it was designed.

IMO I think it's simpler and better to ask the user if they want to take the latest stable as part the channel switch dialog and just send up "_rollback=true" to Omaha as a way to "force" the rollback. Special care has to be taken in case Omaha returns noupdate or something happens making the update not happening.

I also don't think we want to rollback all users in version Y when latest is X, unless they actually want. This because we may want to stop rolling out a version to new users but not to roll them back. Although I guess this is more of a TPM question :)
Cc: zentaro@chromium.org
giving users an option doesn't really improve things.  it just gives us weak cover to say "the user opted-in to a bad security situation so it's their fault".

isn't zentaro@ working on this for the rollback save/restore stuff ?
Marton, Moises, thanks for the correction. I agree that implementing a proper update path from dev to stable isn't as simple as I thought (but it doesn't look impossible either). Yet I still believe that at least the transition from beta to stable would be very simple to implement in a way that keeps the updates flowing: Just pin to the platform branch (10575.* in Marton's example).
Cc: -zentaro@chromium.org hunyadym@chromium.org
Owner: zentaro@chromium.org
Status: Assigned (was: Untriaged)
This issue has an owner, a component and a priority, but is still listed as untriaged or unconfirmed. By definition, this bug is triaged. Changing status to "assigned". Please reach out to me if you disagree with how I've done this.

Comment 10 by zentaro@chromium.org, Jan 17 (5 days ago)

Cc: -hunyadym@chromium.org baileyberro@chromium.org
Labels: ent-rollback

Sign in to add a comment