dist-upgrades are riskier than normal upgrades, and if users have their own apt configs (particularly if they add repos for newer Debian versions) we could easily break them.

I think running an `apt-get install cros-guest-tools` would be enough to upgrade the guest tools + install any new dependencies. That's less risky and we could either do it automatically, or provide a UI to let the user do so.

IMO this should go through PackageKit. jkardatzke@ FYI.

Totally agree this should be done by my new PackageKit interface. :)

Any opinions on what frequency I should do this with? Maybe on every startup of the container (after a short delay, maybe 5 minutes) and then also daily as well?  This would need to do a 'RefreshCache' (i.e. apt-get update) before it tries this to even determine if an update is available so it's not a lightweight operation unfortunately.  I'd also rather not introduce any kind of statefulness into garcon where it needs to remember the last time it did this across invocations.

After container startup and daily SGTM. We might want an opt-out mechanism as well, since automatic upgrades of any kind on Arch are a scary proposition.

Can PackageKit limit the upgrade to cros-guest-tools + installing any dependencies?

I can definitely limit the upgrade to cros-guest-tools, that's what I was planning on doing (by simply invoking an install operation for it, which I'm assuming will do an upgrade just like apt-get does...there's no singular package 'upgrade' functionality in PackageKit). The 'update' will be for all the repos (I didn't see a way to make it for just ours).

When this is done the documentation here should also be updated:

https://chromium.googlesource.com/chromiumos/docs/+/master/containers_and_vms.md#Do-I-have-to-manage-container-updates

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/containers/cros-container-guest-tools/+/4f26c8d3deebbef9424a425bc5fcaab7c7a2aabc

commit 4f26c8d3deebbef9424a425bc5fcaab7c7a2aabc
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Wed Jul 11 23:29:36 2018

Added polkit permission for refreshing the package cache

This is needed for doing automatic upgrades of the packages that we
distribute ourselves.

Bug:  852453 
Test: Manually verified has needed permissions on eve
Change-Id: I2464a4fe60be6bfef2b75fd05c751464b4ff3e8a
Reviewed-on: https://chromium-review.googlesource.com/1134190
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Trybot-Ready: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Commit-Queue: Jeffrey Kardatzke <jkardatzke@google.com>

[modify] https://crrev.com/4f26c8d3deebbef9424a425bc5fcaab7c7a2aabc/cros-garcon/garcon-packagekit.pkla
[modify] https://crrev.com/4f26c8d3deebbef9424a425bc5fcaab7c7a2aabc/cros-garcon/BUILD

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/6bf601a9ebf7b4d895da80554a31c4df948afecf

commit 6bf601a9ebf7b4d895da80554a31c4df948afecf
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Thu Jul 12 09:33:24 2018

vm_tools: garcon: Automatic upgrades of managed packages

This adds functionality to garcon where it will periodically perform a
refresh on the package repository caches (5 min after startup and then
daily), then look for any upgradable packages that are in the google
controlled repository, and then finally apply any of those upgrades to
those packages.  If new dependencies are needed, that will automatically
be handled by PackageKit; but dependencies will not be upgraded
automatically (which is how normal package updates work vs.
dist-updates).

BUG= chromium:852453 
TEST=Manually verified updating works on eve
Change-Id: I5d543fd86fd023ca85f6b9db58fd6cd305a1ccfb
Reviewed-on: https://chromium-review.googlesource.com/1134390
Commit-Ready: Jeffrey Kardatzke <jkardatzke@google.com>
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[modify] https://crrev.com/6bf601a9ebf7b4d895da80554a31c4df948afecf/vm_tools/garcon/package_kit_proxy.h
[modify] https://crrev.com/6bf601a9ebf7b4d895da80554a31c4df948afecf/vm_tools/garcon/package_kit_proxy.cc

The way we ended up doing this is through the following set of steps, which we run 5 minutes after startup and then daily after that:

1. Do a RefreshCache in PackageKit, which is essentially the same as 'apt-get update'
2. Do a GetUpdates call in PackageKit, which will find all of the installed packages that have updates available. Then we filter that list by ones that are in the 'google-stable-main' repository, which corresponds to all of our created packages.
3. If any updates exist, install all of them using InstallUpdates in PackageKit. This will install the updates for all of our packages as well as install any dependencies that do not exist. It will NOT upgrade dependencies automatically, only install nonexistent ones.

Overall this seemed like the safest approach since actually running dist-upgrade could potentially break the users container if something went wrong (especially because we are doing this in the background w/out user interaction). This will ensure our packages are kept up to date; but we do of course still need to maintain backwards compatibility from the Chrome OS host side because this is a 3rd thing that will not be synchronously updated (Chrome OS, Termina and Cros Debian packages).

Thanks Jeff, sounds like a good plan. Can you update the containers_and_vms doc to note this once it all lands? And tell users they should update once in a while if they care to.

is there an apt-get target for installing security updates only ?

i understand not wanting to do a dist-upgrade all the time, but seems like we should at least have it roll security updates ...

When I query for the available updates for all packages (in order to get the ones for our packages) there is a code attached to each of the updates which corresponds to this enum (these values cover other cases as well, but 3-9 should be the relevant ones):

https://www.freedesktop.org/software/PackageKit/gtk-doc/PackageKit-Enumerations.html#PkInfoEnum

When I tested this, our packages were labelled as NORMAL. But then there were also updates for libcups as well which were labelled as SECURITY.

It would be trivial for me to change the filter to also allow through any packages that have the SECURITY flag set.

Do we want to do that?  I'm not familiar enough with how Linux packages are tagged to know if that's a reliable mechanism or not.

We do this now via unattended-upgrades, but I'm all for a solution that doesn't require installing a boatload of dependencies.

Sorry Jeff, I posted when you changed the summary.

I just noticed the /etc/apt/apt.conf.d/95cros-unattended-upgrades setting for apt which *looks* like it's supposed to do what I just implemented...but as we know it clearly wasn't doing that because people had out of date packages unless they ran dist-upgrade.  Do we know if the existing unattended-upgrades package we have installed is actually doing what it's supposed to do for security updates?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/docs/+/ee327f60f6bc03b6369cbc2107de9437f97471a2

commit ee327f60f6bc03b6369cbc2107de9437f97471a2
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Thu Jul 12 23:55:43 2018

containers_and_vms: Update doc for container package auto updating

BUG= chromium:852453 

Change-Id: I098a7012b744c2c9962ea362d58ba37def3fdc6f
Reviewed-on: https://chromium-review.googlesource.com/1135813
Commit-Ready: Jeffrey Kardatzke <jkardatzke@google.com>
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/ee327f60f6bc03b6369cbc2107de9437f97471a2/containers_and_vms.md

Re-opening with a modification to also apply security updates as well.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/dd1e9f186c8c27d792e2adc016cb9d7120a154f8

commit dd1e9f186c8c27d792e2adc016cb9d7120a154f8
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Wed Sep 05 23:11:12 2018

vm_tools: garcon: Apply security updates for packages automatically

This extends the PackageKit update functionality so that it also applies
any updates for installed packages that are marked as 'security'
updates. A setting is also added to the config file to disable this if
desired.

BUG= chromium:852453 
TEST=Verified on eve that outdated libcups package gets security update

Change-Id: I3363a825388ad2980ac83707131b0f55ee9b246f
Reviewed-on: https://chromium-review.googlesource.com/1199943
Commit-Ready: Jeffrey Kardatzke <jkardatzke@google.com>
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[modify] https://crrev.com/dd1e9f186c8c27d792e2adc016cb9d7120a154f8/vm_tools/garcon/README.md
[modify] https://crrev.com/dd1e9f186c8c27d792e2adc016cb9d7120a154f8/vm_tools/garcon/package_kit_proxy.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/containers/cros-container-guest-tools/+/ed56315cc32c34b19a60a752ef3ff8accfd906c2

commit ed56315cc32c34b19a60a752ef3ff8accfd906c2
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Wed Sep 05 23:11:10 2018

cros-garcon: Add config setting for disabling automatic security updates

This adds a config setting for disabling automatically installing
package updates that are marked as 'security' updates.

BUG= chromium:852453 
TEST=Verified on eve that this disables security updates if set

Change-Id: I656c905a24d350fdd9bfe884f551c45346774c3c
Reviewed-on: https://chromium-review.googlesource.com/1200263
Commit-Ready: Jeffrey Kardatzke <jkardatzke@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[modify] https://crrev.com/ed56315cc32c34b19a60a752ef3ff8accfd906c2/cros-garcon/skel.cros-garcon.conf

Requesting merge of changes in comments #21 & #22.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/d7698d53987cb26805683df8242f5dfd1ed177bc

commit d7698d53987cb26805683df8242f5dfd1ed177bc
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Fri Sep 07 18:07:58 2018

vm_tools: garcon: Apply security updates for packages automatically

This extends the PackageKit update functionality so that it also applies
any updates for installed packages that are marked as 'security'
updates. A setting is also added to the config file to disable this if
desired.

BUG= chromium:852453 
TEST=Verified on eve that outdated libcups package gets security update

Change-Id: I3363a825388ad2980ac83707131b0f55ee9b246f
Reviewed-on: https://chromium-review.googlesource.com/1213902
Commit-Queue: Jeffrey Kardatzke <jkardatzke@google.com>
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[add] https://crrev.com/d7698d53987cb26805683df8242f5dfd1ed177bc/vm_tools/garcon/README.md
[modify] https://crrev.com/d7698d53987cb26805683df8242f5dfd1ed177bc/vm_tools/garcon/package_kit_proxy.cc

I decided to only merge the change in #21. The one in #22 has a dependency on another change not in M70 and only puts a default setting in a config file (which doesn't exist without that other change).

 jkardatzke@ Please share bug verification steps. Thanks!

What *should* work is that if you startup a Crostini container, and then immediately in the Terminal enter:

sudo apt-get update
sudo apt-get dist-upgrade

Then answer 'n' to the dist-upgrade when it asks to do the upgrades and you'll see a few packages listed.  Then wait for over 5 minutes and in 'sudo journalctl' you will see that it did the update via garcon in the background. Then at that point if you do a 'sudo apt-get dist-upgrade' you may still have things that can be upgraded, but the list will be much shorter than it was the first time you ran it.

Verified as describe above on kevin device on M71-11120.0.0. Observed few packages auto updated.