@dgreid, is there someone who can look at this?

I tried to reproduce this...and if the Chromebook is only sleeping for a few minutes, then the clocks are in sync. I'll try overnight and see if that makes a difference.  

After sleeping the Chromebook overnight then the clocks were off by a little over 1 second.  The time doesn't appear to be 'way off' like the original report...unless they were thinking it was way off because of the difference due to the time zone and they didn't notice that part.

Seeing as this isn't being reproduced, I'm closing this. Please re-open and comment if this happens again.

I'm using a c101pa (bob) chromebook, and this issue is extremely pronounced. time zone drift can be hours or even days if I don't use my chromebook, more than enough to cause TLS errors.

I can fix the error by running `vmc stop termina` followed by `vmc start termina`. 

ChromeOS version: `Version 69.0.3497.73 (Official Build) beta (32-bit)`

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.5 (stretch)
Release:        9.5
Codename:       stretch

Here's a screenshot of the drift. This happened when I left my laptop asleep overnight.

Deleted: Screenshot 2018-09-26 at 09.25.54.png 13.7 KB

	Screenshot 2018-09-26 at 09.25.54.png 13.7 KB View Download

Reopening because there's recent feedback.

Thanks for the report; this is very plausibly related to  https://crbug.com/878908  and gives me a pretty strong hunch about what the issue is. I want to investigate a bit more before de-dupeing, though.

I discussed offline with the reporter. This bug is happening on an ARM Chromebook, so while this may still be related (and may even have a near-identical fix if my current theory is correct), it's not the exact same bug as  https://crbug.com/878908 .

Has anyone who already looked at this bug reproduced it on an x86 chromebook?

(sorry -- I didn't discuss with the original reporter, but the person who posted comment#9.)

I have just successfully reproduced this on an ARM chromebook (Kevin), with a bit of patience waiting for the clock drift to be noticeable.

Changing TZ and running date made no difference in whether it reproduced, but the time reported in the terminal is different from the system bar time.

Next I'll update the guest kernel on the device to 4.19 and see if getting those patches fixes it.

It does not seem like the arm kernel config enables CONFIG_KVM_GUEST, which is necessary for proper timekeeping. I'm turning this on and testing with it now.

CONFIG_KVM_GUEST is not available on ARM. I'm not sure what the intended timekeeping method is for ARM VMs. On x86 (and other architectures where it's available), CONFIG_KVM_GUEST enables paravirtualization for the clock (kvm-clock) to allow the host to communicate time changes (e.g. after sleep/suspend) to the guest.

Per a thread on the kvmarm mailing list (https://lists.cs.columbia.edu/pipermail/kvmarm/2018-October/032863.html), paravirt support for time does not exist on ARM (yet).

I'm not sure what the timeline for support is, but as I see it in the meantime we have a couple of options:

1) Run an ntp daemon in the container. This would adjust time to be correct but wouldn't provide an accurate high-resolution timer or clocksource, as far as I know
2) Modify garcon or some other crostini component to provide access to the host clock while we're waiting for ARM KVM to support it. Depending on how we do this, it could allow for quicker updates after returning from resume. It'd certainly be more work than just enabling an ntp daemon, but I believe less than proper paravirtualization support in KVM.

Any thoughts on these options or which one we should prefer?

we're in the process of plumbing through timezone changes (issue 829934), so it seems to me that extending that to include a "set clock" operation would be easiest ?  it should be easy for daemons running on the host to listen to resume events (i believe they're already broadcast over dbus) in which case it should be easy to have garcon react and send a "set clock" message to all active VMs.

we're already running tlsdated (and put a lot of thought into the security/runtime of it) on the host, so would strongly prefer not replicating that in every container.

i wouldn't go beyond this to trying to provide our own hrtimer or such.  at this point, we should just let the ecosystem do the work for us :).  we've got enough crostini-specific issues to sort out as it is ...

Sounds mostly-reasonable, but I am a bit concerned that the clock might gradually drift if the system is up for a while and the host needs to steal time from the guest. Can we have it send a "set clock" message on resume as well as every N minutes for some appropriate value of N that's small enough to be useful and large enough not to impact performance?

i don't think having garcon set a ~1 hour timer or something would be a big deal.

we can always implement the resume/sync behavior and just wait re-evaluate or wait for reports to say we should expend more effort.

Reopening; this is sufficiently different from issue 829934 to not be tracked as the same issue or even to share much code.

Current plan:

Modify concierge and maitre'd to communicate current time from host to guest. It'll send the time once an hour and on resume from sleep.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/8b652944d0ec55235942fe7b374a057423477fe2

commit 8b652944d0ec55235942fe7b374a057423477fe2
Author: Miriam Zimmerman <mutexlox@google.com>
Date: Wed Nov 07 19:40:56 2018

vm_tools: Add functionality to update guest time.

For now this can only be invoked manually, with `concierge_client --set_time`.
This call will update the time in all VMs to the current host time.
Invoking this code automatically on resume (and potentially periodically) is
deferred to a future CL.

This RPC is necessary because Linux on ARM does not have paravirtualized
time, so (especially on resume) there can be significant clock drift as
compared to the host.  We work around this on ARM by triggering guest
clock update on resume, using this RPC.

BUG= chromium:823406 
TEST=Unit tests (in this CL); manually ran concierge_client call and
verified that guest time was updated.  In a future CL I will add tests
in src/platform/tast-tests/src/chromiumos/tast/local/bundles/cros/vm/subtest/.

Change-Id: Ieb46889ad0bad975c4a360af12e45097f92b5ab8
Reviewed-on: https://chromium-review.googlesource.com/1286087
Commit-Ready: Miriam Zimmerman <mutexlox@chromium.org>
Tested-by: Miriam Zimmerman <mutexlox@chromium.org>
Reviewed-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>

[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/concierge/virtual_machine_test.cc
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/proto/vm_guest.proto
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/system_api/dbus/vm_concierge/service.proto
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/concierge/client.cc
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/maitred/service_impl.cc
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/maitred/service_impl.h
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/system_api/dbus/vm_concierge/dbus-constants.h
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/concierge/service.cc
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/concierge/service.h
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/concierge/virtual_machine.cc
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/concierge/virtual_machine.h
[modify] https://crrev.com/8b652944d0ec55235942fe7b374a057423477fe2/vm_tools/maitred/service_impl_test.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/92c37364abe9572b27d60baae3cd2b78602e23f0

commit 92c37364abe9572b27d60baae3cd2b78602e23f0
Author: Miriam Zimmerman <mutexlox@google.com>
Date: Thu Nov 15 10:16:59 2018

vm_tools: Allow "maitred_client" to set time.

BUG= chromium:823406 
TEST=Loaded change onto a kevin chromebook, set time to a time in the
past, and re-set it to time now.

Change-Id: I50c4110feca26999961c5e4ed9bac1d3f620e340
Reviewed-on: https://chromium-review.googlesource.com/1327661
Commit-Ready: Miriam Zimmerman <mutexlox@chromium.org>
Tested-by: Miriam Zimmerman <mutexlox@chromium.org>
Reviewed-by: Jeffrey Kardatzke <jkardatzke@google.com>

[modify] https://crrev.com/92c37364abe9572b27d60baae3cd2b78602e23f0/vm_tools/maitred/client.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/cd8f9a56849b967e65b665270f7bd198be303b40

commit cd8f9a56849b967e65b665270f7bd198be303b40
Author: Miriam Zimmerman <mutexlox@google.com>
Date: Tue Nov 20 13:52:35 2018

tast-test: Add test for SyncTime RPC call.

BUG= chromium:823406 
TEST=ran `tast -verbose run 100.127.92.23 vm.CrostiniStartEverything`; only failure is existing.
CQ-DEPEND=CL:1327661

Change-Id: I5cbf06947fa4ec8da8b12e1ea181681e1bce6f94
Reviewed-on: https://chromium-review.googlesource.com/1330848
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Miriam Zimmerman <mutexlox@chromium.org>
Reviewed-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/cd8f9a56849b967e65b665270f7bd198be303b40/src/chromiumos/tast/local/vm/vm.go
[add] https://crrev.com/cd8f9a56849b967e65b665270f7bd198be303b40/src/chromiumos/tast/local/bundles/cros/vm/subtest/set_time.go
[modify] https://crrev.com/cd8f9a56849b967e65b665270f7bd198be303b40/src/chromiumos/tast/local/bundles/cros/vm/crostini_start_everything.go
[modify] https://crrev.com/cd8f9a56849b967e65b665270f7bd198be303b40/src/chromiumos/tast/local/vm/concierge.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/d8fcd5f562fb14aeaef9133ed267092c022ee855

commit d8fcd5f562fb14aeaef9133ed267092c022ee855
Author: Miriam Zimmerman <mutexlox@google.com>
Date: Wed Nov 28 03:14:08 2018

vm_tools: Call SyncTimes method on resume on ARM.

BUG= chromium:823406 
TEST=Loaded code onto test kevin device and verified that time was correctly updated after closing and opening laptop.

Change-Id: I68d68472a47be3668caf799f0ae4b601ac88209a
Reviewed-on: https://chromium-review.googlesource.com/1345314
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Miriam Zimmerman <mutexlox@chromium.org>
Reviewed-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/d8fcd5f562fb14aeaef9133ed267092c022ee855/vm_tools/concierge/service.h
[modify] https://crrev.com/d8fcd5f562fb14aeaef9133ed267092c022ee855/vm_tools/concierge/service.cc

Fix is merged and should be in M72 (which is scheduled for beta on Dec 13 and stable on Feb 5),