An alpine-based image may work for some tests, but it also doesn't reflect what users end up running. I'm sure we don't need all of the packages included in the Debian image.

Another option to improve test time would be to snapshot the LXD container and reuse the same stateful disk image between tests. The first test would take the hit for the download, but as long as we run all VM tests sequentially then subsequent tests would be fast.

I don't think that helps tighten up the timeouts for individual tests, though.

Thanks for giving this some more thought!

I'd probably recommend aiming for a smaller container for testing (which I assume would also be useful for development, right?). Is the majority of the time spent downloading the container right now? How large is it?

I'm trying to stay away from having one test depend on state that's left behind by another test due to the potential for non-determinism. I think that the Chrome builders go so far as to run tests in a random order to try to discover dependencies and conflicts, although I'm not planning to do that. :-P

Here's a quick stats dump.

upstream Debian stretch image: ~105 MiB compressed
our Debian stretch image: ~200 MiB compressed

eve vm.Webserver runtime: 2m54.097s
23s   test start -> OOBE dismissed
8s    VM boot; stateful & LXD setup
17s   container image download (over 100 Mb/s adapter)
1m43s container image extraction
10s   start container, setup user, wait for garcon

kevin vm.Webserver runtime: 3m19.931s
17s   test start -> OOBE dismissed
32s   VM boot; stateful & LXD setup
7s   container image download (over 1000 Mb/s adapter)
1m50s container image extraction
11s   start container, setup user, wait for garcon

Something I haven't experimented with is publishing a squashfs image for the rootfs. That should improve the extraction time a lot.

+dgreid in case he's interested
+sonnyrao had looked briefly at why the first boot is so slow on ARM.

 that 32 seconds of VM boot + lxd setup on Kevin is an issue, and that is tracked in https://bugs.chromium.org/p/chromium/issues/detail?id=837445

But I haven't prioritized that bug yet -- will try to look at it for M-70

I talked to chirantan today about enabling 9p rootfs. I was thinking of using it to export a RO copy of the rootfs in to the VM so we don't have to custom roll a testing image for each architecture.

CTS test bundles are in size of hundred of MBs and we cache them on lab server to re-use them.

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/cros/tradefed_test.py?l=605

The difference is we don't copy the entire test bundle on the DUTs while we copy the container image on the DUT but that shouldn't take minutes.

Filed a related bug, issue 864877. When using a raw disk image, the basic vm.StartTerminaVM test runs in 1m15s-ish on both eve and kevin. < 40s to download and extract the full container images.

Removing the "< 10 MB" requirement since discussions seem to have steered us towards building larger test containers with all test utilities preinstalled. This will remove the need to install apt dependencies at test runtime.

We can look separately at caching the container images on lab servers.

This is done under the "debian/stretch/test" alias in LXD.

Closing based on test duration on eve: https://stainless.corp.google.com/search?exclude_retried=true&first_date=2018-10-06&master_builder_name=&builder_name_number=&shard=&exclude_acts=true&builder_name=&master_builder_name_number=&owner=&retry=&exclude_cts=true&exclude_non_production=false&hostname=&board=%5Eeve%24&test=%5Etast%5C.vm%5C.&suite=&build=%5ER71%5C-11137%5C.0%5C.0%24&status=GOOD&reason=&waterfall=&exclude_not_run=false&last_date=2018-10-08&exclude_non_release=true&exclude_au=true&model=&view=list

Please reopen if any other verification required. Thanks!