Some more background: keiichiw@ is working on a test that can fail on VM boards (video related). He wants to run it on Chrome CQ by adding it to vm_sanity, but he can't make sure it's safe because Chrome OS builders do not run the test on VM boards.

I'm wondering how to resolve this -- maybe shall we run tast in VMTest of betty?

Thanks nya@ for complementing. Yes, this is related to crbug.com/852302.

Added bpastene@ and achuith@ to CC, who are working on ChromeOS testing in Chrome CQ.

Tast tests run on betty-release; see https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/master/doc/tast_integration.md#VMs. There's a TastVMTest stage.

I'll look into making us run non-informational a.k.a. important tests on the betty paladin builders as well.

Sent https://crrev.com/c/1232554 for review.

The change in #5 is held up since there are a bunch of other VM builders that either don't include Tast tests (e.g. lakitu) or that don't skip unsupported tests because they aren't VM-only boards (e.g. amd64-generic). I'm still trying to come up with a good solution for that.

In https://crrev.com/c/1222254, Achuith says that vm_sanity runs on the CrOS CQ. Just to make sure I understand, that's only on betty-paladin via the login_VMSanity Autotest test in the VMTest stage, right? I don't see anything similar on amd64-generic-paladin. (As mentioned above, I think that unsupported tests would actually be be failing on amd64-generic-paladin.)

So it sounds like I still need to come up with a way for unsupported tests to be skipped on amd64-generic.

login_VMSanity runs on all boards at the moment. 

Even on amd64-generic-paladin? I don't see a VMTest stage in http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8934863604717913648. This makes me anxious if it's the same build that we're using to run vm_sanity.py on the Chrome CQ.

Re the "VM builders that ... don't skip unsupported tests because they aren't VM-only boards" issue described in #6, I'm thinking that a feasible workaround might be to add a -fakeuseflags or similar argument that the TastVMTest stage can use to inject USE flags suitable for making audio, TPM, etc. tests get skipped.

Note to self:  issue 887094  says that VMTest stages (and presumably also TastVMTest stages) should be opt-in per config.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d

commit 0adae1cbfe137d8d76a4546ee250aa6d994a4f4d
Author: Daniel Erat <derat@chromium.org>
Date: Tue Oct 02 20:23:04 2018

tast: Add -extrauseflags to tast and local_test_runner.

Add a -extrauseflags flag to the "tast run" subcommand that
can be used to inject a comma-separated list of additional
USE flags to treat as being present when determining a DUT's
features for the purpose of checking test dependencies.

Also add an identically-named flag to local_test_runner, and
introduce a new command.ListFlag type for splitting flag
values into slices.

This can be used to exercise dependency-checking code, and
it can also be set by VM builders that test non-VM-specific
images like amd64-generic. Right now, we can't run Tast
tests on these builders since tests that depend on non-VM
features like audio hardware and TPMs run and fail. With
this, we can get the same behavior that we already get when
testing dedicated VM images like betty.

BUG= chromium:885016 
TEST=added unit tests; also ran "tast run" with
     -extrauseflags=chromeless_tty and -checktestdeps=always
     and verified that ui.ChromeLogin gets skipped, and that
     local_test_runner behaves similarly with -extrauseflags

Change-Id: I427df7b7b60591f522d7fe294958ff95d5fd05a6
Reviewed-on: https://chromium-review.googlesource.com/1256024
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>

[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/tast/runner/runner_test.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/tast/command/flags_test.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/tast/command/flags.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/cmd/tast/run/local_test.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/cmd/tast/run/deps_test.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/tast/runner/features_test.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/cmd/tast/run/deps.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/cmd/tast/run/config.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/tast/runner/args.go
[modify] https://crrev.com/0adae1cbfe137d8d76a4546ee250aa6d994a4f4d/src/chromiumos/tast/runner/features.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crostestutils/+/346c558eec2a1f29c19750abd58a9b8ec4580517

commit 346c558eec2a1f29c19750abd58a9b8ec4580517
Author: Daniel Erat <derat@chromium.org>
Date: Thu Oct 04 07:37:11 2018

cros_run_tast_vm_test: Pass -extrauseflags=tast_vm.

Make cros_run_tast_vm_test pass -extrauseflags=tast_vm to
the "tast run" command. This will be used to skip tests that
require features that are only provided by real hardware.

BUG= chromium:885016 
TEST=betty-release-tryjob passed -extrauseflags=tast_vm

Change-Id: Ib6a93d4198368abe2e266d2fe3b3276b4f92d1c5
Reviewed-on: https://chromium-review.googlesource.com/1258126
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/346c558eec2a1f29c19750abd58a9b8ec4580517/cros_run_tast_vm_test

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/1ec6b9e43a5304c18312fed4249557dcd00c6848

commit 1ec6b9e43a5304c18312fed4249557dcd00c6848
Author: Daniel Erat <derat@chromium.org>
Date: Wed Oct 10 16:45:01 2018

tast: Use internal_mic and tast_vm USE flags.

Make local_test_runner disable the audio_play, audio_record,
and tpm features when the tast_vm USE flag is set, and also
require the internal_mic USE flag for audio_record.

tast_vm is now injected via -extrauseflags when "tast run"
is executed by VM builders that use chromite's
TastVMTestStage, and internal_mic has been unset on boards
that lack internal microphones.

BUG= chromium:885016 , chromium:892557 
TEST=none

Change-Id: I04c353d3a03f6c94742c764a0b9700419be7c235
Reviewed-on: https://chromium-review.googlesource.com/1272224
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/1ec6b9e43a5304c18312fed4249557dcd00c6848/src/chromiumos/cmd/local_test_runner/main.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/89e4bf93b1a1382183d6665102723f84184908f2

commit 89e4bf93b1a1382183d6665102723f84184908f2
Author: Daniel Erat <derat@chromium.org>
Date: Fri Oct 12 02:14:29 2018

chromeos_config: Run Tast on {amd64-generic,betty}-paladin.

Run "important" Tast tests matched by the expression
"(!disabled && !informational)" in VMs on
amd64-generic-paladin and betty-paladin. This matches the
set of tests that are currently running on HW paladins.

BUG= chromium:885016 
TEST=inspected config_dump.json

Change-Id: I73f1365afa94eda98c4f0836ae09d2a84cdbb576
Reviewed-on: https://chromium-review.googlesource.com/1232554
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/89e4bf93b1a1382183d6665102723f84184908f2/config/chromeos_config.py
[modify] https://crrev.com/89e4bf93b1a1382183d6665102723f84184908f2/config/config_dump.json

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c0f7a92e2e5efc58dff72ae552f7a03f44c323da

commit c0f7a92e2e5efc58dff72ae552f7a03f44c323da
Author: Daniel Erat <derat@chromium.org>
Date: Sat Oct 13 00:28:17 2018

This hasn't stuck yet.  Issue 894820  was filed to track an issue where TastVMTest was apparently modifying chromiumos_test_image.bin while UploadTestArtifacts was trying to mount it. Once TastVMTest is using cros_vm.py ( issue 891928 ), I think that using --copy-on-write may fix this.

Open question for myself: once amd64-generic-paladin and betty-paladin are working, should I also add Tast tests to the following builders?

betty-arc64-paladin
betty-arcnext-paladin
betty-arc64-release

https://crrev.com/c/1292681 enabled TastVMTest on amd64-generic-paladin and betty-paladin and seems to be working (sorry, forgot to update the BUG= line).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5a5616f3bb3bc706fbd30d3b4fc6eee710ea2d7a

commit 5a5616f3bb3bc706fbd30d3b4fc6eee710ea2d7a
Author: Daniel Erat <derat@chromium.org>
Date: Mon Oct 29 22:46:53 2018

chromeos_config: Run Tast tests on betty-arc64 builders.

Run Tast integration tests on betty-arc64-paladin and
betty-arc64-release.

BUG= chromium:885016 
TEST=ran tryjobs

Change-Id: If6d7240fe7d75fe4180460627d16d2f729c93050
Reviewed-on: https://chromium-review.googlesource.com/1302333
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Yusuke Sato <yusukes@chromium.org>
Reviewed-by: Yaakov Shaul <yshaul@google.com>

[modify] https://crrev.com/5a5616f3bb3bc706fbd30d3b4fc6eee710ea2d7a/config/config_dump.json
[modify] https://crrev.com/5a5616f3bb3bc706fbd30d3b4fc6eee710ea2d7a/config/chromeos_config_test.py

I'll let the arcnext conversation happen in the future, as it sounds from email conversations like there's some subtlety about which tests we want to run there.