Everything on the chrome side goes through cros_vm.py (via cros_run_vm_test.py) so this sgtm.

Some similarity to this CL: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1174925

Still WIP: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1261895

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/569d86fd6d96f8b3ea45d48646c48b7e224891d4

commit 569d86fd6d96f8b3ea45d48646c48b7e224891d4
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Wed Oct 10 04:14:25 2018

tast_test_stages: Use cros_run_vm_test

Use cros_run_vm_test instead of cros_run_tast_vm_test.

The command, with results in /var/tmp/results, and test ui.ChromeLogin,
should look something like this:
chromite/bin/cros_sdk -- cros_run_vm_test --board=amd64-generic --results-dir=/var/tmp/results --no-display --host-cmd -- tast -verbose run -build=false -resultsdir=/var/tmp/results -extrauseflags=tast_vm 127.0.0.1:9222 ui.ChromeLogin

BUG= chromium:891928 
TEST=bots

Change-Id: I5fb65c6b0e4a8db632900ea755d2c825506738e2
Reviewed-on: https://chromium-review.googlesource.com/1261895
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/569d86fd6d96f8b3ea45d48646c48b7e224891d4/cbuildbot/stages/tast_test_stages.py
[modify] https://crrev.com/569d86fd6d96f8b3ea45d48646c48b7e224891d4/cbuildbot/stages/tast_test_stages_unittest.py

Achuith, is there any chance that this broke betty-release? I see failures in the TastVMTest stage at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8933030871723947392 and http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8933060696934486784.

From the logs, it looks like it may be a permissions issue on chromiumos_qemu_image.bin:

08:43:35: INFO: Running Tast VM test suite tast_vm_canary ((!disabled))
08:43:35: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/bin/cros_sdk -- cros_run_vm_test '--board=betty' --no-display '--results-dir=/tmp/cbuildbotCatAeh/tast_vm_canary' --host-cmd -- tast -verbose run '-build=false' '-resultsdir=/tmp/cbuildbotCatAeh/tast_vm_canary' '-extrauseflags=tast_vm' 127.0.0.1:9222 '(!disabled)'
08:43:36: INFO: /tmp/cros_vm_9222/kvm.pid does not exist.
08:43:36: DEBUG: Stop VM
08:43:36: INFO: /tmp/cros_vm_9222/kvm.pid does not exist.
08:43:36: DEBUG: Start VM
08:43:36: DEBUG: QEMU path: /mnt/host/source/chroot/usr/bin/qemu-system-x86_64
08:43:36: INFO: RunCommand: /mnt/host/source/chroot/usr/bin/qemu-system-x86_64 --version
08:43:36: INFO: QEMU version 2.6.0
08:43:36: DEBUG: VM image path: /mnt/host/source/src/build/images/betty/latest/chromiumos_qemu_image.bin
08:43:36: INFO: Pid file: /tmp/cros_vm_9222/kvm.pid
08:43:36: INFO: RunCommand: /mnt/host/source/chroot/usr/bin/qemu-system-x86_64 -m 8G -smp 8 -vga virtio -daemonize -usbdevice tablet -pidfile /tmp/cros_vm_9222/kvm.pid -chardev 'pipe,id=control_pipe,path=/tmp/cros_vm_9222/kvm.monitor' -serial file:/tmp/cros_vm_9222/kvm.monitor.serial -mon 'chardev=control_pipe' -cpu SandyBridge,-invpcid,-tsc-deadline,check -device 'virtio-net,netdev=eth0' -netdev 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22' -drive 'file=/mnt/host/source/src/build/images/betty/latest/chromiumos_qemu_image.bin,index=0,media=disk,cache=unsafe,format=raw' -enable-kvm -display none
qemu-system-x86_64: -drive file=/mnt/host/source/src/build/images/betty/latest/chromiumos_qemu_image.bin,index=0,media=disk,cache=unsafe,format=raw: Could not open '/mnt/host/source/src/build/images/betty/latest/chromiumos_qemu_image.bin': Permission denied
cros_run_vm_test: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/bin/cros_run_vm_test", line 169, in <module>
    DoMain()
  File "/mnt/host/source/chromite/bin/cros_run_vm_test", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/scripts/cros_run_vm_test.py", line 418, in main
    return VMTest(ParseCommandLine(argv)).Run()
  File "/mnt/host/source/chromite/scripts/cros_run_vm_test.py", line 70, in Run
    self._StartVM()
  File "/mnt/host/source/chromite/scripts/cros_run_vm_test.py", line 90, in _StartVM
    self._vm.Start()
  File "/mnt/host/source/chromite/scripts/cros_vm.py", line 378, in Start
    self._RunCommand(qemu_args)
  File "/mnt/host/source/chromite/scripts/cros_vm.py", line 113, in _RunCommand
    return cros_build_lib.RunCommand(*args, **kwargs)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /mnt/host/source/chroot/usr/bin/qemu-system-x86_64 -m 8G -smp 8 -vga virtio -daemonize -usbdevice tablet -pidfile /tmp/cros_vm_9222/kvm.pid -chardev 'pipe,id=control_pipe,path=/tmp/cros_vm_9222/kvm.monitor' -serial file:/tmp/cros_vm_9222/kvm.monitor.serial -mon 'chardev=control_pipe' -cpu SandyBridge,-invpcid,-tsc-deadline,check -device 'virtio-net,netdev=eth0' -netdev 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22' -drive 'file=/mnt/host/source/src/build/images/betty/latest/chromiumos_qemu_image.bin,index=0,media=disk,cache=unsafe,format=raw' -enable-kvm -display none
cmd=['/mnt/host/source/chroot/usr/bin/qemu-system-x86_64', '-m', '8G', '-smp', '8', '-vga', 'virtio', '-daemonize', '-usbdevice', 'tablet', '-pidfile', '/tmp/cros_vm_9222/kvm.pid', '-chardev', 'pipe,id=control_pipe,path=/tmp/cros_vm_9222/kvm.monitor', '-serial', 'file:/tmp/cros_vm_9222/kvm.monitor.serial', '-mon', 'chardev=control_pipe', '-cpu', 'SandyBridge,-invpcid,-tsc-deadline,check', '-device', 'virtio-net,netdev=eth0', '-netdev', 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22', '-drive', 'file=/mnt/host/source/src/build/images/betty/latest/chromiumos_qemu_image.bin,index=0,media=disk,cache=unsafe,format=raw', '-enable-kvm', '-display', 'none']
08:43:36: DEBUG: Stop VM
08:43:36: INFO: Killing 264.
08:43:36: INFO: RunCommand: kill -9 264
kill: (264): No such process
08:43:36: INFO: Time elapsed 0:00:00.255519.
08:43:37: ERROR: Tast VM tests failed

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7b5fdb9c2a7f35145dda05caf3bef1de8929c50b

commit 7b5fdb9c2a7f35145dda05caf3bef1de8929c50b
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Thu Oct 11 08:05:36 2018

Revert "tast_test_stages: Use cros_run_vm_test"

This reverts commit 569d86fd6d96f8b3ea45d48646c48b7e224891d4.

Reason for revert: https://bugs.chromium.org/p/chromium/issues/detail?id=891928#c7

Looks like betty-release is broken due to this change

Original change's description:
> tast_test_stages: Use cros_run_vm_test
> 
> Use cros_run_vm_test instead of cros_run_tast_vm_test.
> 
> The command, with results in /var/tmp/results, and test ui.ChromeLogin,
> should look something like this:
> chromite/bin/cros_sdk -- cros_run_vm_test --board=amd64-generic --results-dir=/var/tmp/results --no-display --host-cmd -- tast -verbose run -build=false -resultsdir=/var/tmp/results -extrauseflags=tast_vm 127.0.0.1:9222 ui.ChromeLogin
> 
> BUG= chromium:891928 
> TEST=bots
> 
> Change-Id: I5fb65c6b0e4a8db632900ea755d2c825506738e2
> Reviewed-on: https://chromium-review.googlesource.com/1261895
> Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
> Tested-by: Achuith Bhandarkar <achuith@chromium.org>
> Reviewed-by: Dan Erat <derat@chromium.org>

Bug:  chromium:891928 
Change-Id: Ibee0dfbc9e14e7352341182cddeada4b1e803c71
Reviewed-on: https://chromium-review.googlesource.com/c/1275351
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Trybot-Ready: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/7b5fdb9c2a7f35145dda05caf3bef1de8929c50b/cbuildbot/stages/tast_test_stages.py
[modify] https://crrev.com/7b5fdb9c2a7f35145dda05caf3bef1de8929c50b/cbuildbot/stages/tast_test_stages_unittest.py

Reverted for now: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1275351

Will take a look.

Try again:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1285111

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a9ebac2333ba4295a38e7cc906ad12cf0ac53c2d

commit a9ebac2333ba4295a38e7cc906ad12cf0ac53c2d
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Thu Oct 18 05:43:18 2018

tast_test_stages: Use cros_run_vm_test

Use cros_run_vm_test instead of cros_run_tast_vm_test.

The command, with results in /var/tmp/results, and test ui.ChromeLogin,
should look something like this:
chromite/bin/cros_sdk -- cros_run_vm_test --board=amd64-generic --image-path=/mnt/host/source/src/build/images/betty/latest-cbuildbot/chromiumos_qemu_image.bin --results-dir=/var/tmp/results --no-display --host-cmd -- tast -verbose run -build=false -resultsdir=/var/tmp/results -extrauseflags=tast_vm 127.0.0.1:9222 ui.ChromeLogin

BUG= chromium:891928 
TEST=bots

Change-Id: I65e4919881f81478bcf637dbb93252440c658254
Reviewed-on: https://chromium-review.googlesource.com/1285111
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/a9ebac2333ba4295a38e7cc906ad12cf0ac53c2d/cbuildbot/stages/tast_test_stages.py
[modify] https://crrev.com/a9ebac2333ba4295a38e7cc906ad12cf0ac53c2d/cbuildbot/stages/tast_test_stages_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/785371775dd409fec5048260877191af68feeb59

commit 785371775dd409fec5048260877191af68feeb59
Author: Dan Erat <derat@chromium.org>
Date: Thu Oct 18 17:05:04 2018

Revert "tast_test_stages: Use cros_run_vm_test"

This reverts commit a9ebac2333ba4295a38e7cc906ad12cf0ac53c2d.

Reason for revert: This seems to have broken betty-release again.

Original change's description:
> tast_test_stages: Use cros_run_vm_test
> 
> Use cros_run_vm_test instead of cros_run_tast_vm_test.
> 
> The command, with results in /var/tmp/results, and test ui.ChromeLogin,
> should look something like this:
> chromite/bin/cros_sdk -- cros_run_vm_test --board=amd64-generic --image-path=/mnt/host/source/src/build/images/betty/latest-cbuildbot/chromiumos_qemu_image.bin --results-dir=/var/tmp/results --no-display --host-cmd -- tast -verbose run -build=false -resultsdir=/var/tmp/results -extrauseflags=tast_vm 127.0.0.1:9222 ui.ChromeLogin
> 
> BUG= chromium:891928 
> TEST=bots
> 
> Change-Id: I65e4919881f81478bcf637dbb93252440c658254
> Reviewed-on: https://chromium-review.googlesource.com/1285111
> Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
> Tested-by: Achuith Bhandarkar <achuith@chromium.org>
> Reviewed-by: Dan Erat <derat@chromium.org>

Bug:  chromium:891928 
Change-Id: I946fe4e54b0fb28e245d2eba74b8797eea447b85
Reviewed-on: https://chromium-review.googlesource.com/c/1287812
Tested-by: Dan Erat <derat@chromium.org>
Trybot-Ready: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/785371775dd409fec5048260877191af68feeb59/cbuildbot/stages/tast_test_stages.py
[modify] https://crrev.com/785371775dd409fec5048260877191af68feeb59/cbuildbot/stages/tast_test_stages_unittest.py

Achuith, feel free to reassign this to me if you want me to work through the bugs here and try to reland the change. I'll probably wait until VM tests are running on the CQ, though.

I'm giving this a try with --copy-on-write.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ea294acc019e0354fc0fca98dee2a7bf61d52f82

commit ea294acc019e0354fc0fca98dee2a7bf61d52f82
Author: Daniel Erat <derat@chromium.org>
Date: Mon Oct 22 02:25:23 2018

cbuildbot: Reland "tast_test_stages: Use cros_run_vm_test"

This reverts commit 785371775dd409fec5048260877191af68feeb59
to enable using cros_run_vm_test in the TastVMTest stage. It
passes the --copy-on-write flag to cros_run_vm_test, which
passes it through to cros_vm. This will hopefully avoid
"permission denied" errors when opening
chromiumos_test_image.bin.

BUG= chromium:891928 , chromium:894820 
TEST=ran tryjobs

Change-Id: Id2e01becd58131806410976360dfda9a73a5261d
Reviewed-on: https://chromium-review.googlesource.com/c/1292680
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/ea294acc019e0354fc0fca98dee2a7bf61d52f82/cbuildbot/stages/tast_test_stages.py
[modify] https://crrev.com/ea294acc019e0354fc0fca98dee2a7bf61d52f82/cbuildbot/stages/tast_test_stages_unittest.py

This looks like it finally stuck; see e.g. http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8931972429527974208.

I'll delete cros_run_tast_vm_test now.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/48f342fa276a7ef77b355d120027855a9ae22be4

commit 48f342fa276a7ef77b355d120027855a9ae22be4
Author: Daniel Erat <derat@chromium.org>
Date: Tue Oct 23 00:39:43 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crostestutils/+/0529a98f95f2c2c611c42120a4044380bfd343c1

commit 0529a98f95f2c2c611c42120a4044380bfd343c1
Author: Daniel Erat <derat@chromium.org>
Date: Tue Oct 23 06:48:40 2018

crostestutils: Delete cros_run_tast_vm_test.

Remove the cros_run_tast_vm_test helper script. The
TastVMTest stage uses cros_run_vm_test to execute the tast
command in a VM now.

BUG= chromium:891928 
TEST=none

Change-Id: I76ba571de46579621528943a4ca84f20a832a6d4
Reviewed-on: https://chromium-review.googlesource.com/1294814
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[delete] https://crrev.com/6a11e6b1f3c1b5262bf0c6a3865c95c23f2483e9/cros_run_tast_vm_test

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f4c4ce79c2d86d8899473a303962e6c92af21c7d

commit f4c4ce79c2d86d8899473a303962e6c92af21c7d
Author: Daniel Erat <derat@chromium.org>
Date: Tue Oct 23 06:48:39 2018

cbuildbot: Clean up cros_run_tast_vm_test mention.

Remove a mention of the no-longer-used cros_run_tast_vm_test
script in a comment in TastVMTestStage. Also drop the
cros_run_vm_test command's cleanup timeout from 30 minutes
down to 10 minutes to match VMTestStage.

BUG= chromium:891928 
TEST=none

Change-Id: I85e99240a8f42d5c5623184cc20e1e20ae0d437c
Reviewed-on: https://chromium-review.googlesource.com/1295009
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/f4c4ce79c2d86d8899473a303962e6c92af21c7d/cbuildbot/stages/tast_test_stages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/4d2d94be1c174a8d710098e8c8951491cec27112

commit 4d2d94be1c174a8d710098e8c8951491cec27112
Author: Daniel Erat <derat@chromium.org>
Date: Tue Oct 23 06:48:39 2018

scripts: Remove cros_run_tast_vm_test symlink.

Remove the symlink to the cros_run_tast_vm_test script from
crostestutils. This script is no longer used.

BUG= chromium:891928 
TEST=none

Change-Id: I2ecbdbb44cfd0423d4eb7990d45a36df5094229f
Reviewed-on: https://chromium-review.googlesource.com/1294772
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[delete] https://crrev.com/e2f328c822ac7c81a53a4f2b54a5b7717a2e84b1/bin/cros_run_tast_vm_test