If the Python code is running directly within the VM, then yes, you should execute local_test_runner. I'd recommend using:

  local_test_runner ui.ChromeLogin

if you just want to verify that login works -- using a wildcard like 'ui.Chrome*' will also match some experimental tests.

If the Python code is running outside of the VM, you could use the "tast run" command, which will SSH into it to run tests and do additional nice things like collecting crash dumps and system logs. That would require that the "tast" command is installed outside of the VM, though, which won't be the case if you're not running in a Chrome OS chroot.

https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/running_tests.md has more information about all of this.

We have a way to run test commands on the host, but doesn't look like it's going to work. We don't have a ChromeOS chroot available, but we have access to chromite and the simple chrome shell, which doesn't seem sufficient. 

Maybe there's a way to move things around to make the tast binary available in the simple chrome shell?

For now, it seems most productive to focus on local_test_runner. I think Ben's run_vm_test script already fetches logs. Not sure about crash dumps.

Sure, we can start with ui.ChromeLogin, but I imagine we'll want to have something similar to a smoke suite that we can invoke with a single command.

Yeah, I ran into ui.ChromeLoginForever which doesn't seem like something we want. 

If you want to run a set of tests that's similar to what the Chrome PFQ runs, you can use this expression instead of ui.ChromeLogin:

  '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))'

(That's adapted from server/site_tests/tast/control.mustpass-chrome in the Autotest repo.)

Dan, this fails on amd64-generic:

It looks like it's assuming ARC is available; I think it needs to examine the lsb-release file:
https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/common_lib/lsbrelease_utils.py?l=197-198


DEBUG:root:Running 'local_test_runner '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))''
DEBUG:root:'local_test_runner \'(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))\''
DEBUG:root:[stdout] 2018/08/22 16:38:12 Running arc.Boot
DEBUG:root:[stdout] 2018/08/22 16:38:12 Restarting ui job
DEBUG:root:[stdout] 2018/08/22 16:38:13 Waiting for org.chromium.SessionManager D-Bus service
DEBUG:root:[stdout] 2018/08/22 16:38:13 Asking session_manager to enable Chrome testing
DEBUG:root:[stdout] 2018/08/22 16:38:13 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
DEBUG:root:[stdout] 2018/08/22 16:38:13 Removing cryptohome for testuser@gmail.com
DEBUG:root:[stdout] 2018/08/22 16:38:13 Finding OOBE DevTools target
DEBUG:root:[stdout] 2018/08/22 16:38:13 Connecting to Chrome at ws://127.0.0.1:36761/devtools/page/380277978C49EB10EC040C792DB0963A
DEBUG:root:[stdout] 2018/08/22 16:38:13 Waiting for OOBE
DEBUG:root:[stdout] 2018/08/22 16:38:16 Logging in as user "testuser@gmail.com"
DEBUG:root:[stdout] 2018/08/22 16:38:16 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
DEBUG:root:[stdout] 2018/08/22 16:38:23 Waiting for OOBE to be dismissed
DEBUG:root:[stdout] 2018/08/22 16:38:25 Waiting for Android boot
DEBUG:root:[stdout] 2018/08/22 16:39:55 Error: [/build/amd64-generic/tmp/portage/chromeos-base/tast-local-tests-cros-0.0.1-r140/work/tast-local-tests-cros-0.0.1/src/chromiumos/tast/local/bundles/cros/arc/boot.go:39] Failed to start ARC: service.adb.tcp.port not set: context deadline exceeded
DEBUG:root:[stdout] 2018/08/22 16:39:55 Finished arc.Boot
DEBUG:root:[stdout] 2018/08/22 16:39:55 Running ui.ChromeCrashLoggedIn
DEBUG:root:[stdout] 2018/08/22 16:39:55 Restarting ui job
DEBUG:root:[stdout] 2018/08/22 16:39:56 Waiting for org.chromium.SessionManager D-Bus service
DEBUG:root:[stdout] 2018/08/22 16:39:56 Asking session_manager to enable Chrome testing
DEBUG:root:[stdout] 2018/08/22 16:39:56 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
DEBUG:root:[stdout] 2018/08/22 16:39:56 Removing cryptohome for testuser@gmail.com
DEBUG:root:[stdout] 2018/08/22 16:39:56 Finding OOBE DevTools target
DEBUG:root:[stdout] 2018/08/22 16:39:56 Connecting to Chrome at ws://127.0.0.1:40447/devtools/page/53AB851F5DA8F9A76F2D71DBDB13933C
DEBUG:root:[stdout] 2018/08/22 16:39:56 Waiting for OOBE
DEBUG:root:[stdout] 2018/08/22 16:39:59 Logging in as user "testuser@gmail.com"
DEBUG:root:[stdout] 2018/08/22 16:39:59 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
DEBUG:root:[stdout] 2018/08/22 16:40:06 Waiting for OOBE to be dismissed
DEBUG:root:[stdout] 2018/08/22 16:40:08 Sending SIGSEGV to root Chrome process 14423
DEBUG:root:[stdout] 2018/08/22 16:40:08 Waiting for 19 Chrome process(es) to exit
DEBUG:root:[stdout] 2018/08/22 16:40:08 All Chrome processes exited
DEBUG:root:[stdout] 2018/08/22 16:40:08 Deleting (expected) new minidump: /home/chronos/crash/f603b8ec-40b5-4d8e-6d0536a8-c93ca8dd.dmp
DEBUG:root:[stdout] 2018/08/22 16:40:08 Finished ui.ChromeCrashLoggedIn
DEBUG:root:[stdout] 2018/08/22 16:40:08 Running ui.ChromeCrashNotLoggedIn
DEBUG:root:[stdout] 2018/08/22 16:40:08 Restarting ui job
DEBUG:root:[stdout] 2018/08/22 16:40:09 Waiting for org.chromium.SessionManager D-Bus service
DEBUG:root:[stdout] 2018/08/22 16:40:09 Asking session_manager to enable Chrome testing
DEBUG:root:[stdout] 2018/08/22 16:40:09 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
DEBUG:root:[stdout] 2018/08/22 16:40:09 Removing cryptohome for testuser@gmail.com
DEBUG:root:[stdout] 2018/08/22 16:40:09 Sending SIGSEGV to root Chrome process 14917
DEBUG:root:[stdout] 2018/08/22 16:40:09 Waiting for 5 Chrome process(es) to exit
DEBUG:root:[stdout] 2018/08/22 16:40:09 All Chrome processes exited
DEBUG:root:[stdout] 2018/08/22 16:40:09 Deleting (expected) new minidump: /home/chronos/crash/009ef02d-6d2d-482b-492d2b8d-44257732.dmp
DEBUG:root:[stdout] 2018/08/22 16:40:09 Finished ui.ChromeCrashNotLoggedIn
DEBUG:root:[stdout] 2018/08/22 16:40:09 Running ui.ChromeLogin
DEBUG:root:[stdout] 2018/08/22 16:40:09 Restarting ui job
DEBUG:root:[stdout] 2018/08/22 16:40:10 Waiting for org.chromium.SessionManager D-Bus service
DEBUG:root:[stdout] 2018/08/22 16:40:10 Asking session_manager to enable Chrome testing
DEBUG:root:[stdout] 2018/08/22 16:40:10 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
DEBUG:root:[stdout] 2018/08/22 16:40:10 Removing cryptohome for testuser@gmail.com
DEBUG:root:[stdout] 2018/08/22 16:40:10 Finding OOBE DevTools target
DEBUG:root:[stdout] 2018/08/22 16:40:10 Connecting to Chrome at ws://127.0.0.1:45869/devtools/page/B7CED710C89453E25F12722A497AC8B2
DEBUG:root:[stdout] 2018/08/22 16:40:10 Waiting for OOBE
DEBUG:root:[stdout] 2018/08/22 16:40:13 Logging in as user "testuser@gmail.com"
DEBUG:root:[stdout] 2018/08/22 16:40:13 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
DEBUG:root:[stdout] 2018/08/22 16:40:21 Waiting for OOBE to be dismissed
DEBUG:root:[stdout] 2018/08/22 16:40:22 Waiting for SessionStateChanged "started" D-Bus signal from session_manager
DEBUG:root:[stdout] 2018/08/22 16:40:22 Got SessionStateChanged signal
DEBUG:root:[stdout] 2018/08/22 16:40:22 Creating new blank page
DEBUG:root:[stdout] 2018/08/22 16:40:22 Connecting to Chrome at ws://127.0.0.1:45869/devtools/page/7A7EB4EE5A1232B4E02034B2AF7E1C5D
DEBUG:root:[stdout] 2018/08/22 16:40:23 Navigating to http://127.0.0.1:33292
DEBUG:root:[stdout] 2018/08/22 16:40:23 Got content "Hooray, it worked!"
DEBUG:root:[stdout] 2018/08/22 16:40:23 Finished ui.ChromeLogin
ERROR:root:[stderr] test(s) failed
INFO:root:Try printing formatted exception: <class 'autotest_lib.client.common_lib.error.CmdError'> Command <local_test_runner '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))'> failed, rc=6, Command returned non-zero exit status
* Command: 
    local_test_runner '(!informational && !disabled && ("dep:chrome" ||
    "dep:chrome_login"))'
Exit status: 6
Duration: 131.07437706

stdout:
2018/08/22 16:38:12 Running arc.Boot
2018/08/22 16:38:12 Restarting ui job
2018/08/22 16:38:13 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:38:13 Asking session_manager to enable Chrome testing
2018/08/22 16:38:13 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:38:13 Removing cryptohome for testuser@gmail.com
2018/08/22 16:38:13 Finding OOBE DevTools target
2018/08/22 16:38:13 Connecting to Chrome at ws://127.0.0.1:36761/devtools/page/380277978C49EB10EC040C792DB0963A
2018/08/22 16:38:13 Waiting for OOBE
2018/08/22 16:38:16 Logging in as user "testuser@gmail.com"
2018/08/22 16:38:16 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
2018/08/22 16:38:23 Waiting for OOBE to be dismissed
2018/08/22 16:38:25 Waiting for Android boot
2018/08/22 16:39:55 Error: [/build/amd64-generic/tmp/portage/chromeos-base/tast-local-tests-cros-0.0.1-r140/work/tast-local-tests-cros-0.0.1/src/chromiumos/tast/local/bundles/cros/arc/boot.go:39] Failed to start ARC: service.adb.tcp.port not set: context deadline exceeded
2018/08/22 16:39:55 Finished arc.Boot
2018/08/22 16:39:55 Running ui.ChromeCrashLoggedIn
2018/08/22 16:39:55 Restarting ui job
2018/08/22 16:39:56 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:39:56 Asking session_manager to enable Chrome testing
2018/08/22 16:39:56 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:39:56 Removing cryptohome for testuser@gmail.com
2018/08/22 16:39:56 Finding OOBE DevTools target
2018/08/22 16:39:56 Connecting to Chrome at ws://127.0.0.1:40447/devtools/page/53AB851F5DA8F9A76F2D71DBDB13933C
2018/08/22 16:39:56 Waiting for OOBE
2018/08/22 16:39:59 Logging in as user "testuser@gmail.com"
2018/08/22 16:39:59 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
2018/08/22 16:40:06 Waiting for OOBE to be dismissed
2018/08/22 16:40:08 Sending SIGSEGV to root Chrome process 14423
2018/08/22 16:40:08 Waiting for 19 Chrome process(es) to exit
2018/08/22 16:40:08 All Chrome processes exited
2018/08/22 16:40:08 Deleting (expected) new minidump: /home/chronos/crash/f603b8ec-40b5-4d8e-6d0536a8-c93ca8dd.dmp
2018/08/22 16:40:08 Finished ui.ChromeCrashLoggedIn
2018/08/22 16:40:08 Running ui.ChromeCrashNotLoggedIn
2018/08/22 16:40:08 Restarting ui job
2018/08/22 16:40:09 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:40:09 Asking session_manager to enable Chrome testing
2018/08/22 16:40:09 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:40:09 Removing cryptohome for testuser@gmail.com
2018/08/22 16:40:09 Sending SIGSEGV to root Chrome process 14917
2018/08/22 16:40:09 Waiting for 5 Chrome process(es) to exit
2018/08/22 16:40:09 All Chrome processes exited
2018/08/22 16:40:09 Deleting (expected) new minidump: /home/chronos/crash/009ef02d-6d2d-482b-492d2b8d-44257732.dmp
2018/08/22 16:40:09 Finished ui.ChromeCrashNotLoggedIn
2018/08/22 16:40:09 Running ui.ChromeLogin
2018/08/22 16:40:09 Restarting ui job
2018/08/22 16:40:10 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:40:10 Asking session_manager to enable Chrome testing
2018/08/22 16:40:10 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:40:10 Removing cryptohome for testuser@gmail.com
2018/08/22 16:40:10 Finding OOBE DevTools target
2018/08/22 16:40:10 Connecting to Chrome at ws://127.0.0.1:45869/devtools/page/B7CED710C89453E25F12722A497AC8B2
2018/08/22 16:40:10 Waiting for OOBE
2018/08/22 16:40:13 Logging in as user "testuser@gmail.com"
2018/08/22 16:40:13 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
2018/08/22 16:40:21 Waiting for OOBE to be dismissed
2018/08/22 16:40:22 Waiting for SessionStateChanged "started" D-Bus signal from session_manager
2018/08/22 16:40:22 Got SessionStateChanged signal
2018/08/22 16:40:22 Creating new blank page
2018/08/22 16:40:22 Connecting to Chrome at ws://127.0.0.1:45869/devtools/page/7A7EB4EE5A1232B4E02034B2AF7E1C5D
2018/08/22 16:40:23 Navigating to http://127.0.0.1:33292
2018/08/22 16:40:23 Got content "Hooray, it worked!"
2018/08/22 16:40:23 Finished ui.ChromeLogin
stderr:
test(s) failed <traceback object at 0x7f71d4de6128>

Traceback (most recent call last):
  <module> at /usr/local/autotest/bin/vm_sanity.py:81
    sys.exit(main(sys.argv[1:]))
  main at /usr/local/autotest/bin/vm_sanity.py:74
    '("dep:chrome" || "dep:chrome_login"))\'')
  system at /usr/local/autotest/common_lib/utils.py:1025
    stdout_tee=TEE_TO_LOGS, stderr_tee=TEE_TO_LOGS).exit_status
  run at /usr/local/autotest/common_lib/utils.py:754
    "Command returned non-zero exit status")
CmdError: Command <local_test_runner '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))'> failed, rc=6, Command returned non-zero exit status
* Command: 
    local_test_runner '(!informational && !disabled && ("dep:chrome" ||
    "dep:chrome_login"))'
Exit status: 6
Duration: 131.07437706

stdout:
2018/08/22 16:38:12 Running arc.Boot
2018/08/22 16:38:12 Restarting ui job
2018/08/22 16:38:13 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:38:13 Asking session_manager to enable Chrome testing
2018/08/22 16:38:13 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:38:13 Removing cryptohome for testuser@gmail.com
2018/08/22 16:38:13 Finding OOBE DevTools target
2018/08/22 16:38:13 Connecting to Chrome at ws://127.0.0.1:36761/devtools/page/380277978C49EB10EC040C792DB0963A
2018/08/22 16:38:13 Waiting for OOBE
2018/08/22 16:38:16 Logging in as user "testuser@gmail.com"
2018/08/22 16:38:16 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
2018/08/22 16:38:23 Waiting for OOBE to be dismissed
2018/08/22 16:38:25 Waiting for Android boot
2018/08/22 16:39:55 Error: [/build/amd64-generic/tmp/portage/chromeos-base/tast-local-tests-cros-0.0.1-r140/work/tast-local-tests-cros-0.0.1/src/chromiumos/tast/local/bundles/cros/arc/boot.go:39] Failed to start ARC: service.adb.tcp.port not set: context deadline exceeded
2018/08/22 16:39:55 Finished arc.Boot
2018/08/22 16:39:55 Running ui.ChromeCrashLoggedIn
2018/08/22 16:39:55 Restarting ui job
2018/08/22 16:39:56 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:39:56 Asking session_manager to enable Chrome testing
2018/08/22 16:39:56 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:39:56 Removing cryptohome for testuser@gmail.com
2018/08/22 16:39:56 Finding OOBE DevTools target
2018/08/22 16:39:56 Connecting to Chrome at ws://127.0.0.1:40447/devtools/page/53AB851F5DA8F9A76F2D71DBDB13933C
2018/08/22 16:39:56 Waiting for OOBE
2018/08/22 16:39:59 Logging in as user "testuser@gmail.com"
2018/08/22 16:39:59 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
2018/08/22 16:40:06 Waiting for OOBE to be dismissed
2018/08/22 16:40:08 Sending SIGSEGV to root Chrome process 14423
2018/08/22 16:40:08 Waiting for 19 Chrome process(es) to exit
2018/08/22 16:40:08 All Chrome processes exited
2018/08/22 16:40:08 Deleting (expected) new minidump: /home/chronos/crash/f603b8ec-40b5-4d8e-6d0536a8-c93ca8dd.dmp
2018/08/22 16:40:08 Finished ui.ChromeCrashLoggedIn
2018/08/22 16:40:08 Running ui.ChromeCrashNotLoggedIn
2018/08/22 16:40:08 Restarting ui job
2018/08/22 16:40:09 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:40:09 Asking session_manager to enable Chrome testing
2018/08/22 16:40:09 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:40:09 Removing cryptohome for testuser@gmail.com
2018/08/22 16:40:09 Sending SIGSEGV to root Chrome process 14917
2018/08/22 16:40:09 Waiting for 5 Chrome process(es) to exit
2018/08/22 16:40:09 All Chrome processes exited
2018/08/22 16:40:09 Deleting (expected) new minidump: /home/chronos/crash/009ef02d-6d2d-482b-492d2b8d-44257732.dmp
2018/08/22 16:40:09 Finished ui.ChromeCrashNotLoggedIn
2018/08/22 16:40:09 Running ui.ChromeLogin
2018/08/22 16:40:09 Restarting ui job
2018/08/22 16:40:10 Waiting for org.chromium.SessionManager D-Bus service
2018/08/22 16:40:10 Asking session_manager to enable Chrome testing
2018/08/22 16:40:10 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2018/08/22 16:40:10 Removing cryptohome for testuser@gmail.com
2018/08/22 16:40:10 Finding OOBE DevTools target
2018/08/22 16:40:10 Connecting to Chrome at ws://127.0.0.1:45869/devtools/page/B7CED710C89453E25F12722A497AC8B2
2018/08/22 16:40:10 Waiting for OOBE
2018/08/22 16:40:13 Logging in as user "testuser@gmail.com"
2018/08/22 16:40:13 Waiting for cryptohome /home/user/ba91a79fe7be848ee609fcd444cb38aafaf56ab4
2018/08/22 16:40:21 Waiting for OOBE to be dismissed
2018/08/22 16:40:22 Waiting for SessionStateChanged "started" D-Bus signal from session_manager
2018/08/22 16:40:22 Got SessionStateChanged signal
2018/08/22 16:40:22 Creating new blank page
2018/08/22 16:40:22 Connecting to Chrome at ws://127.0.0.1:45869/devtools/page/7A7EB4EE5A1232B4E02034B2AF7E1C5D
2018/08/22 16:40:23 Navigating to http://127.0.0.1:33292
2018/08/22 16:40:23 Got content "Hooray, it worked!"
2018/08/22 16:40:23 Finished ui.ChromeLogin
stderr:
test(s) failed

Locals:
  args               : ()
  bg_job             : <autotest_lib.client.common_lib.utils.BgJob object at 0x7f71d77695d0>
  command            : 'local_test_runner \'(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))\''
  env                : None
  extra_paths        : None
  ignore_status      : False
  ignore_timeout     : False
  nickname           : None
  stderr_is_expected : False
  stderr_level       : 40
  stderr_tee         : <object object at 0x7f71d80c1120>
  stdin              : None
  stdout_level       : 10
  stdout_tee         : <object object at 0x7f71d80c1120>
  timeout            : None
  verbose            : True

I was wondering if ARC was available in this VM. :-P

Yeah, you'd need to use "tast run" to check dependencies to skip unsupported tests. Please use this expression instead:

'(!informational && !disabled && ("dep:chrome" || "dep:chrome_login") && !"dep:android")'

Should be fairly trivial. Achuith's correct about not having access to the chroot/the tast tool in chrome-land. (Currently no plans to change that.) So we'd have to use the local_test_runner cmd in the VM. Tried it myself and it seems pretty stable.

We'd have to add a dummy gn target for the test, and translate its result into the gtest format our infra expects. We currently do that for the sanity test, so all of that is pretty easy.

And +dpranke who was curious about running tast/chromeos-defined tests in chrome-land.

Thanks for the summary, Ben -- sounds good!

Let me know if it'd be helpful for local_test_runner to support logging test results in a different format when it's run directly -- that would be easy for me to add. So far, my focus has been on the standard use case where it's executed via the "tast run" command, and the human-readable output that you see when running it directly was mostly an afterthought. If you're going to parse that output directly from vm_sanity, I'd like to make it more structured and unit-tested. :-)

It throws an exception on failure. Not sure if we prefer to just return an error code. 

We should probably list how many tests ran, how many were successful, what failed, and how long everything took perhaps.

> We should probably list how many tests ran,
> how many were successful, what failed, and
> how long everything took perhaps.

Does vm_sanity need to parse that information, or do you just want a human-readable summary at the end of the current output?

Just a human readable summary

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f275fd94375783f830c24aae5a4a6cc07da93da

commit 7f275fd94375783f830c24aae5a4a6cc07da93da
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Aug 24 22:01:39 2018

Generate a single shell script for cros VM tests.

And invoke that instead of invoking a long list of chained commands.

This generates a temporary shell script on the host at test-time,
pushes it over to the VM, then removes it after the test on the host.
(The VM gets auto cleaned-up.)

Bug: 732531,  876587 
Change-Id: I7a2e5749d329d6f7a8716724c8c1060d4f8f80d3
Reviewed-on: https://chromium-review.googlesource.com/1149189
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#586014}
[modify] https://crrev.com/7f275fd94375783f830c24aae5a4a6cc07da93da/build/chromeos/run_vm_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/20d63e10191c59b37f9164489e9422123c0d7d90

commit 20d63e10191c59b37f9164489e9422123c0d7d90
Author: Daniel Erat <derat@chromium.org>
Date: Tue Aug 28 03:54:08 2018

tast: Improve test runner logging when run manually.

Make local_test_runner write easier-to-read output when run
manually (as is done by the vm_sanity.py script):

- Write a separator after each test
- Just print base filenames when tests generate errors
- Print a summary at the end of the run

BUG= chromium:876587 
TEST=ran it manually and observed output

Change-Id: I75def473dce0fa693c93b90c2fb63898e8bcfa95
Reviewed-on: https://chromium-review.googlesource.com/1188029
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/20d63e10191c59b37f9164489e9422123c0d7d90/src/chromiumos/tast/runner/runner.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0c64eb8782641ba38486922082b37bd1a010e905

commit 0c64eb8782641ba38486922082b37bd1a010e905
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Wed Aug 29 18:15:40 2018

vm_sanity: Run Tast tests.

BUG= chromium:876587 
TEST=This is the test.

Change-Id: I9e6873d7aed518bd539bf1062e8c9c0ab89fcc75
Reviewed-on: https://chromium-review.googlesource.com/1186066
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/0c64eb8782641ba38486922082b37bd1a010e905/client/bin/vm_sanity.py

Actually, with achuith's change in #13, we've now started running some tast tests during the sanity test on chrome's chromeos-amd64-generic-rel bot. You can see "local_test_runner" invocations running and passing on the bot:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8936188520240576048/+/steps/cros_vm_sanity_test/0/stdout

Is that sufficient to close this bug out? Or did we want to separate them out at some point and add more cases to the tast target?

That sounds reasonable to me, but I'll let Achuith confirm.

But a question: how does chromeos-amd64-generic-rel pick up new OS images to run in the VM? That seems like it could be a source of fragility.

Say that the Chrome CQ is building ToT Chrome version c2 and using an older OS version o1 for sanity tests, and the Chrome OS CQ is building ToT OS version o2 and using an older Chrome version c1. Someone enables a new Tast test in o2 that passes with c1 but fails with c2 (due to a regression in Chrome ToT).

If chromeos-amd64-generic-rel picks up o2 and starts running the new test, would that suddenly close the Chrome CQ? Or do we have some sort of reverse-PFQ that verifies that the sanity tests in new OS versions pass with Chrome ToT before we pull them into chromeos-amd64-generic-rel?

(This isn't Tast-specific; I think that it applies to anything running in vm_sanity.)

The addition of tast to vm_sanity is stop-gap imo. I think we should have a separate Tast step which we can expand in the future to include more expansive functional testing.

Dan: OS updates are tied to CHROMEOS_LKGM, and LKGM updates also go through the chrome CQ. So as the automated LKGM update from o1->o2 would fail on the chromeos-amd64-generic-rel trybot, as I understand it.

> I think we should have a separate Tast step which we can expand in the future to include more expansive functional testing.

Sounds good. I'll work on getting a separate test target out sometime next week.

There seems to be a sizable leak in the new tast tests we've started running. It looks like two of them (ui.ChromeCrashLoggedIn and ui.ChromeCrashNotLoggedIn) intentionally cause the browser to crash, which creates crash dumps at both "/var/spoole/crash/" and "/home/chronos/crash/". Looking at the logs, the test runner cleans up the ones at /home/chronos/crash/ but not the ones at /var/spoole/crash/, which leaves ~400MB files that look like "chrome.20180906.155826.15036.core" lying around. Could we get those cleaned up as well? (Or alternatively, not generated in the first place?)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70ce32d2309713c1c9b650e4d0d422ffa1c34ca1

commit 70ce32d2309713c1c9b650e4d0d422ffa1c34ca1
Author: Ben Pastene <bpastene@chromium.org>
Date: Mon Sep 10 22:30:52 2018

Add an is_chromeos_chrome gn var, and use it for vm tests.

We're planning on adding more types of VM tests, so this flag should
come in handy in the future.

Bug:  876587 
Change-Id: I0db628151ae2016b4fd891fdbd5064ee41b1720f
Reviewed-on: https://chromium-review.googlesource.com/1217073
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590094}
[modify] https://crrev.com/70ce32d2309713c1c9b650e4d0d422ffa1c34ca1/build/config/chromeos/rules.gni
[modify] https://crrev.com/70ce32d2309713c1c9b650e4d0d422ffa1c34ca1/chromeos/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/679dc3761169cb9d5bb9eeeaf23fb46f6d74ce4f

commit 679dc3761169cb9d5bb9eeeaf23fb46f6d74ce4f
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Tue Sep 11 00:17:52 2018

vm_sanity: Split into functions.

* Add VMSanity class.
* Split into separate functions for Cryptohome, Incognito, Tast, Mash.
* Run Android Tast tests where appropriate.
* Allow running individual tests.
* Run all by default.

BUG= chromium:876587 
TEST=This is the test.

Change-Id: I20e0569eeee7730e976d93c74a1bafb4ef890154
Reviewed-on: https://chromium-review.googlesource.com/1187139
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/679dc3761169cb9d5bb9eeeaf23fb46f6d74ce4f/client/site_tests/login_VMSanity/login_VMSanity.py
[modify] https://crrev.com/679dc3761169cb9d5bb9eeeaf23fb46f6d74ce4f/client/bin/vm_sanity.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/ff25833562534f7af091571517d1d57dce0fd2f8

commit ff25833562534f7af091571517d1d57dce0fd2f8
Author: Daniel Erat <derat@chromium.org>
Date: Wed Sep 12 00:31:43 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b7c18c08fc612dbee789d3d413ce8e3625fd6ea8

commit b7c18c08fc612dbee789d3d413ce8e3625fd6ea8
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Sep 13 23:36:44 2018

Temporarily stop running cros_vm_sanity_test on the CQ.

It's leaving around crash dumps that are filling up the disk and
causing proceeding tests to fail.  crbug.com/881638  fixed this, but that
hasn't rolled into chromium yet. Disable the test until it does.

Bug:  876587 
Change-Id: Iacfdfb6f2f84a32027d4518a15949ab1e3d88677
Reviewed-on: https://chromium-review.googlesource.com/1225934
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591210}
[modify] https://crrev.com/b7c18c08fc612dbee789d3d413ce8e3625fd6ea8/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/b7c18c08fc612dbee789d3d413ce8e3625fd6ea8/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/b7c18c08fc612dbee789d3d413ce8e3625fd6ea8/testing/buildbot/gn_isolate_map.pyl
[modify] https://crrev.com/b7c18c08fc612dbee789d3d413ce8e3625fd6ea8/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/43c1ac34f25b8128be29bc035ab024bbdc889f1e

commit 43c1ac34f25b8128be29bc035ab024bbdc889f1e
Author: Ben Pastene <bpastene@chromium.org>
Date: Mon Sep 17 21:36:14 2018

Revert "Temporarily stop running cros_vm_sanity_test on the CQ."

This reverts commit b7c18c08fc612dbee789d3d413ce8e3625fd6ea8.

Reason for revert: new CHROMEOS_LKGM picked up crash file fix

Original change's description:
> Temporarily stop running cros_vm_sanity_test on the CQ.
> 
> It's leaving around crash dumps that are filling up the disk and
> causing proceeding tests to fail.  crbug.com/881638  fixed this, but that
> hasn't rolled into chromium yet. Disable the test until it does.
> 
> Bug:  876587 
> Change-Id: Iacfdfb6f2f84a32027d4518a15949ab1e3d88677
> Reviewed-on: https://chromium-review.googlesource.com/1225934
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Ben Pastene <bpastene@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#591210}

TBR=bpastene@chromium.org,jbudorick@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  876587 
Change-Id: I330a72a6220cb25405df47733b97596dbea83b14
Reviewed-on: https://chromium-review.googlesource.com/1228301
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591829}
[modify] https://crrev.com/43c1ac34f25b8128be29bc035ab024bbdc889f1e/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/43c1ac34f25b8128be29bc035ab024bbdc889f1e/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/43c1ac34f25b8128be29bc035ab024bbdc889f1e/testing/buildbot/gn_isolate_map.pyl
[modify] https://crrev.com/43c1ac34f25b8128be29bc035ab024bbdc889f1e/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3ccfea10a581352fa02f55c4cc426cfdbcc75ff8

commit 3ccfea10a581352fa02f55c4cc426cfdbcc75ff8
Author: Daniel Erat <derat@chromium.org>
Date: Tue Sep 18 09:25:23 2018

autotest: Don't pass !"dep:android" to Tast in vm_sanity.

Update the local_test_runner command executed by
vm_sanity.py to stop explicitly excluding tests with the
"dep:android" attribute. local_test_runner will now skip
these tests automatically if the DUT doesn't provide the
"android" software feature.

BUG= chromium:882454 , chromium:876587 
TEST=ran updated command after updating local_test_runner
     and verified that android tests are skipped in simple
     chrome VM

Change-Id: Ie7f40623ff0cc42c5aeab69818c8a098108f5ab1
Reviewed-on: https://chromium-review.googlesource.com/1227692
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/3ccfea10a581352fa02f55c4cc426cfdbcc75ff8/client/bin/vm_sanity.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/13f9b8aba1063b5b962146f34bcd202487c701f2

commit 13f9b8aba1063b5b962146f34bcd202487c701f2
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Tue Sep 18 21:04:03 2018

vm_sanity: --run-all instead of --all.

BUG= chromium:876587 
TEST=This is the test.

Change-Id: Ic28dd4125a01631cb8f6c1a463fbc5e287b5a440
Reviewed-on: https://chromium-review.googlesource.com/1217490
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/13f9b8aba1063b5b962146f34bcd202487c701f2/client/bin/vm_sanity.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1688203b04952b7260f7b149fd9f60f440792093

commit 1688203b04952b7260f7b149fd9f60f440792093
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Sep 21 01:16:39 2018

Refactor chromeos GN test configs. Consolidate vm_sanity_test's targets.

We're planning on adding more non-gtest tests. The cleanup here will make
that simpler/easier.

Bug:  876587 
Change-Id: Ie23bd9ed1e38a3dc9df90a087cd8637557683c8a
Reviewed-on: https://chromium-review.googlesource.com/1227148
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#593040}
[modify] https://crrev.com/1688203b04952b7260f7b149fd9f60f440792093/build/config/chromeos/rules.gni
[modify] https://crrev.com/1688203b04952b7260f7b149fd9f60f440792093/chromeos/BUILD.gn
[modify] https://crrev.com/1688203b04952b7260f7b149fd9f60f440792093/testing/test.gni

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70849d7634f014baa057b61d6647b574b638b94d

commit 70849d7634f014baa057b61d6647b574b638b94d
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Sep 25 20:13:25 2018

Add a Tast mode to cros VM test runner.

Tast is a new suite of tests. It's invoked via "local_test_runner" in
the VM. You pass it a set of test names and it'll exercise some test
cases. We shouldn't use much more than the browser flavored ones.

Bug:  876587 
Change-Id: I1d6ae9b6b5a929daadb6c12751cb0cd632a24d95
Reviewed-on: https://chromium-review.googlesource.com/1237673
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#594075}
[modify] https://crrev.com/70849d7634f014baa057b61d6647b574b638b94d/build/chromeos/run_vm_test.py

Ben, how does the run_vm_test.py script in #32 relate to vm_sanity.py?

I tried to document Tast's relationship with the Chrome CQ at https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/master/doc/tast_integration.md#Chrome-Commit-Queue. Is that outdated now?

> Is that outdated now?

Not yet. There's still one more change needed on the chrome side before we can split tast tests out from vm_sanity and into a new target.

> how does the run_vm_test.py script in #32 relate to vm_sanity.py?

It's a wrapper around chromite's vm scripts (which call vm_sanity.py on the vm). It handles how we define tests in chrome land (as GN targets), calculate their data deps, launch them, collect results, etc etc, so we don't have to hard code all that in chromite. It should be mostly invisible for someone creating new tests.

#34: Thanks for the details. I have more questions. :-)

a) Is the case that vm_sanity.py eventually won't execute local_test_runner to run Tast tests? That would probably affect  issue 888887 .

b) The run_vm_test.py change looks like it's adding a --test flag that can be repeated to specify which tests to run, while vm_sanity.py passes the attribute expression '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))' to local_test_runner to run the same set of tests as the Chrome PFQ. What's the reason for --test in run_vm_test.py instead of also passing an attribute expression to match the PFQ?

c) You mentioned that run_vm_tests.py still runs vm_sanity.py via Chrome's VM scripts, but in https://crrev.com/c/1237673, it looks like local_test_runner is being called directly. I assume there's something I'm still misunderstanding here.

> Is the case that vm_sanity.py eventually won't execute local_test_runner to run Tast tests? That would probably affect  issue 888887 .

At the very least, it would no longer be the *only* thing running tast tests via local_test_runner on chrome's waterfalls. Wether or not we remove the tast tests from vm_sanity.py after we're done here is up to y'all. (See also comment #16 earlier.)

> What's the reason for --test in run_vm_test.py instead of also passing an attribute expression to match the PFQ?

Ah, no good reason, just an oversight. I was mainly going off of james' comment at https://bugs.chromium.org/p/chromium/issues/detail?id=847901#c24, where two tests were referenced by name and not a condition. We'll certainly want to pass conditions to. I'll fix that.

> I assume there's something I'm still misunderstanding here.

Hopefully an example will help :)
See https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel/96019, a chromium CQ build that ran with my change in #32 + a WIP followup. It runs both cros_vm_sanity_test and chrome_login_tast_tests (a dummy POC suite I made up that just runs ui.ChromeLogin, ui.ChromeCrashLoggedIn, & ui.ChromeCrashNotLoggedIn)

cros_vm_sanity_test is ran by ssh'ing into the vm and running "vm_sanity.py" (which has some tast tests baked in)
chrome_login_tast_tests is ran by ssh'ing into the vm and running "local_test_runner blah blah" (where "blah blah" are the tests passed in via "--test")

Hopefully that clears things up a bit. LMK if it doesn't and I can go into more detail.

Thanks for pointing at that build; it definitely makes things clearer. And, nice -- it's easier to see what's going on with the Tast tests in their own stage.

Are there any plans for collecting additional system information (e.g. logs and crashes) from the VM to help with debugging test failures? I wonder if it'd be possible to switch to using "tast run" (which will collect this information) instead of executing local_test_runner directly. We package files related to the "tast" command separately for the lab (see https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/master/doc/tast_integration.md#Running-Lab and https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/master/doc/tast_failures.md#Reproducing-failures), although I don't know where the VM images that the Chrome CQ uses come from or whether the corresponding autotest_server_package.tar.bz2 files are available.

In  issue 888887  (where I proposed making login_VMSanity stop running Tast tests via vm_sanity.py), Achuith expressed the desire (which I have too) to make sure that the Chrome OS CQ is running tests using the same configuration that the Chrome CQ is using; this lets us ensure that OS changes don't break the Chrome CQ. If the Chrome CQ starts running Tast via run_vm_test.py, then we won't have that guarantee, right? run_vm_test.py looks like it's part of the chromium repository, so I'm not sure how easy it'd be to run it in the Chrome OS CQ.

I view vm_sanity running the entire Tast suite as a stop-gap measure. Once Tast tests are running independently of vm_sanity, we can either drop them altogether or run a few.

In theory, --host-cmd allows us to run test commands on the host instead of the VM, but we don't have an chromeos checkout, so neither autotest_server_package.tar.bz2 nor Tast is available. You'd have to make everything you need available in chromite.

As for maintaining coherence between the filter expressions we use on the chrome and chromeos CQ, I'm wondering if we should have a wrapper script in chromite and call that from run_vm_test in the chrome CQ and an autotest in chromeOS, similar to what we're doing with vm_sanity and login_VMSanity, so there's no chance of divergence. 

> Are there any plans for collecting additional system information (e.g. logs and crashes) from the VM to help with debugging test failures?

There is a mechanism to collect logs from the VM after each test (currently only enabled for sanity test... we can enable it for tast tests too.) It looks something like: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=a97b46447b68694ce676c42f0e2df6e96416ddbb

> I wonder if it'd be possible to switch to using "tast run"

Like achuith said, we don't have a chromeos checkout on chrome waterfalls, so that'd be tough :(
Though there are a couple other ways of getting the tast tool on chrome bots:
- put it in the simplechrome-sdk (currently contains tarballs of sysroots/toolchains for building simplechrome)
- create a CIPD package for it

Actually, now that I type that out, I realize there's already a cipd package called "chromiumos/infra/tast-cmd"... doh. We have pretty strong support for fetching dependent cipd packages before tests, so we may be able to just use that and do away w/ the direct local_test_runner usage. Lemme play around with it.

Hmmm, when I enter my local cros chroot and run "tast run -build=false localhost:9222 ui.ChromeLogin" it passes fine. But if I download the latest CIPD package (which was last uploaded May 22 03:51 PDT 2018... so pretty old) and run the same command, it spits out the error "Failed to run tests: fork/exec /usr/bin/remote_test_runner: no such file or directory"... but appears to continue running the test anyway? Any idea what's up with that?

I'll move ahead on using local_test_runner directly on chrome's CQ. If/when we get the kinks worked out w/ using the cipd package, we can switch to using "tast run" instead.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/33015a01eb411a769dd83e80b86399be1d117e72

commit 33015a01eb411a769dd83e80b86399be1d117e72
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Sep 28 18:59:14 2018

Add a cros-tast target for chrome login tests. Enable it on cros VM fyi bot.

Bug:  876587 
Change-Id: I340c12102a697d6f1b710594111536de564a338c
Reviewed-on: https://chromium-review.googlesource.com/1239354
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595169}
[modify] https://crrev.com/33015a01eb411a769dd83e80b86399be1d117e72/build/chromeos/create_vm_test_script.py
[modify] https://crrev.com/33015a01eb411a769dd83e80b86399be1d117e72/build/config/chromeos/rules.gni
[modify] https://crrev.com/33015a01eb411a769dd83e80b86399be1d117e72/chromeos/BUILD.gn
[modify] https://crrev.com/33015a01eb411a769dd83e80b86399be1d117e72/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/33015a01eb411a769dd83e80b86399be1d117e72/testing/buildbot/gn_isolate_map.pyl
[modify] https://crrev.com/33015a01eb411a769dd83e80b86399be1d117e72/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/068dcd5c36bc46213253d1cdeb70e4867e97ee54

commit 068dcd5c36bc46213253d1cdeb70e4867e97ee54
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Sep 28 23:46:00 2018

Fix runtime_deps_file arg for tast targets.

TBR=jbudorick

Bug:  876587 
Change-Id: Ic121bccd2343fe48eefbbaf2ccf60d2d01969ca2
Reviewed-on: https://chromium-review.googlesource.com/1252911
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595255}
[modify] https://crrev.com/068dcd5c36bc46213253d1cdeb70e4867e97ee54/build/config/chromeos/rules.gni

#39: Regarding the "chromiumos/infra/tast-cmd" CIPD package, that was just a stopgap measure that I added for running Tast on Chrome OS shards back before switching to use SSP. It was non-ideal since it meant that the version of the tast executable (and remote tests) that we ran didn't match that of the system image (and local tests) on the DUT, which caused all sorts of interesting backward- and forward-compatibility challenges.

I'm pretty sure that I deleted all of the code that generated those packages, so I'd bet that whatever is there is ancient and doesn't work anymore. It'd probably be possible to turn it on again if it'd be useful, but I bet we'd still have the same problems around those packages as before.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/09de394786c06d9f1143770e4fe744aba93cd585

commit 09de394786c06d9f1143770e4fe744aba93cd585
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 02 18:06:42 2018

Support conditionals for tast cros vm tests.

And add a new test "chrome_all_tast_tests" that uses a conditional. And
add that test to the FYI bot.

Bug:  876587 
Change-Id: I9518389338b99207ba045bfc62d87dda02c4e3e9
Reviewed-on: https://chromium-review.googlesource.com/c/1256209
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595905}
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/build/chromeos/create_vm_test_script.py
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/build/chromeos/run_vm_test.py
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/build/config/chromeos/rules.gni
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/chromeos/BUILD.gn
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/testing/buildbot/gn_isolate_map.pyl
[modify] https://crrev.com/09de394786c06d9f1143770e4fe744aba93cd585/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f46cfe8f74ef2b2e9b8286d5bab2492beb40bc14

commit f46cfe8f74ef2b2e9b8286d5bab2492beb40bc14
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Oct 04 00:31:17 2018

Capture system logs for cros VM tast tests.

R=jbudorick

Bug:  876587 
Change-Id: I322398cb10d4c68de8e31231759ebac322c6cbe6
Reviewed-on: https://chromium-review.googlesource.com/c/1260263
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#596442}
[modify] https://crrev.com/f46cfe8f74ef2b2e9b8286d5bab2492beb40bc14/testing/buildbot/gn_isolate_map.pyl

There should be reasonable support for this now in chromium/src. You can defined a tast test by listing test names:
https://codesearch.chromium.org/chromium/src/chromeos/BUILD.gn?rcl=582b7bb5feebc2324793077332cfe86ab845a09e&l=798
Or by a conditional:
https://codesearch.chromium.org/chromium/src/chromeos/BUILD.gn?rcl=582b7bb5feebc2324793077332cfe86ab845a09e&l=806

Then add the target to a chromium bot's test list:
https://codesearch.chromium.org/chromium/src/testing/buildbot/test_suites.pyl?rcl=e05f7e7ce194afc91a53bd0933be5a2187be2337&l=428

Then it'll start running on the bot:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8933583232603841168/+/steps/chrome_all_tast_tests/0/stdout

w/ system logs collected afterwards:
https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=fad9eb22caea6d78e845b6e4285b8c70556a05f2


This is all done by invoking local_test_runner directly... so if we want to use the host-side tast binary instead, we'll still have to come up with a way to make it accessible from chrome land.

Nice!

What does it do if Chrome crashes in a Tast test? Does it get a symbolized stack?

> Does it get a symbolized stack?

Unfortunately not. By default we strip symbols from chrome to get it to fit on the VM. So a crash ends up looking something like:
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=0c2f5a159649297fb9ed196b997e6cb423eb6fcc&as=ui.20181005-105914

That said, if the crash is reproducible, you can rerun it locally with some tweaks and you can get chrome running w/o the stripping and get a symbolized stack.

I've been meaning to explore turning those tweaks on for the bots so we don't have to worry about it. If that's something we'd like for these tests, I can start working on it.

Tast can symbolize .dmp files automagically:

tast symbolize foo.dmp

I'm not sure where it gets the symbol files, though. I'm also not sure if it requires a Chrome OS checkout.

"tast symbolize" downloads symbols from e.g. gs://chromeos-image-archive/cave-release/R65-10286.0.0/debug_breakpad.tar.xz and uses them to run minidump_stackwalk. I don't think it'll help here (since the Chrome binary that's being tested is presumably coming from outside of Chrome OS builders).

Yeah, that prob wouldn't work since the symbols here are entirely local.

Anyway, here's the same trace in #48 but w/ the tweaks I mentioned:
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=27003a6fb30bbe684e69c2e8e362a67983a6627f&as=ui.20181005-134512

I'll work on making that change so it's live/everything gets symbolized.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7100e4eeb9e75353d23a8e0410b0931d2218fc76

commit 7100e4eeb9e75353d23a8e0410b0931d2218fc76
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 09 16:41:57 2018

Move chrome_all_tast_tests from cros VM FYI bot to CQ bot.

Bug:  876587 
Change-Id: Ifb2db6bbe1de690cb9b49ae930c617d520bda97b
Reviewed-on: https://chromium-review.googlesource.com/c/1269722
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#597952}
[modify] https://crrev.com/7100e4eeb9e75353d23a8e0410b0931d2218fc76/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/7100e4eeb9e75353d23a8e0410b0931d2218fc76/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/7100e4eeb9e75353d23a8e0410b0931d2218fc76/testing/buildbot/test_suites.pyl

It looks like the chrome_all_tast_tests step on chromeos-amd64-generic-rel is now running Tast tests matched by '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))' in a VM.

The cros_vm_sanity_test step looks like it's also running Tast tests using exactly the same expression.

We should probably stop running Tast tests as part of cros_vm_sanity_test now, right? Ben, is that the next step here? Just trying to make sure that I'm keeping up. :-)

Yep, all correct. I'm waiting on the symbolization change in #51 to land + be deployed (stuck on https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1266021), then I think we can close this out.

And you're right that we're double running all '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login"))' tests. So we can pull them out of cros_vm_sanity_test now. (Meant to point that out after #52 but got distracted.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e4c8cd9919f63b1a23beb06cb77530a2f3ee2812

commit e4c8cd9919f63b1a23beb06cb77530a2f3ee2812
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 10 04:14:46 2018

cros_run_vm_test: Add option to not strip chrome binaries when deploying to the VM.

We're currently getting unsymbolized traces from the tests when chrome crashes.
This option should fix that.

Note that it appears to slow down the test by ~40s:
Stripping enabled: https://chromium-swarm.appspot.com/task?id=405f26ffed5d3910
No stripping: https://chromium-swarm.appspot.com/task?id=405f282dd7f43010

BUG= chromium:876587 
TEST=cros_run_vm_test --build-dir out/chromeos_amd64_generic/ --deploy --nostrip
as well as:
cros_run_vm_test --build-dir out/chromeos_amd64_generic/ --deploy

Change-Id: I09fe93a392cd2d799bdc231a63b62e9745e85d2b
Reviewed-on: https://chromium-review.googlesource.com/1266021
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/e4c8cd9919f63b1a23beb06cb77530a2f3ee2812/scripts/cros_run_vm_test.py

I'll take care of removing them from vm_sanity

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff7150fe8707d9ca0bb3f00f49d46077f7df0021

commit ff7150fe8707d9ca0bb3f00f49d46077f7df0021
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 10 17:15:46 2018

Don't strip symbols from Chrome when running cros VM tests.

When testing a chrome that crashes, we end up w/ unhelpful stack traces:
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=0c2f5a159649297fb9ed196b997e6cb423eb6fcc&as=ui.20181005-105914

With this option, they become symbolized and much more helpful:
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=27003a6fb30bbe684e69c2e8e362a67983a6627f&as=ui.20181005-134512

Needs https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1266021
before this can land.

Bug:  876587 
Change-Id: Ibea029dad2a7c9385ec533d1ef8f3b043696981c
Reviewed-on: https://chromium-review.googlesource.com/c/1266116
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#598376}
[modify] https://crrev.com/ff7150fe8707d9ca0bb3f00f49d46077f7df0021/build/chromeos/run_vm_test.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c0f7a92e2e5efc58dff72ae552f7a03f44c323da

commit c0f7a92e2e5efc58dff72ae552f7a03f44c323da
Author: Daniel Erat <derat@chromium.org>
Date: Sat Oct 13 00:28:17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7c2fd4105eb2e1f8941433c0cb48a42e7a5ed93b

commit 7c2fd4105eb2e1f8941433c0cb48a42e7a5ed93b
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Oct 18 00:09:36 2018

Ignore --gtest_repeat arg for browser-sanity and tast cros VM tests.

See the retries on something like:
https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel/110795

Failing due to the unknown args. This will just ignore them.

Bug:  876587 
Change-Id: Ibfe012d74af2dff8ee9b35c3dc921ba53c671b83
Reviewed-on: https://chromium-review.googlesource.com/c/1284443
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600613}
[modify] https://crrev.com/7c2fd4105eb2e1f8941433c0cb48a42e7a5ed93b/build/chromeos/run_vm_test.py

There's now a few ways of defining tast test targets for chrome bots:
https://codesearch.chromium.org/chromium/src/chromeos/BUILD.gn?rcl=8b5fda70b6a6aee68242aa1ba3a8c69e345dc468&l=413

Those have been running & passing on the amd64-generic VM bot for a while, so I think it's safe to close this out. LMK if there's anything else or if I missed something, thnx.

Thanks, Ben!

Since I've heard this question and had it myself, is there an easy way to see the results of all recent chrome_all_tast_tests runs on amd64-generic?

That would be the woefully unowned and user-unfriendly test-results dashboard.

The CI bot's history: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&testType=chrome_all_tast_tests
The CQ bot: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&testType=chrome_all_tast_tests%20(with%20patch)

It's not very granular; it only knows the status of the whole suite and not individual tests (eg ui.ChromeLogin). It could be smart enough, but we'd have to parse the stdout of the local_test_runner cmd. That'd prob be easier if/when we switch to using the host-side tast cmd, yeah?

Thanks for the links! Yeah, when using "tast run", you can just parse the results.json or streamed_results.jsonl files in the results dir (more details at https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/running_tests.md#Interpreting-test-results).

Are we sure that this bot is ready for prime-time? I can't seem to debug a failure that's happening only on it... (filed issue 901471).

Tast doesn't use Telemetry, so the failure that you're seeing (in the "telemetry_unittests (with patch)" step) doesn't have anything to do with what got enabled here.