Add a new chromite script that has the same features/functionality of cros_run_vm_test, minus the VM parts |
||||||
Issue descriptionI'm thinking about running cros hardware tests on chrome's bots, and essentially I'm looking for a test runner that has the same args as cros_run_vm_test: --files covers pushing and mapping data dependencies to the device (and cleans up the cwd after the test) --files-from takes care of retrieving result files from the device after the test --remote-cmd builds the ssh command that we want to invoke (gtest binary/sanity test script/local_test_runner) --host-cmd takes care of telemetry's weird use case --deploy also deploys chrome to the device before running any tests, which we use on a bunch of tests That API covers every use-case for all the tests that we'd want to run, and it utilizes the support libs throughout chromite, which are far more stable than anything I could come up with in a reasonable amount of time. So I pretty much want cros_run_vm_test minus the VM bits... WDYT? If no one objects, I can take a stab at adding this. I'd prob try to reuse as much in cros_run_vm_test.py as possible (or maybe refactor it out).
,
Oct 19
I can probably do this for you unless you're keen on doing it.
,
Oct 19
As background, I'd prefer to move away from running local_test_runner directly if possible since there are benefits provided by the "tast" command that we'll miss out on: - clean, readable logging about pre-testing state - nice test status reporting - collection of timing information - smart log/crash collection (i.e. only stuff that's generated during testing) - auto-symbolization of minidump files (easy to add if symbol files are on the bot) - smart handling of DUT reboots (planned) etc.
,
Oct 19
> it's likely easier to just resurrect the Tast CIPD packages Resurrecting it sgtm. I think I recall you mentioning somewhere that using the CIPD package was prone to version skew w/ the local_test_runner tool on the device? Would that be a problem for us? FTR, we pin the version of CIPD packages we use on chrome bots in chromium/src. eg: https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/test_suites.pyl#409 > I can probably do this for you unless you're keen on doing it. I'm happy to do it, but I wouldn't complain if someone else did ;)
,
Oct 19
I'll do it - your time is better spent on infra tasks that I cannot do :)
,
Oct 19
,
Oct 19
#4: I try to maintain compatibility (within reason) between different versions of the tast and local_test_runner executables. The potential problem was rather version skew between remote tests (which live alongside the tast executable rather than on the DUT) and the system image that's being tested. I don't think we'll want to run any remote tests here, so that probably won't be an issue. I may need to make a tiny change to support not trying to run remote tests. How do the pinned package versions get updated? It'd probably be ideal if we could always use the latest version of the tast-cmd package on bots. Is that feasible to do, or is the uprev process manual right now?
,
Oct 22
+Alex who really should be looped in here, as he is working on a VMTest rearchitecture and has already invested some effort in simplifying the existing stack.
,
Oct 26
,
Oct 31
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/469fef49546c02c030c2b82cd4afa7b777d95523 commit 469fef49546c02c030c2b82cd4afa7b777d95523 Author: Achuith Bhandarkar <achuith@chromium.org> Date: Wed Oct 31 15:17:37 2018 cros_run_vm_test: type 'path' for arg parsing. Also replace self.board with self._vm.board. BUG= chromium:896967 TEST=manual Change-Id: I4d59fd2bb17f95f9ea37022f4fb545b1246737cf Reviewed-on: https://chromium-review.googlesource.com/1308739 Commit-Ready: Achuith Bhandarkar <achuith@chromium.org> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> [modify] https://crrev.com/469fef49546c02c030c2b82cd4afa7b777d95523/scripts/cros_run_vm_test.py
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/9f49acaa684ad4db182a5740957a4596e76b5067 commit 9f49acaa684ad4db182a5740957a4596e76b5067 Author: Achuith Bhandarkar <achuith@chromium.org> Date: Fri Nov 16 23:10:41 2018 cros_vm: Add super class Device. Move some VM methods into Device class in preparation for support for DUTs. BUG= chromium:896967 TEST=manual CQ-DEPEND=CL:1311741 Change-Id: I1cac7316f4f607fdb4e1e67401740a6d7d667532 Reviewed-on: https://chromium-review.googlesource.com/1311740 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/9f49acaa684ad4db182a5740957a4596e76b5067/scripts/cros_vm.py
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/0aff062ab51bd610436041876bd1b144f35d6250 commit 0aff062ab51bd610436041876bd1b144f35d6250 Author: Achuith Bhandarkar <achuith@chromium.org> Date: Fri Nov 16 23:10:42 2018 cros_run_vm_test: Support for running tests on a device. * Rename VMTest to CrOSTest. * self._device instead of self._vm * Other fixes. BUG= chromium:896967 TEST=manual CQ-DEPEND=CL:1311740 Change-Id: Ieaec0012caa7651e52d6d0d946d49fbf96b293bf Reviewed-on: https://chromium-review.googlesource.com/1311741 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> [modify] https://crrev.com/0aff062ab51bd610436041876bd1b144f35d6250/scripts/cros_run_vm_test.py
,
Dec 6
,
Dec 6
Awesome, thanks Achuith!! I'll start trying to use it. Will let you know if I run into any issues
,
Dec 15
This is really great. With the new script that was added*, I've successfully ran a GTest (https://chromium-swarm-dev.appspot.com/task?id=41c84cd09572ac10), the sanity test (https://chromium-swarm-dev.appspot.com/task?id=41c84c35fbc3b010), and all the chrome-related tast tests (https://chromium-swarm-dev.appspot.com/task?id=41c860ab64393810) on chromebooks in our lab. And hooking everything up was trivially easy since we already had stuff pretty much working with cros_run_vm_test. Thnx again! * plus one tiny fix (https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1378910)
,
Dec 18
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/d4cbc7359fcd9a6b26ae95848460783f2d5a6735 commit d4cbc7359fcd9a6b26ae95848460783f2d5a6735 Author: Ben Pastene <bpastene@chromium.org> Date: Tue Dec 18 08:42:51 2018 device: Move vm.VM's RunCommand method into its parent class. bin/cros_test's deploy-chrome hook calls "device.RunCommand()" which works for instances of VMs, but not the generic Device class. This fixes that. TEST=`cros_run_test --deploy` with a VM and DUT BUG= chromium:896967 Change-Id: I1e50d23f7cab1ed47a344f222ee0cb42d58df9b4 Reviewed-on: https://chromium-review.googlesource.com/1378910 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> [modify] https://crrev.com/d4cbc7359fcd9a6b26ae95848460783f2d5a6735/lib/device.py [modify] https://crrev.com/d4cbc7359fcd9a6b26ae95848460783f2d5a6735/lib/vm.py |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by derat@chromium.org
, Oct 19