tast: Let tests declare software dependencies |
||||
Issue descriptionAs discussed in http://doc/133fqdEFbUZ6g0IhHFtPQOvl2YHFJg1_QBne8p3rTgz8, I'm planning to update Tast so that individual tests can declare software-based dependencies via a new attribute (probably named "SoftwareDeps"). Breaking it into multiple steps: - Add a new package that installs a file in /etc listing software properties of the system (probably just select USE flags right now). - Update local_test_runner to expose a new mode where it parses the file and returns the properties. - Update the tast command to use local_test_runner to get the properties and then pass them to local_test_runner and remote_test_runner when listing or running tests. - Update local_tast_runner to pass the properties to test bundles, and update the shared test bundle code to use them to filter out tests with unsatisfied dependencies (probably also reporting that the tests have been skipped, and why). - Merge the bvt_arc suite into the bvt suite (after adding appropriate dependencies). - Also set dependencies for tests that exercise login so they won't run on veyron_rialto.
,
Apr 30 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/8b41abc12c5724a957e7bc67bfd37e91d4222615 commit 8b41abc12c5724a957e7bc67bfd37e91d4222615 Author: Daniel Erat <derat@chromium.org> Date: Mon Apr 30 07:40:10 2018 tast: Rename testing/attr package to expr. testing.Registry currently uses the testing/attr package to find tests whose attributes are matched by boolean expressions. Rename attr to expr and promote it to a top-level package so that test runners can use similar expressions to compose USE flags into software features that test bundles will use to check whether tests' dependencies are satisfied before trying to run them. BUG= chromium:835991 TEST=unit tests still pass; also manually verified that test attribute matching works as before when performing a test run via the tast command Change-Id: I6cf0b753201d9af3975b07fa59e6a058e32b6667 Reviewed-on: https://chromium-review.googlesource.com/1034383 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [rename] https://crrev.com/8b41abc12c5724a957e7bc67bfd37e91d4222615/src/chromiumos/tast/expr/expr_test.go [modify] https://crrev.com/8b41abc12c5724a957e7bc67bfd37e91d4222615/src/chromiumos/tast/testing/registry.go [rename] https://crrev.com/8b41abc12c5724a957e7bc67bfd37e91d4222615/src/chromiumos/tast/expr/expr.go
,
May 1 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/c1f15a0f128588e6c2d83c41157351b3eaec5ce3 commit c1f15a0f128588e6c2d83c41157351b3eaec5ce3 Author: Daniel Erat <derat@chromium.org> Date: Tue May 01 15:31:53 2018 tast: Add command.EnumFlag and use for -buildtype. Add a new EnumFlag type that can be used for flags that accept a fixed set of values. Update the tast command's -buildtype flag to use it. BUG= chromium:835991 TEST=added unit tests for EnumFlag; also verified that -buildtype works as before Change-Id: I6406d153ee0c7e622882ad7d1d9bae8ac9b87731 Reviewed-on: https://chromium-review.googlesource.com/1034460 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/c1f15a0f128588e6c2d83c41157351b3eaec5ce3/src/chromiumos/cmd/tast/run/config.go [add] https://crrev.com/c1f15a0f128588e6c2d83c41157351b3eaec5ce3/src/chromiumos/tast/command/flags.go [modify] https://crrev.com/c1f15a0f128588e6c2d83c41157351b3eaec5ce3/src/chromiumos/cmd/tast/run/run.go [add] https://crrev.com/c1f15a0f128588e6c2d83c41157351b3eaec5ce3/src/chromiumos/tast/command/flags_test.go
,
May 2 2018
Sorry, that was probably an autocomplete fail on my part.
,
May 3 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/62f5ad0255e5b690842d2241aa45d6f344769782 commit 62f5ad0255e5b690842d2241aa45d6f344769782 Author: Daniel Erat <derat@chromium.org> Date: Thu May 03 05:20:59 2018 tast-use-flags: Add package. Add a new tast-use-flags package that installs a text file to /etc/tast_use_flags.txt listing USE flags that Tast will examine to determine which tests are relevant for a given system image. This is the same approach used by libchromeos-use-flags to write /etc/ui_use_flags.txt so it can be inspected by session_manager when building the Chrome command line. BUG= chromium:835991 TEST=built it and checked the file Change-Id: I6276469fab3f87e7b7dcc3fdbb2048f7009968c5 Reviewed-on: https://chromium-review.googlesource.com/1039006 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [add] https://crrev.com/62f5ad0255e5b690842d2241aa45d6f344769782/chromeos-base/tast-use-flags/tast-use-flags-0.0.1.ebuild
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/5b270edd587b707a408c3203b2c89d0c3972d219 commit 5b270edd587b707a408c3203b2c89d0c3972d219 Author: Daniel Erat <derat@chromium.org> Date: Fri May 04 10:01:54 2018 tast: Add testing.Test.SoftwareDeps. Add a SoftwareDeps field to the testing package's Test struct. Also add a MissingSoftwareDeps method that can be used to check a test's dependencies against the software features supported by the DUT's system image. BUG= chromium:835991 TEST=added unit tests and manually verified in conjunction with followup changes Change-Id: I5f108bf1a2f664aae6a3b6f97127d61d95eb1fbc Reviewed-on: https://chromium-review.googlesource.com/1038890 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/5b270edd587b707a408c3203b2c89d0c3972d219/src/chromiumos/tast/testing/test.go [modify] https://crrev.com/5b270edd587b707a408c3203b2c89d0c3972d219/docs/writing_tests.md [modify] https://crrev.com/5b270edd587b707a408c3203b2c89d0c3972d219/src/chromiumos/tast/testing/test_test.go [add] https://crrev.com/5b270edd587b707a408c3203b2c89d0c3972d219/docs/test_dependencies.md
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed commit eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed Author: Daniel Erat <derat@chromium.org> Date: Fri May 04 10:01:55 2018 tast: Update bundles to check test software dependencies. Update the bundle package to check tests' software dependencies against features supported by the system image before running them. Tests with unsatisfied dependencies are skipped, and a test error is additionally reported if a test depends on unknown software features. Also update TestEnd control messages to include a new MissingSoftwareDeps field that is set when the test is skipped due to unsatisfied dependencies. (This change shouldn't have any effect since test runners don't yet pass the DUT's software features to test bundles.) BUG= chromium:835991 TEST=added unit tests and manually verified in conjunction with followup changes Change-Id: Ieb7fafe9541d2c739fe0a97d8ef9575a68df38c8 Reviewed-on: https://chromium-review.googlesource.com/1038891 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed/src/chromiumos/tast/control/control.go [modify] https://crrev.com/eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed/src/chromiumos/tast/bundle/bundle_test.go [modify] https://crrev.com/eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed/src/chromiumos/tast/control/control_test.go [modify] https://crrev.com/eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed/src/chromiumos/tast/bundle/bundle.go [modify] https://crrev.com/eb04ee24d30d2a44a4efe0c30e6ce67ec660a1ed/src/chromiumos/tast/bundle/args.go
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/31322b2d59c05edfb9caa21497c8c47985bc1a94 commit 31322b2d59c05edfb9caa21497c8c47985bc1a94 Author: Daniel Erat <derat@chromium.org> Date: Fri May 04 10:01:56 2018 tast: Update tast command to check test dependencies. Update the tast command to use local_test_runner to get the DUT's software features and to later pass those features when running tests so that test bundles can skip tests with unsatisfied dependencies. Dependency checking is controlled by a new -checktestdeps flag on the "run" subcommand. The default value, "auto", attempts to do the right thing (dependencies are checked when the DUT reports them and an attribute expression was used to match tests, as happens in the lab). "always" and "never" values are also available to force specific behavior. TestResult objects written to results.json are updated to include a new "skipReason" string field that is nonempty if the test was skipped (currently, just due to missing dependencies). BUG= chromium:835991 TEST=added unit tests; also manually tested by manipulating /etc/tast_use_flags.txt, tests' SoftwareDeps fields, and the -checktestdeps flag and test patterns passed to the "tast run" command Change-Id: If4f8bfcb14e91f845c38081f4b58570011fcc91f Reviewed-on: https://chromium-review.googlesource.com/1038892 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/remote.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/results_test.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/remote_test.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/local.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/local_test.go [add] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/deps_test.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/results.go [add] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/deps.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/src/chromiumos/cmd/tast/run/config.go [modify] https://crrev.com/31322b2d59c05edfb9caa21497c8c47985bc1a94/docs/running_tests.md
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/aba96b9f6f0872e47fe535333eac45cb9e7afa7d commit aba96b9f6f0872e47fe535333eac45cb9e7afa7d Author: Daniel Erat <derat@chromium.org> Date: Fri May 04 10:01:55 2018 tast: Update runners for test software dependencies. Update the runner package to pass args related to software dependencies/features through to test bundles. Also update local_test_runner to support a new GetSoftwareFeaturesMode command telling it to return a list of software features supported by the DUT's system image and add initial definitions for the "android", "chrome", and "chrome_login" features. BUG= chromium:835991 TEST=added unit tests and manually verified in conjunction with followup changes Change-Id: I9b2a2887775feb7cebd2914758a80a01b6588e55 Reviewed-on: https://chromium-review.googlesource.com/1040911 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/aba96b9f6f0872e47fe535333eac45cb9e7afa7d/src/chromiumos/tast/runner/args.go [modify] https://crrev.com/aba96b9f6f0872e47fe535333eac45cb9e7afa7d/src/chromiumos/cmd/local_test_runner/main.go [modify] https://crrev.com/aba96b9f6f0872e47fe535333eac45cb9e7afa7d/src/chromiumos/tast/runner/runner.go [add] https://crrev.com/aba96b9f6f0872e47fe535333eac45cb9e7afa7d/src/chromiumos/tast/runner/features_test.go [add] https://crrev.com/aba96b9f6f0872e47fe535333eac45cb9e7afa7d/src/chromiumos/tast/runner/features.go
,
May 5 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/21167abd12a0648b2870ab1c8ef3238f414f660f commit 21167abd12a0648b2870ab1c8ef3238f414f660f Author: Daniel Erat <derat@chromium.org> Date: Sat May 05 03:37:37 2018 target-chromium-os-test: Depend on tast-use-flags. Make target-chromium-os-test depend on tast-use-flags so /etc/tast_use_flags.txt will be installed on test images. BUG= chromium:835991 TEST=none Change-Id: Ia5824166a35d2dbbe00d194860db0209a5e0d8cd Reviewed-on: https://chromium-review.googlesource.com/1043103 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [rename] https://crrev.com/21167abd12a0648b2870ab1c8ef3238f414f660f/virtual/target-chromium-os-test/target-chromium-os-test-1-r67.ebuild [modify] https://crrev.com/21167abd12a0648b2870ab1c8ef3238f414f660f/virtual/target-chromium-os-test/target-chromium-os-test-1.ebuild
,
May 8 2018
,
May 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/36c43084655db1722463cf5d8a9dc24518982c99 commit 36c43084655db1722463cf5d8a9dc24518982c99 Author: Daniel Erat <derat@chromium.org> Date: Tue May 08 00:23:53 2018 tast-tests: Set SoftwareDeps for local tests. Specify "android", "chrome", and "chrome_login" software dependencies as appropriate for various local tests in the "cros" bundle. This will ensure that tests that require Android won't run on boards that have been mislabeled as supporting it and that tests that require Chrome login won't run on veyron_rialto, for instance. BUG= chromium:835991 TEST=manipulated /etc/tast_use_flags.txt on a device and checked that (in conjunction with updated tast/runner/bundle) tests were skipped when their dependencies weren't satisfied CQ-DEPEND=I5f108bf1a2f664aae6a3b6f97127d61d95eb1fbc Change-Id: Icd63b110c3388867e2e8623025a9aca5d82a2727 Reviewed-on: https://chromium-review.googlesource.com/1042028 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/36c43084655db1722463cf5d8a9dc24518982c99/src/chromiumos/tast/local/bundles/cros/ui/arc_startup.go [modify] https://crrev.com/36c43084655db1722463cf5d8a9dc24518982c99/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_logged_in.go [modify] https://crrev.com/36c43084655db1722463cf5d8a9dc24518982c99/src/chromiumos/tast/local/bundles/cros/ui/chrome_login.go [modify] https://crrev.com/36c43084655db1722463cf5d8a9dc24518982c99/src/chromiumos/tast/local/bundles/cros/ui/mash_login.go [modify] https://crrev.com/36c43084655db1722463cf5d8a9dc24518982c99/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_not_logged_in.go
,
May 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/da93534b150ccb18e117fa4afb3598d74a1dcda2 commit da93534b150ccb18e117fa4afb3598d74a1dcda2 Author: Daniel Erat <derat@chromium.org> Date: Tue May 08 00:23:54 2018 tast: Read tast_use_flags.txt from /usr/local/etc. Make local_test_runner read tast_use_flags.txt from /usr/local/etc rather than /etc. The file is diverted to /usr/local since the tast-use-flags package is installed by virtue of a dependency in target-chromium-os-test-1. BUG= chromium:835991 TEST=none Change-Id: I9a3ec9d40779e91f59f452a4ac74f57637ecf2cd Reviewed-on: https://chromium-review.googlesource.com/1045908 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/da93534b150ccb18e117fa4afb3598d74a1dcda2/src/chromiumos/cmd/local_test_runner/main.go
,
May 9 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8f486e293f2597e49074739cc56dd01676b6c77d commit 8f486e293f2597e49074739cc56dd01676b6c77d Author: Daniel Erat <derat@chromium.org> Date: Wed May 09 04:15:43 2018 autotest: Make tast server test handle skipped tests. Make the tast server test avoid reporting tests that were skipped (which currently indicates that the test declared software dependencies that were unfulfilled by the DUT). BUG= chromium:835991 TEST=changed DUT's tast_use_flags.txt file to list "rialto" and checked that tests that depend on chrome_login are omitted by status.log Change-Id: I29b48272811fd13f10288f41c71ecd6ac9992222 Reviewed-on: https://chromium-review.googlesource.com/1046105 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/8f486e293f2597e49074739cc56dd01676b6c77d/server/site_tests/tast/tast.py
,
May 9 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/366f96e2fb9d1e96571b4aa9bffa4a09f12366ca commit 366f96e2fb9d1e96571b4aa9bffa4a09f12366ca Author: Daniel Erat <derat@chromium.org> Date: Wed May 09 04:15:40 2018 autotest: Remove tast.bvt_arc test. Tast automatically skips running ARC tests on non-ARC systems now, so there's no reason to have separate ARC and non-ARC BVT suites. Make tast.bvt run all BVT tests instead. BUG= chromium:835991 TEST=none Change-Id: I17c21b4a7b185908f021e05e7c180f591381a79a Reviewed-on: https://chromium-review.googlesource.com/1046109 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/366f96e2fb9d1e96571b4aa9bffa4a09f12366ca/server/site_tests/tast/control.bvt [delete] https://crrev.com/8b9812e3c13db2ec632765a59e05b42f89b68751/server/site_tests/tast/control.bvt_arc
,
May 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/5c7d82fffa8b4f71dabcf3027e6d86a14419ccee commit 5c7d82fffa8b4f71dabcf3027e6d86a14419ccee Author: Daniel Erat <derat@chromium.org> Date: Thu May 10 22:09:51 2018 tast-tests: Remove "arc" and "chrome" attributes. Remove the "arc" and "chrome" attributes from tests' Attr fields. These have been replaced by the automatically-added "dep:android" and "dep:chrome"/"dep:chrome_login" attributes. BUG= chromium:835991 TEST=none Change-Id: Ieb81c41f7a8fac375e578b06d634842a2782d4e3 Reviewed-on: https://chromium-review.googlesource.com/1053629 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/5c7d82fffa8b4f71dabcf3027e6d86a14419ccee/src/chromiumos/tast/local/bundles/cros/ui/arc_startup.go [modify] https://crrev.com/5c7d82fffa8b4f71dabcf3027e6d86a14419ccee/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_logged_in.go [modify] https://crrev.com/5c7d82fffa8b4f71dabcf3027e6d86a14419ccee/src/chromiumos/tast/local/bundles/cros/ui/chrome_login.go [modify] https://crrev.com/5c7d82fffa8b4f71dabcf3027e6d86a14419ccee/src/chromiumos/tast/local/bundles/cros/ui/mash_login.go [modify] https://crrev.com/5c7d82fffa8b4f71dabcf3027e6d86a14419ccee/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_not_logged_in.go
,
May 11 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/877634e6b1e5d87de6494cbe585027d5f5c2735f commit 877634e6b1e5d87de6494cbe585027d5f5c2735f Author: Daniel Erat <derat@chromium.org> Date: Fri May 11 16:30:42 2018 tast: Avoid getting DUT's software features unnecessarily. Update the chromiumos/cmd/tast/run package's getSoftwareFeatures function to bail out early if it's already run successfully. This avoids making an extra unnecessary round trip to the DUT when we're doing both a local and remote test run. Also avoid fetching features when running "tast list" since we don't check software dependencies in that mode. Finally, make the dependency list in "Skipped test ____ due to missing dependencies" log messages a bit easier to read and shorten a message about skipping dependency checks. BUG= chromium:835991 TEST=added and updated unit tests; also ran both local and remote tests and verified that "Getting software features supported by DUT" is only logged once Change-Id: I6b194ace97b214af5343fc43dc5c118c983e0a1d Reviewed-on: https://chromium-review.googlesource.com/1054138 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/877634e6b1e5d87de6494cbe585027d5f5c2735f/src/chromiumos/cmd/tast/run/deps_test.go [modify] https://crrev.com/877634e6b1e5d87de6494cbe585027d5f5c2735f/src/chromiumos/cmd/tast/run/config.go [add] https://crrev.com/877634e6b1e5d87de6494cbe585027d5f5c2735f/src/chromiumos/cmd/tast/run/config_test.go [modify] https://crrev.com/877634e6b1e5d87de6494cbe585027d5f5c2735f/src/chromiumos/cmd/tast/run/results.go [modify] https://crrev.com/877634e6b1e5d87de6494cbe585027d5f5c2735f/src/chromiumos/cmd/tast/run/deps.go
,
May 11 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by ihf@chromium.org
, Apr 24 2018