Prepare to run performance tests in Tast |
||
Issue descriptionTW team wants to run performance (aka crosbolt) tests periodically. They have some special requirements: - Performance tests should run in a special DUT pool, different from the usual bvt pool. - Performance tests should not run in CQ and release builders (because their tests often run very long). We need some work to make this happen. 1. Define a new attribute for performance tests. 2. Update existing control files to exclude tests with the new attribute. 3. Add a control file to run tests with the new attribute. 4. Add performance tests. My plan is to name the attribute "crosbolt" and simply exclude it from existing control files. One obvious cons is that it does not scale well if we have more cases like this. I welcome your thoughts.
,
Oct 12
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/0f84279bad57090e6433e0f94a31c7b31b4f3451 commit 0f84279bad57090e6433e0f94a31c7b31b4f3451 Author: Daniel Erat <derat@chromium.org> Date: Fri Oct 12 22:16:49 2018 tast: Make expr package support wildcards. Update the expr package (used to evaluate boolean expressions specifying which tests to run or how to determine features from USE flags) to treat '*' characters in identifiers as wildcards. I've heard requests in the past to have some way to match all tests named "ui.*" in attribute expressions, and this also seems like it will be helpful if we want to add test attributes like "suite:foo" and then exclude those tests from the builders in the CQ and PFQs. BUG= chromium:894375 TEST=added unit tests Change-Id: I1f7eba748580d3a339e022c5ee6d6433d0699ca6 Reviewed-on: https://chromium-review.googlesource.com/1277709 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/src/chromiumos/tast/testing/registry.go [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/src/chromiumos/tast/testing/registry_test.go [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/src/chromiumos/tast/runner/features_test.go [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/src/chromiumos/tast/expr/expr_test.go [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/src/chromiumos/tast/runner/args.go [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/docs/running_tests.md [modify] https://crrev.com/0f84279bad57090e6433e0f94a31c7b31b4f3451/src/chromiumos/tast/expr/expr.go
,
Oct 20
To recap discussion in https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1275926 We'll introduce new attribute "group:foo". It is assigned to tests to be run outside of the usual CQ/PFQ/canary/release. We'll update the boolean expressions used to select Tast tests in those places to include !"group:*".
,
Oct 22
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/e4be69c35d4c1902b6c43f51a300be47d054118d commit e4be69c35d4c1902b6c43f51a300be47d054118d Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Oct 22 17:14:28 2018 chromeos_config: Exclude Tast tests in any group from VM tests. BUG= chromium:894375 TEST=run_tests Change-Id: I8cbc27201581c354df3b78c35025dd750ed88773 Reviewed-on: https://chromium-review.googlesource.com/1292690 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/e4be69c35d4c1902b6c43f51a300be47d054118d/config/chromeos_config.py [modify] https://crrev.com/e4be69c35d4c1902b6c43f51a300be47d054118d/config/config_dump.json
,
Oct 22
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/63bf30ea7effb4183c826bacbfedd7f0413a2ec0 commit 63bf30ea7effb4183c826bacbfedd7f0413a2ec0 Author: Shuhei Takahashi <nya@google.com> Date: Mon Oct 22 17:59:44 2018 tast: Exclude tests in any group. BUG= chromium:894375 TEST=None Change-Id: I905b4dd02a40e515fdd7f47f3b3efb845953841e Reviewed-on: https://chromium-review.googlesource.com/c/1292650 Reviewed-by: Dan Erat <derat@chromium.org> Commit-Queue: Shuhei Takahashi <nya@chromium.org> Cr-Commit-Position: refs/heads/master@{#601646} [modify] https://crrev.com/63bf30ea7effb4183c826bacbfedd7f0413a2ec0/chromeos/BUILD.gn
,
Oct 23
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f91f84d3b395f82849c7859fb050e2e292c0860d commit f91f84d3b395f82849c7859fb050e2e292c0860d Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Oct 23 20:01:38 2018 tast: Add control files for crosbolt. Three control files (perbuild, nightly, weekly) are added to run crosbolt Tast tests separately from CQ and release builders. This CL defines following new attributes: - "group:crosbolt": Indicates the tests should be run for crosbolt. CQ and release builders avoid running these tests. - "crosbolt_{perbuild, nightly, weekly}": Indicates the tests to run in specified frequency. Always used with "group:crosbolt" attribute. CQ-DEPEND=CL:1277709 BUG= chromium:894375 TEST=test_that DUT tast.crosbolt-{nightly,perbuild,weekly} Change-Id: I6b228c9c85cfa2253830abd217cf2c51b1e61ba0 Reviewed-on: https://chromium-review.googlesource.com/1275926 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [add] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.crosbolt-perbuild [modify] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.mustpass-system [modify] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.mustpass-android [add] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.crosbolt-nightly [modify] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.informational-chrome [modify] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.informational-android [add] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.crosbolt-weekly [modify] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.informational-system [modify] https://crrev.com/f91f84d3b395f82849c7859fb050e2e292c0860d/server/site_tests/tast/control.mustpass-chrome
,
Nov 15
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f90e229a9fb9c488c94e68fcc611447a2c25dbae commit f90e229a9fb9c488c94e68fcc611447a2c25dbae Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Nov 16 23:10:50 2018 tast-tests: Skip informational check if group: attr is set. BUG= chromium:894375 TEST=Manually created offending commit Change-Id: Idda33c31784bd78c0452b6ceaaadfe5173d641d7 Reviewed-on: https://chromium-review.googlesource.com/1335370 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> Reviewed-by: Hidehiko Abe <hidehiko@chromium.org> [modify] https://crrev.com/f90e229a9fb9c488c94e68fcc611447a2c25dbae/tools/check_informational.py |
||
►
Sign in to add a comment |
||
Comment 1 by nya@chromium.org
, Oct 11