Rework Tast test suite composition |
||
Issue descriptionRight now, the tast.bvt Autotest server test runs all tests with the "bvt" attribute. Per http://doc/1ZyozTa907IJqw7QJNHaWIQ07v0szpUOzdLG2K6cURII, I'm planning to instead introduce "informational" and "disabled" attributes and use them to divide tests into two suites ("(!informational && !disabled)" and "(informational && !disabled)"). The way that these suites will be assigned to Autotest suites is still under discussion.
,
Jun 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/c33e95eb676826b4828ef79a15c71b6461692014 commit c33e95eb676826b4828ef79a15c71b6461692014 Author: Daniel Erat <derat@chromium.org> Date: Wed Jun 20 01:02:53 2018 tast: Add disabled and informational to test_attributes.md. Document the new "disabled" and "informational" test attributes and remove the old "bvt" attribute. BUG= chromium:853545 TEST=viewed with md_browser Change-Id: I5c185031ea5518744f7a9265a0c60c298febded4 Reviewed-on: https://chromium-review.googlesource.com/1107267 Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/c33e95eb676826b4828ef79a15c71b6461692014/docs/test_attributes.md
,
Jun 20 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/41451552f2eae83fe55c317a274145b7d911ff15 commit 41451552f2eae83fe55c317a274145b7d911ff15 Author: Daniel Erat <derat@chromium.org> Date: Wed Jun 20 05:04:00 2018
,
Jun 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f98fe1901466b4239fb46911754fb453eb238d7f commit f98fe1901466b4239fb46911754fb453eb238d7f Author: Daniel Erat <derat@chromium.org> Date: Thu Jun 21 15:14:23 2018 tast-tests: Add "informational" and "disabled" attributes. Add new test attributes that replace the existing "bvt" attribute: - Tests with none of the new labels must pass. - Tests with "informational" run in bvt-perbuild. - Tests with "disabled" aren't run automatically. BUG= chromium:853545 TEST=none CQ-DEPEND=I3abb61d2999c22ed110d9e6ec1d405333d2051ea Change-Id: Ice0d28ad5c27ca019667f5f2baba43c986ec57d9 Reviewed-on: https://chromium-review.googlesource.com/1105128 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_logged_in.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/arc/intent_forward.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/chrome_login.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/security/log_perms.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/remote/bundles/cros/example/reconnect_to_dut.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/vm/start_termina_vm.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/dbus.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/platform/check_processes.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/power/check_status.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_not_logged_in.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/pass.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/arc_startup.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/fail.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/data_files.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/mash_login.go [modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/remote/bundles/cros/power/reboot.go
,
Jun 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c527bcc4ce659e27c76d427a6f7844c4e5617bc8 commit c527bcc4ce659e27c76d427a6f7844c4e5617bc8 Author: Daniel Erat <derat@chromium.org> Date: Thu Jun 21 15:14:22 2018 autotest: Add tast.informational and tast.must_pass. Add control files for new tast.informational and tast.must_pass server tests and remove the existing tast.bvt test. Both of the new tests run in bvt-perbuild initially. BUG= chromium:853545 TEST=none CQ-DEPEND=Ice0d28ad5c27ca019667f5f2baba43c986ec57d9 Change-Id: I3abb61d2999c22ed110d9e6ec1d405333d2051ea Reviewed-on: https://chromium-review.googlesource.com/1105132 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [copy] https://crrev.com/c527bcc4ce659e27c76d427a6f7844c4e5617bc8/server/site_tests/tast/control.must_pass [rename] https://crrev.com/c527bcc4ce659e27c76d427a6f7844c4e5617bc8/server/site_tests/tast/control.informational
,
Jun 22 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/15a63a040c9d96903acdc526f3c10a3311b3cfdb commit 15a63a040c9d96903acdc526f3c10a3311b3cfdb Author: Daniel Erat <derat@chromium.org> Date: Fri Jun 22 13:00:17 2018 autotest: Add debug links to tast.{informational,must_pass}. Update the control files for the tast.informational and tast.must_pass to point to http://go/tast-failures. BUG= chromium:853545 TEST=none Change-Id: If44af2b7071d469ca4b56cf76ab24ec988b2bbf8 Reviewed-on: https://chromium-review.googlesource.com/1110938 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/15a63a040c9d96903acdc526f3c10a3311b3cfdb/server/site_tests/tast/control.must_pass [modify] https://crrev.com/15a63a040c9d96903acdc526f3c10a3311b3cfdb/server/site_tests/tast/control.informational
,
Jun 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/eb9079ec8d3d2a8fe82008ccd364feab2ef03415 commit eb9079ec8d3d2a8fe82008ccd364feab2ef03415 Author: Daniel Erat <derat@chromium.org> Date: Fri Jun 29 21:16:38 2018 chromeos_config: Update attribute expression for TastVMTest. Change the test attribute expression for betty-release's TastVMTest stage to match all non-disabled tests. It formerly matched all tests with 'bvt' or 'canary' attributes, neither of which exist anymore. BUG= chromium:853545 TEST=ran a tryjob Change-Id: I405a5557817169aa2dbcbc3610b2cc667cd67b24 Reviewed-on: https://chromium-review.googlesource.com/1120429 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/eb9079ec8d3d2a8fe82008ccd364feab2ef03415/config/chromeos_config.py [modify] https://crrev.com/eb9079ec8d3d2a8fe82008ccd364feab2ef03415/lib/config_lib_unittest.py [modify] https://crrev.com/eb9079ec8d3d2a8fe82008ccd364feab2ef03415/config/config_dump.json
,
Jun 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/525be26fdc4ad98beff2cf284524c140f765f54d commit 525be26fdc4ad98beff2cf284524c140f765f54d Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Fri Jun 29 22:39:39 2018 Roll src/third_party/chromite f028e346c07f..1bf75a651fe5 (3 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/f028e346c07f..1bf75a651fe5 git log f028e346c07f..1bf75a651fe5 --date=short --no-merges --format='%ad %ae %s' 2018-06-29 xixuan@chromium.org report_stage: Add abortSkylabHWTestStage. 2018-06-29 derat@chromium.org chromeos_config: Update attribute expression for TastVMTest. 2018-06-29 vadimsh@chromium.org cipd: Bump cipd client version to the one that supports v2 protocol. Created with: gclient setdep -r src/third_party/chromite@1bf75a651fe5 The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. BUG= chromium:845314 , chromium:853545 ,chromium:801423 TBR=chrome-os-gardeners@chromium.org Change-Id: I132c3b105686a4bcf8573121cdad6e7ba6246b59 Reviewed-on: https://chromium-review.googlesource.com/1121217 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#571685} [modify] https://crrev.com/525be26fdc4ad98beff2cf284524c140f765f54d/DEPS
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/03373426c272cfec66a1149c5aab7d25b89b9be5 commit 03373426c272cfec66a1149c5aab7d25b89b9be5 Author: Daniel Erat <derat@chromium.org> Date: Fri Jul 20 18:41:01 2018 chromeos_config: Add bvt-tast HWTest suite. Add a new synchronous hardware test suite that runs the bvt-tast Autotest suite. It currently just warns on failure and runs on -release and -tryjob builders, but not on "real" -paladin or -pfq builders. BUG= chromium:853545 TEST=inspected config dump CQ-DEPEND=I83d0c7f3c5485be91a300e5ebaa7d39313db98f4 Change-Id: Iae21d6cc1a4a03afac14da5c99805b70f909f650 Reviewed-on: https://chromium-review.googlesource.com/1111333 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/03373426c272cfec66a1149c5aab7d25b89b9be5/config/chromeos_config.py [modify] https://crrev.com/03373426c272cfec66a1149c5aab7d25b89b9be5/lib/constants.py [modify] https://crrev.com/03373426c272cfec66a1149c5aab7d25b89b9be5/config/config_dump.json
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/143fe69d61fa223577b06596cd01ee39ab3f1062 commit 143fe69d61fa223577b06596cd01ee39ab3f1062 Author: Daniel Erat <derat@chromium.org> Date: Fri Jul 20 18:41:00 2018 autotest: Add bvt-tast test suite. Add a new bvt-tast test suite consisting of tast.mustpass-{android,chrome,system} server tests that together run all non-informational, non-disabled Tast tests. BUG= chromium:853545 TEST=none CQ-DEPEND=Iae21d6cc1a4a03afac14da5c99805b70f909f650 Change-Id: I83d0c7f3c5485be91a300e5ebaa7d39313db98f4 Reviewed-on: https://chromium-review.googlesource.com/1111339 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [add] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/test_suites/control.bvt-tast [rename] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/server/site_tests/tast/control.mustpass-arc [copy] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/server/site_tests/tast/control.mustpass-system [copy] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/server/site_tests/tast/control.mustpass-chrome
,
Aug 3
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a709d73bd703ae94b375f259649fa05ab6733fb7 commit a709d73bd703ae94b375f259649fa05ab6733fb7 Author: Daniel Erat <derat@chromium.org> Date: Fri Aug 03 04:50:10 2018 chromeos_config: Add bvt-tast-{android,chrome}-pfq. Add new HWTest suites for running the bvt-tast-android-pfq and bvt-tast-chrome-pfq suites, resulting in the following configuration: - Chrome OS CQ-related builders run bvt-tast-cq to validate that the OS, Chrome, and ARC are functioning properly. - Chrome PFQ builders run bvt-tast-chrome-pfq to validate that Chrome and ARC are functioning properly. - Android PFQ builders run bvt-tast-android-pfq to validate that ARC is functioning properly. All of these suites currently run in warn-only mode. BUG= chromium:853545 TEST=inspected config_dump.json: - bvt-tast-cq runs on -release and -paladin - bvt-tast-chrome-pfq runs on -chrome-pfq - bvt-tast-android-pfq runs on -{android,chrome}-pfq CQ-DEPEND=Iae9c641f5123e3721bfecdfc4df42fd7f873b035 Change-Id: I7df46722c07f45c419c04565a03f60f45fb85b3c Reviewed-on: https://chromium-review.googlesource.com/1146146 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@google.com> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/a709d73bd703ae94b375f259649fa05ab6733fb7/config/chromeos_config.py [modify] https://crrev.com/a709d73bd703ae94b375f259649fa05ab6733fb7/lib/constants.py [modify] https://crrev.com/a709d73bd703ae94b375f259649fa05ab6733fb7/config/config_dump.json
,
Aug 3
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/feabe10404848efba69c2f9dfca7b683d694a39f commit feabe10404848efba69c2f9dfca7b683d694a39f Author: Daniel Erat <derat@chromium.org> Date: Fri Aug 03 04:50:11 2018 autotest: Add bvt-tast-android-pfq and bvt-tast-chrome-pfq. Add two new test suites for running Tast tests and rename the existing bvt-tast suite and tast.mustpass-arc test, resulting in the following configuration: - bvt-tast-cq runs on the Chrome OS CQ and executes the tast.mustpass-android, tast.mustpass-chrome, and tast.mustpass-system server tests. - bvt-tast-chrome-pfq runs on the Chrome PFQ and executes the tast.mustpass-android and tast.mustpass-chrome server tests. - bvt-tast-android-pfq runs on the Android PFQ and executes the tast.mustpass-android server test. BUG= chromium:853545 TEST=none CQ-DEPEND=I7df46722c07f45c419c04565a03f60f45fb85b3c Change-Id: Iae9c641f5123e3721bfecdfc4df42fd7f873b035 Reviewed-on: https://chromium-review.googlesource.com/1146144 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/server/site_tests/tast/control.mustpass-system [rename] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/server/site_tests/tast/control.mustpass-android [modify] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/server/site_tests/tast/control.mustpass-chrome [rename] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/test_suites/control.bvt-tast-cq [add] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/test_suites/control.bvt-tast-android-pfq [add] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/test_suites/control.bvt-tast-chrome-pfq
,
Aug 8
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/9169fb65d15caecd3fc114ffedccac416a5fa212 commit 9169fb65d15caecd3fc114ffedccac416a5fa212 Author: Daniel Erat <derat@chromium.org> Date: Wed Aug 08 21:37:16 2018
,
Aug 8
Declaring victory here, but I still need to make these suites be non-warn-only on the CQ and PFQs. |
||
►
Sign in to add a comment |
||
Comment 1 by derat@chromium.org
, Jun 20 2018