New issue
Advanced search Search tips

Issue 853545 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 8
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug


Show other hotlists

Hotlists containing this issue:
Hotlist-Tast


Sign in to add a comment

Rework Tast test suite composition

Project Member Reported by derat@chromium.org, Jun 17 2018

Issue description

Right now, the tast.bvt Autotest server test runs all tests with the "bvt" attribute. Per http://doc/1ZyozTa907IJqw7QJNHaWIQ07v0szpUOzdLG2K6cURII, I'm planning to instead introduce "informational" and "disabled" attributes and use them to divide tests into two suites ("(!informational && !disabled)" and "(informational && !disabled)").

The way that these suites will be assigned to Autotest suites is still under discussion.
 

Comment 1 by derat@chromium.org, Jun 20 2018

Status: Started (was: Assigned)
Project Member

Comment 2 by bugdroid1@chromium.org, Jun 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/c33e95eb676826b4828ef79a15c71b6461692014

commit c33e95eb676826b4828ef79a15c71b6461692014
Author: Daniel Erat <derat@chromium.org>
Date: Wed Jun 20 01:02:53 2018

tast: Add disabled and informational to test_attributes.md.

Document the new "disabled" and "informational" test
attributes and remove the old "bvt" attribute.

BUG= chromium:853545 
TEST=viewed with md_browser

Change-Id: I5c185031ea5518744f7a9265a0c60c298febded4
Reviewed-on: https://chromium-review.googlesource.com/1107267
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/c33e95eb676826b4828ef79a15c71b6461692014/docs/test_attributes.md

Project Member

Comment 3 by bugdroid1@chromium.org, Jun 20 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/41451552f2eae83fe55c317a274145b7d911ff15

commit 41451552f2eae83fe55c317a274145b7d911ff15
Author: Daniel Erat <derat@chromium.org>
Date: Wed Jun 20 05:04:00 2018

Project Member

Comment 4 by bugdroid1@chromium.org, Jun 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f98fe1901466b4239fb46911754fb453eb238d7f

commit f98fe1901466b4239fb46911754fb453eb238d7f
Author: Daniel Erat <derat@chromium.org>
Date: Thu Jun 21 15:14:23 2018

tast-tests: Add "informational" and "disabled" attributes.

Add new test attributes that replace the existing "bvt"
attribute:

- Tests with none of the new labels must pass.
- Tests with "informational" run in bvt-perbuild.
- Tests with "disabled" aren't run automatically.

BUG= chromium:853545 
TEST=none
CQ-DEPEND=I3abb61d2999c22ed110d9e6ec1d405333d2051ea

Change-Id: Ice0d28ad5c27ca019667f5f2baba43c986ec57d9
Reviewed-on: https://chromium-review.googlesource.com/1105128
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_logged_in.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/arc/intent_forward.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/chrome_login.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/security/log_perms.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/remote/bundles/cros/example/reconnect_to_dut.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/vm/start_termina_vm.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/dbus.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/platform/check_processes.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/power/check_status.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/chrome_crash_not_logged_in.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/pass.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/arc_startup.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/fail.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/example/data_files.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/local/bundles/cros/ui/mash_login.go
[modify] https://crrev.com/f98fe1901466b4239fb46911754fb453eb238d7f/src/chromiumos/tast/remote/bundles/cros/power/reboot.go

Project Member

Comment 5 by bugdroid1@chromium.org, Jun 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c527bcc4ce659e27c76d427a6f7844c4e5617bc8

commit c527bcc4ce659e27c76d427a6f7844c4e5617bc8
Author: Daniel Erat <derat@chromium.org>
Date: Thu Jun 21 15:14:22 2018

autotest: Add tast.informational and tast.must_pass.

Add control files for new tast.informational and
tast.must_pass server tests and remove the existing tast.bvt
test. Both of the new tests run in bvt-perbuild initially.

BUG= chromium:853545 
TEST=none
CQ-DEPEND=Ice0d28ad5c27ca019667f5f2baba43c986ec57d9

Change-Id: I3abb61d2999c22ed110d9e6ec1d405333d2051ea
Reviewed-on: https://chromium-review.googlesource.com/1105132
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[copy] https://crrev.com/c527bcc4ce659e27c76d427a6f7844c4e5617bc8/server/site_tests/tast/control.must_pass
[rename] https://crrev.com/c527bcc4ce659e27c76d427a6f7844c4e5617bc8/server/site_tests/tast/control.informational

Project Member

Comment 6 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/15a63a040c9d96903acdc526f3c10a3311b3cfdb

commit 15a63a040c9d96903acdc526f3c10a3311b3cfdb
Author: Daniel Erat <derat@chromium.org>
Date: Fri Jun 22 13:00:17 2018

autotest: Add debug links to tast.{informational,must_pass}.

Update the control files for the tast.informational and
tast.must_pass to point to http://go/tast-failures.

BUG= chromium:853545 
TEST=none

Change-Id: If44af2b7071d469ca4b56cf76ab24ec988b2bbf8
Reviewed-on: https://chromium-review.googlesource.com/1110938
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/15a63a040c9d96903acdc526f3c10a3311b3cfdb/server/site_tests/tast/control.must_pass
[modify] https://crrev.com/15a63a040c9d96903acdc526f3c10a3311b3cfdb/server/site_tests/tast/control.informational

Project Member

Comment 7 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/eb9079ec8d3d2a8fe82008ccd364feab2ef03415

commit eb9079ec8d3d2a8fe82008ccd364feab2ef03415
Author: Daniel Erat <derat@chromium.org>
Date: Fri Jun 29 21:16:38 2018

chromeos_config: Update attribute expression for TastVMTest.

Change the test attribute expression for betty-release's
TastVMTest stage to match all non-disabled tests. It
formerly matched all tests with 'bvt' or 'canary'
attributes, neither of which exist anymore.

BUG= chromium:853545 
TEST=ran a tryjob

Change-Id: I405a5557817169aa2dbcbc3610b2cc667cd67b24
Reviewed-on: https://chromium-review.googlesource.com/1120429
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/eb9079ec8d3d2a8fe82008ccd364feab2ef03415/config/chromeos_config.py
[modify] https://crrev.com/eb9079ec8d3d2a8fe82008ccd364feab2ef03415/lib/config_lib_unittest.py
[modify] https://crrev.com/eb9079ec8d3d2a8fe82008ccd364feab2ef03415/config/config_dump.json

Project Member

Comment 8 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/525be26fdc4ad98beff2cf284524c140f765f54d

commit 525be26fdc4ad98beff2cf284524c140f765f54d
Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 29 22:39:39 2018

Roll src/third_party/chromite f028e346c07f..1bf75a651fe5 (3 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/f028e346c07f..1bf75a651fe5


git log f028e346c07f..1bf75a651fe5 --date=short --no-merges --format='%ad %ae %s'
2018-06-29 xixuan@chromium.org report_stage: Add abortSkylabHWTestStage.
2018-06-29 derat@chromium.org chromeos_config: Update attribute expression for TastVMTest.
2018-06-29 vadimsh@chromium.org cipd: Bump cipd client version to the one that supports v2 protocol.


Created with:
  gclient setdep -r src/third_party/chromite@1bf75a651fe5

The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:845314 , chromium:853545 ,chromium:801423
TBR=chrome-os-gardeners@chromium.org

Change-Id: I132c3b105686a4bcf8573121cdad6e7ba6246b59
Reviewed-on: https://chromium-review.googlesource.com/1121217
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#571685}
[modify] https://crrev.com/525be26fdc4ad98beff2cf284524c140f765f54d/DEPS

Project Member

Comment 9 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/03373426c272cfec66a1149c5aab7d25b89b9be5

commit 03373426c272cfec66a1149c5aab7d25b89b9be5
Author: Daniel Erat <derat@chromium.org>
Date: Fri Jul 20 18:41:01 2018

chromeos_config: Add bvt-tast HWTest suite.

Add a new synchronous hardware test suite that runs the
bvt-tast Autotest suite. It currently just warns on failure
and runs on -release and -tryjob builders, but not on "real"
-paladin or -pfq builders.

BUG= chromium:853545 
TEST=inspected config dump
CQ-DEPEND=I83d0c7f3c5485be91a300e5ebaa7d39313db98f4

Change-Id: Iae21d6cc1a4a03afac14da5c99805b70f909f650
Reviewed-on: https://chromium-review.googlesource.com/1111333
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/03373426c272cfec66a1149c5aab7d25b89b9be5/config/chromeos_config.py
[modify] https://crrev.com/03373426c272cfec66a1149c5aab7d25b89b9be5/lib/constants.py
[modify] https://crrev.com/03373426c272cfec66a1149c5aab7d25b89b9be5/config/config_dump.json

Project Member

Comment 10 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/143fe69d61fa223577b06596cd01ee39ab3f1062

commit 143fe69d61fa223577b06596cd01ee39ab3f1062
Author: Daniel Erat <derat@chromium.org>
Date: Fri Jul 20 18:41:00 2018

autotest: Add bvt-tast test suite.

Add a new bvt-tast test suite consisting of
tast.mustpass-{android,chrome,system} server tests that
together run all non-informational, non-disabled Tast tests.

BUG= chromium:853545 
TEST=none
CQ-DEPEND=Iae21d6cc1a4a03afac14da5c99805b70f909f650

Change-Id: I83d0c7f3c5485be91a300e5ebaa7d39313db98f4
Reviewed-on: https://chromium-review.googlesource.com/1111339
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[add] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/test_suites/control.bvt-tast
[rename] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/server/site_tests/tast/control.mustpass-arc
[copy] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/server/site_tests/tast/control.mustpass-system
[copy] https://crrev.com/143fe69d61fa223577b06596cd01ee39ab3f1062/server/site_tests/tast/control.mustpass-chrome

Project Member

Comment 11 by bugdroid1@chromium.org, Aug 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a709d73bd703ae94b375f259649fa05ab6733fb7

commit a709d73bd703ae94b375f259649fa05ab6733fb7
Author: Daniel Erat <derat@chromium.org>
Date: Fri Aug 03 04:50:10 2018

chromeos_config: Add bvt-tast-{android,chrome}-pfq.

Add new HWTest suites for running the bvt-tast-android-pfq
and bvt-tast-chrome-pfq suites, resulting in the following
configuration:

- Chrome OS CQ-related builders run bvt-tast-cq to validate
  that the OS, Chrome, and ARC are functioning properly.
- Chrome PFQ builders run bvt-tast-chrome-pfq to validate
  that Chrome and ARC are functioning properly.
- Android PFQ builders run bvt-tast-android-pfq to validate
  that ARC is functioning properly.

All of these suites currently run in warn-only mode.

BUG= chromium:853545 
TEST=inspected config_dump.json:
     - bvt-tast-cq runs on -release and -paladin
     - bvt-tast-chrome-pfq runs on -chrome-pfq
     - bvt-tast-android-pfq runs on -{android,chrome}-pfq
CQ-DEPEND=Iae9c641f5123e3721bfecdfc4df42fd7f873b035

Change-Id: I7df46722c07f45c419c04565a03f60f45fb85b3c
Reviewed-on: https://chromium-review.googlesource.com/1146146
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/a709d73bd703ae94b375f259649fa05ab6733fb7/config/chromeos_config.py
[modify] https://crrev.com/a709d73bd703ae94b375f259649fa05ab6733fb7/lib/constants.py
[modify] https://crrev.com/a709d73bd703ae94b375f259649fa05ab6733fb7/config/config_dump.json

Project Member

Comment 12 by bugdroid1@chromium.org, Aug 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/feabe10404848efba69c2f9dfca7b683d694a39f

commit feabe10404848efba69c2f9dfca7b683d694a39f
Author: Daniel Erat <derat@chromium.org>
Date: Fri Aug 03 04:50:11 2018

autotest: Add bvt-tast-android-pfq and bvt-tast-chrome-pfq.

Add two new test suites for running Tast tests and rename
the existing bvt-tast suite and tast.mustpass-arc test,
resulting in the following configuration:

- bvt-tast-cq runs on the Chrome OS CQ and executes the
  tast.mustpass-android, tast.mustpass-chrome, and
  tast.mustpass-system server tests.
- bvt-tast-chrome-pfq runs on the Chrome PFQ and executes
  the tast.mustpass-android and tast.mustpass-chrome server
  tests.
- bvt-tast-android-pfq runs on the Android PFQ and executes
  the tast.mustpass-android server test.

BUG= chromium:853545 
TEST=none
CQ-DEPEND=I7df46722c07f45c419c04565a03f60f45fb85b3c

Change-Id: Iae9c641f5123e3721bfecdfc4df42fd7f873b035
Reviewed-on: https://chromium-review.googlesource.com/1146144
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/server/site_tests/tast/control.mustpass-system
[rename] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/server/site_tests/tast/control.mustpass-android
[modify] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/server/site_tests/tast/control.mustpass-chrome
[rename] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/test_suites/control.bvt-tast-cq
[add] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/test_suites/control.bvt-tast-android-pfq
[add] https://crrev.com/feabe10404848efba69c2f9dfca7b683d694a39f/test_suites/control.bvt-tast-chrome-pfq

Project Member

Comment 13 by bugdroid1@chromium.org, Aug 8

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/9169fb65d15caecd3fc114ffedccac416a5fa212

commit 9169fb65d15caecd3fc114ffedccac416a5fa212
Author: Daniel Erat <derat@chromium.org>
Date: Wed Aug 08 21:37:16 2018

Status: Fixed (was: Started)
Declaring victory here, but I still need to make these suites be non-warn-only on the CQ and PFQs.

Sign in to add a comment