Deprecate bvt-arc and move arc tests into bvt-inline and bvt-cq |
|||
Issue descriptionAs ARC++ has become more and more a core feature of Chrome OS, the testing of it becomes more and more critical. At this point we may want to refactor our suite organization to reflect this reality, going down to only two suites, one that is fast and can run in parallel with a builder (bvt-inline) and one that is slower and can be run asynchronously (bvt-cq). It appears that ARC++ tests no longer need to be isolated into bvt-arc, as we have ARC++ tests in bvt-inline now (see https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1265155 ). This proposal is to continue this paradigm through the rest of the ARC++ tests. This would be a few steps. 1. Distribute bvt-arc tests into bvt-cq and bvt-inline based on run time. 2. Empty bvt-arc suite from autotests. 3. Stop execution of bvt-arc suite within chromite. 4. Remove bvt-arc from dashboards (e.g. Goldeneye). Thoughts?
,
Oct 5
The reason we have suite:bvt-arc is that the ARC++ PFQ only has to run suite:bvt-arc. Once you distribute bvt-arc over the other suites now you have to run all of remaining bvt-* on the ARC++ PFQ. This means extra load on the ARC++ constables and DUTs in the lab. Instead of the proposal here we should sort bvt-cq and bvt-inline to only have Chrome and kernel tests, say bvt-chrome and bvt-system. That would allow us to skip running kernel tests on the Chrome PFQ and reduce the flakes Chrome gardeners have to deal with. Indeed tast does exactly that by design and I am sure Dan can post a link to a doc with details (could not find it atm).
,
Oct 5
Issue 891467 has more about bvt-cq being async. Perhaps it's off-topic here, but I think that there's has been (and will continue to be) constant developer confusion if test failures in a suite named "bvt-cq" don't actually make CQ runs fail (at least, that's my understanding of what sync/async means here -- please let me know if I'm misunderstanding). http://go/tast-infra has details about where and how Tast tests are run. The basic idea is that tests specify dependencies (Chrome, ARC, etc.) and whether they're "informational" or "important", and we decide where they should run based on that.
,
Oct 5
Regarding comment 2, how certain are we that the tests are really isolated? My thinking here is that we want to run all the tests in all the places CLs can get in, as it is hard to be certain that there would not be any side effects. In particular I fear that just bvt-arc in the ARC PFQ may be a hole in the mechanism if a container rev inadvertently breaks some test that is in the Chrome PFQ. I can understand the hesitation in adding tests if they are flaky, but we need to be careful we don't optimize out coverage too much. I am also fine renaming the suites to something more logical, or maybe do this as part of migrating to tast?
,
Nov 27
|
|||
►
Sign in to add a comment |
|||
Comment 1 by dgarr...@chromium.org
, Oct 5