Tast tests can be double-run via bvt-tast* and login_VMSanity |
||
Issue description
I think we're potentially double-running various Tast tests on some builders:
- The bvt-tast-cq suite runs all tests matched by '(!informational && !disabled)'.
- The login_VMSanity Autotest test runs vm_sanity.py in the bvt-inline and smoke suites, which runs tests matched by '(!informational && !disabled && ("dep:chrome" || "dep:chrome_login")'.
So we can run Chrome-related tests in two different things that run on (some) -paladin builders, I think.
Tast was initially added to vm_sanity.py so we could run Tast tests on the Chrome CQ. Achuith, should login_VMSanity be updated to skip running Tast tests? There's already plenty of support for running Tast tests in the Chrome OS lab (see http://go/tast-infra for details), so I don't know if it's necessary to also nest it within login_VMSanity in the lab. Maybe there's a coverage-related reason why we need this, though (e.g. to make sure that we don't pull broken tests into the Chrome CQ).
,
Sep 25
The cost of running these tests is a few seconds, so why not just run them? login_VMSanity exists to protect the chrome CQ from chromeos changes that break vm_sanity. I think there's value in running the exact same tests in both cases instead of relying on third-party tests that are expected to run.
,
Sep 25
Maybe I'm misunderstanding, but I would like a way to run Tast tests for mash in the VMs for Chrome CQ. Does this bug intend to remove that capability? Or just move the functionality out of VMSanity into somewhere else?
,
Sep 25
#3: No, that's something that I want too. :-) It's tracked by issue 885016 . #2: I understand wanting to run vm_sanity.py to protect the Chrome CQ. I think that my real question is: why do we run login_VMSanity on non-VM paladin builders? See e.g. the elm-paladin failure at issue 888884 .
,
Sep 25
As to why not to run duplicate tests: more tests mean more chances for flakiness. Chrome login fails non-deterministically in testing: issue 888520, issue 884454 , issue 880823 , issue 880818 , issue 877720 , issue 875263 , issue 865813, etc.
,
Sep 26
We run maybe a dozen tests in the smoke suite that depend on login being solid, and hundreds of unit tests in a single run of telemetry_unittests have the same requirement. In the chrome CQ, we'll run vm_sanity many hundreds of times a day. So there's an assumption that vm_sanity will not flake at all. (I don't think the recent bugs with login flakiness are indicative of the long-run stability of chrome login). We used vm_sanity as a way to quickly introduce Tast/Mash tests which seems to be causing some pain. As to only running in the VM - I didn't think the test had any subtests that are VM-only (but now it looks like Tast isn't handling some boards well?), and we'll probably have kevins in the chrome CQ at some point. I think we're probably close enough to having Tast run separately that we can look at reducing the number of Tast tests in vm_sanity so we stop seeing board-related failures.
,
Oct 11
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/9095c547cde3d096e8bf8e462d9c96df773664f0 commit 9095c547cde3d096e8bf8e462d9c96df773664f0 Author: Daniel Erat <derat@chromium.org> Date: Thu Oct 11 22:02:15 2018 autotest: Don't run Tast tests as part of login_VMSanity. Avoid running Tast tests as part of the login_VMSanity test. Doing so has the potential to make us double-run these tests on the CQ, as they're already run as part of the bvt-tast-* test suites. BUG= chromium:888887 TEST=none Change-Id: Iea312e8a5a9ad951bbe0738844e812c4c1a8c25b Reviewed-on: https://chromium-review.googlesource.com/c/1242315 Tested-by: Dan Erat <derat@chromium.org> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Trybot-Ready: Dan Erat <derat@chromium.org> Trybot-Ready: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Commit-Queue: Achuith Bhandarkar <achuith@chromium.org> [modify] https://crrev.com/9095c547cde3d096e8bf8e462d9c96df773664f0/client/site_tests/login_VMSanity/login_VMSanity.py
,
Oct 23
|
||
►
Sign in to add a comment |
||
Comment 1 by derat@chromium.org
, Sep 25Status: Started (was: Assigned)