Make cbuildbot support running Tast-based tests on VMs |
|||||
Issue descriptionI'm working on adding a TastVMTestStage class to cbuildbot to support running Tast-based integration tests on virtual machines, along with the associated scaffolding to support this in builder configs. There's some discussion here: http://doc/1SjaUE-v3ftojad1CGSKm5SGRvJeLlmfwf6RH9RseeQo/edit?ts=59d3ccee#heading=h.8xlctge3rxeo
,
Oct 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357 commit 3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357 Author: Daniel Erat <derat@chromium.org> Date: Wed Oct 11 07:57:40 2017 chromite: Add TastVMTestStage. Add a tast_test_stages module containing a new cbuildbot test stage named TastVMTestStage. Also add config_lib.TastVMTestConfig. BUG= chromium:772053 TEST=added unit tests for new stage Change-Id: I01ab4448dd30bc152082573e66c04a8fb5f6116d Reviewed-on: https://chromium-review.googlesource.com/703922 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> Reviewed-by: David James <davidjames@chromium.org> [add] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/cbuildbot/stages/tast_test_stages.py [add] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/cbuildbot/stages/tast_test_stages_unittest.py [modify] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/lib/config_lib.py [modify] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/cbuildbot/config_dump.json [modify] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/cbuildbot/builders/simple_builders.py [modify] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/lib/config_lib_unittest.py [modify] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/lib/constants.py [add] https://crrev.com/3cc30d9e8f2b40e498c37f2469aa1e7eb58c6357/cbuildbot/stages/tast_test_stages_unittest
,
Oct 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc commit cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc Author: Daniel Erat <derat@chromium.org> Date: Fri Oct 20 17:44:25 2017 chromite: Run Tast VM tests on lakitu release builder. Add a tast_vm_canary suite running tests matched by "(bvt || canary) && !arc" to the lakitu-release builder. Also lower TastVMTestStage's default timeout to thirty minutes, make failures be forgiven, and fix an invalid CWD that was used when running cros_run_tast_vm_test. BUG= chromium:772053 TEST=none CQ-DEPEND=CL:729119 Change-Id: If8e88913cf0cf600a6835d971a4d1bace66dbfbb Reviewed-on: https://chromium-review.googlesource.com/729243 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc/cbuildbot/config_dump.json [modify] https://crrev.com/cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc/cbuildbot/stages/tast_test_stages.py [modify] https://crrev.com/cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc/cbuildbot/stages/tast_test_stages_unittest.py [modify] https://crrev.com/cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc/lib/config_lib.py [modify] https://crrev.com/cff8aa1c123cc9d31f2abd1cd250b6ac034a59bc/cbuildbot/chromeos_config.py
,
Oct 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/f7d0b187d529a2e1b07e857a6bf2a6b51e44b1cf commit f7d0b187d529a2e1b07e857a6bf2a6b51e44b1cf Author: Daniel Erat <derat@chromium.org> Date: Sat Oct 21 09:07:11 2017 chromite: Move Tast VM tests to betty-release. Move experimental Tast VM tests from lakitu-release to betty-release. lakitu doesn't include Chrome, so it isn't a good choice for trying this out (as most of the tests so far exercise Chrome). BUG= chromium:772053 TEST=ran betty-release tryjob; test stage passed Change-Id: If12f754fe10c68e345b8a57813a68e077db11676 Reviewed-on: https://chromium-review.googlesource.com/730848 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/f7d0b187d529a2e1b07e857a6bf2a6b51e44b1cf/cbuildbot/config_dump.json [modify] https://crrev.com/f7d0b187d529a2e1b07e857a6bf2a6b51e44b1cf/cbuildbot/chromeos_config.py
,
Oct 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/b4c5b52e960b1b8c67dcb8c495d8175d1a39d147 commit b4c5b52e960b1b8c67dcb8c495d8175d1a39d147 Author: Daniel Erat <derat@chromium.org> Date: Tue Oct 24 05:39:05 2017 chromite: Improve TastVMTestStage error handling. Make various improvements to TastVMTestStage's eror-handling code: - Report failure if results.json includes failed tests. - Report failure if any suites' results files are missing. - Report failed tests with "flaky" attributes but don't fail the stage on their behalf. - Make archiving errors not overwrite earlier testing errors. Also add more unit tests and make tests verify stage results (including error messages). BUG= chromium:772053 TEST=added unit tests; also ran betty-release try jobs with always-failing tests (both marked flaky and not) Change-Id: I9646ce1373abde40537811f68eaa396130baf78c Reviewed-on: https://chromium-review.googlesource.com/732329 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/b4c5b52e960b1b8c67dcb8c495d8175d1a39d147/cbuildbot/stages/tast_test_stages.py [modify] https://crrev.com/b4c5b52e960b1b8c67dcb8c495d8175d1a39d147/cbuildbot/stages/tast_test_stages_unittest.py
,
Oct 31 2017
This is running on betty-release now. Issue 779267 tracked a VM testing parallelization issue.
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c1d2280a6cf174f8bfbec440e7feb7a78b19b4f7 commit c1d2280a6cf174f8bfbec440e7feb7a78b19b4f7 Author: Daniel Erat <derat@chromium.org> Date: Thu Nov 09 08:38:14 2017 chromite: Run ARC Tast VM tests on betty-release. Update the test expression used for the tast_vm_canary suite to not exclude ARC tests. Also make TastVMTestConfig raise TypeError if it receives a a non-list object as its test expression. BUG= chromium:772053 TEST=ran betty-release tryjob; TastVMTest succeeded: http://uberchromegw/i/chromiumos.tryserver/builders/release/builds/17086 Change-Id: I92361c6c3d491c791a66d793c8a2dc33abbf1b04 Reviewed-on: https://chromium-review.googlesource.com/759109 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/c1d2280a6cf174f8bfbec440e7feb7a78b19b4f7/cbuildbot/config_dump.json [modify] https://crrev.com/c1d2280a6cf174f8bfbec440e7feb7a78b19b4f7/lib/config_lib.py [modify] https://crrev.com/c1d2280a6cf174f8bfbec440e7feb7a78b19b4f7/cbuildbot/chromeos_config.py
,
Nov 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a3befd978d103b5e3caf7c8dede7fa8df88153a2 commit a3befd978d103b5e3caf7c8dede7fa8df88153a2 Author: Daniel Erat <derat@chromium.org> Date: Fri Nov 10 05:45:57 2017 chromite: Make TastVMTestStage unforgiving. Make TastVMTestStage stop inheriting from generic_stages.ForgivingBuilderStage. It's been passing consistently on betty-release. BUG= chromium:772053 TEST=updated unit tests; also ran a betty-release tryjob: http://uberchromegw/i/chromiumos.tryserver/builders/release/builds/17132 Change-Id: I57ff1c3990b9a260f0b4fb0d037f93803b6fc961 Reviewed-on: https://chromium-review.googlesource.com/760719 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: David James <davidjames@chromium.org> [modify] https://crrev.com/a3befd978d103b5e3caf7c8dede7fa8df88153a2/cbuildbot/stages/tast_test_stages.py [modify] https://crrev.com/a3befd978d103b5e3caf7c8dede7fa8df88153a2/cbuildbot/stages/tast_test_stages_unittest.py
,
Jan 22 2018
,
Jan 23 2018
,
Jan 23 2018
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by bugdroid1@chromium.org
, Oct 6 2017