Support test preconditions in Tast |
||
Issue descriptionIn order to make Tast be able to run large numbers of tests quickly, I'm planning to add support for tests to state preconditions like "Chrome is logged in" or "ARC has booted". This is still a work in progress, but http://doc/1pDt_uN2BBkWekjvKI1iPnq-vD-WY2BxT3U3soJ-VRPE describes some of my initial thoughts. To start out with, I'm just cleaning up some of the code that will need to change. Issue 890546 (passing contexts explicitly into tests) was the first part of this.
,
Oct 6
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/d923692bacc3c914f0381371fbcb45288ed50943 commit d923692bacc3c914f0381371fbcb45288ed50943 Author: Daniel Erat <derat@chromium.org> Date: Sat Oct 06 02:37:57 2018 tast: Refactor Test.Run to use stages. Generalize the testing package's code for running a test into a runStages function that runs a series of functions (currently setup, test, cleanup) within a goroutine. Each function has its own timeout, which is a departure from the old model of also allocating the test's timeout to setup and cleanup. This paves the way for granting longer timeouts to preconditions in the future. The "tast run" command limits the amount of time that it waits for the next control message (to avoid hanging indefinitely if the DUT becomes unresponsive), so this change also adds a new Test.AdditionalTime field that the bundle uses to pass back extra time allocated to setup and cleanup functions. BUG=chromium:892009 TEST=added unit tests for runStages; also verified that tests still work and that "tast list -json" shows that additionalTime field is present in serialized tests Change-Id: I93f0f77159c5847a8b055db4d40b6104be9e4965 Reviewed-on: https://chromium-review.googlesource.com/1262859 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/cmd/tast/run/results_test.go [add] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/stage.go [modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/test_test.go [modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/cmd/tast/run/results.go [modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/test.go [add] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/stage_test.go
,
Oct 9
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/4ba9826b4bc456a82eebba204a5b750769437391 commit 4ba9826b4bc456a82eebba204a5b750769437391 Author: Daniel Erat <derat@chromium.org> Date: Tue Oct 09 23:26:44 2018 tast: Add outputReader for unit tests in testing package. Add a new outputReader struct that implements an infinitely-buffered chan Output. This prevents an issue that I frequently encounter when making changes to the testing package. When additional Output messages are written when running tests, unit tests hang because the number of messages that are written now exceed the buffering of the output channel. Also move shared code into a new output_test.go file. BUG=chromium:892009 TEST=unit tests still pass Change-Id: I5d51981f5539c9530906304eacc4247eb5d1c3f5 Reviewed-on: https://chromium-review.googlesource.com/1268684 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/state_test.go [modify] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/test_test.go [modify] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/stage_test.go [add] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/output_test.go
,
Oct 10
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/613225a68e7ef9ddc6678af826f9552a2e758a53 commit 613225a68e7ef9ddc6678af826f9552a2e758a53 Author: Daniel Erat <derat@chromium.org> Date: Wed Oct 10 16:45:03 2018 tast: Add support for test preconditions. Add a Pre field to the testing package's Test struct that can be used to specify a precondition that must be satisfied before the test is run. Preconditions (represented by a new testing.Precondition interface) contain setup code to get the DUT into a state like "Chrome is logged in" or "ARC has booted". Performing this setup once and then running all tests that require it can greatly speed up testing. This change also modifies the order in which tests are run: tests are grouped by precondition before being sorted lexicographically. BUG=chromium:892009 TEST=added/updated unit tests; also manually verified in conjunction with a change in tast-tests that defines and uses preconditions for logging into chrome and booting android Change-Id: I71975d15ff0d8594243065800fe952419443f22b Reviewed-on: https://chromium-review.googlesource.com/1250030 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/bundle/bundle.go [modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/testing/test_test.go [modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/testing/test.go [modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/testing/state.go [modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/bundle/bundle_test.go [modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/bundle/args.go
,
Oct 12
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast/+/feb9a9ef7c4ca1f73cabeac98cea75f573b11682 commit feb9a9ef7c4ca1f73cabeac98cea75f573b11682 Author: Daniel Erat <derat@chromium.org> Date: Fri Oct 12 04:34:42 2018 tast: Rename setup and cleanup functions. Rename the "run setup", "run cleanup", "test setup", and "test cleanup" functions to "pre-run", "post-run", "pre-test", and "post-test". The new names hopefully describe them more accurately (for example, a post-test function can be used to collect system logs after failures). Also make the post-test function run unconditionally. It previously only ran if the test function was executed. BUG=chromium:892009 TEST=unit tests pass; also verified that tests using preconditions still work Change-Id: I619aed59e566fe2b4f9463521caa06c747839cde Reviewed-on: https://chromium-review.googlesource.com/1275705 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/faillog/faillog.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/bundle.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/local.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/remote.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/testing/test_test.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/testing/state.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/testing/test.go [modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/bundle_test.go
,
Dec 14
|
||
►
Sign in to add a comment |
||
Comment 1 by bugdroid1@chromium.org
, Oct 5