New issue
Advanced search Search tips

Issue 892009 link

Starred by 1 user

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Feature

Blocked on:
issue 915093



Sign in to add a comment

Support test preconditions in Tast

Project Member Reported by derat@chromium.org, Oct 4

Issue description

In order to make Tast be able to run large numbers of tests quickly, I'm planning to add support for tests to state preconditions like "Chrome is logged in" or "ARC has booted". This is still a work in progress, but http://doc/1pDt_uN2BBkWekjvKI1iPnq-vD-WY2BxT3U3soJ-VRPE describes some of my initial thoughts.

To start out with, I'm just cleaning up some of the code that will need to change.  Issue 890546  (passing contexts explicitly into tests) was the first part of this.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Oct 5

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/b28101d5bb66c409b40b9e3dfc637883bdfcbe83

commit b28101d5bb66c409b40b9e3dfc637883bdfcbe83
Author: Daniel Erat <derat@chromium.org>
Date: Fri Oct 05 18:37:40 2018

tast: Refactor test timeout code.

Replace the testing package's createContexts function with
smaller and simpler extendTimeout and timeoutContext
functions. No functional changes are intended here, but I'm
trying to simplify this code to make it easier to introduce
additional contexts with their own timeouts later.

BUG=chromium:892009
TEST=existing unit tests pass and ui.ChromeLogin still works

Change-Id: I7ce747e10ff7e8dc06720acd2e88c6e8c05ae2dc
Reviewed-on: https://chromium-review.googlesource.com/1260626
Commit-Ready: Shuhei Takahashi <nya@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>

[modify] https://crrev.com/b28101d5bb66c409b40b9e3dfc637883bdfcbe83/src/chromiumos/tast/testing/test.go

Project Member

Comment 2 by bugdroid1@chromium.org, Oct 6

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/d923692bacc3c914f0381371fbcb45288ed50943

commit d923692bacc3c914f0381371fbcb45288ed50943
Author: Daniel Erat <derat@chromium.org>
Date: Sat Oct 06 02:37:57 2018

tast: Refactor Test.Run to use stages.

Generalize the testing package's code for running a test
into a runStages function that runs a series of functions
(currently setup, test, cleanup) within a goroutine. Each
function has its own timeout, which is a departure from the
old model of also allocating the test's timeout to setup and
cleanup. This paves the way for granting longer timeouts to
preconditions in the future.

The "tast run" command limits the amount of time that it
waits for the next control message (to avoid hanging
indefinitely if the DUT becomes unresponsive), so this
change also adds a new Test.AdditionalTime field that the
bundle uses to pass back extra time allocated to setup and
cleanup functions.

BUG=chromium:892009
TEST=added unit tests for runStages; also verified that
     tests still work and that "tast list -json" shows that
     additionalTime field is present in serialized tests

Change-Id: I93f0f77159c5847a8b055db4d40b6104be9e4965
Reviewed-on: https://chromium-review.googlesource.com/1262859
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/cmd/tast/run/results_test.go
[add] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/stage.go
[modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/test_test.go
[modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/cmd/tast/run/results.go
[modify] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/test.go
[add] https://crrev.com/d923692bacc3c914f0381371fbcb45288ed50943/src/chromiumos/tast/testing/stage_test.go

Project Member

Comment 3 by bugdroid1@chromium.org, Oct 9

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/4ba9826b4bc456a82eebba204a5b750769437391

commit 4ba9826b4bc456a82eebba204a5b750769437391
Author: Daniel Erat <derat@chromium.org>
Date: Tue Oct 09 23:26:44 2018

tast: Add outputReader for unit tests in testing package.

Add a new outputReader struct that implements an
infinitely-buffered chan Output.

This prevents an issue that I frequently encounter when
making changes to the testing package. When additional
Output messages are written when running tests, unit tests
hang because the number of messages that are written now
exceed the buffering of the output channel.

Also move shared code into a new output_test.go file.

BUG=chromium:892009
TEST=unit tests still pass

Change-Id: I5d51981f5539c9530906304eacc4247eb5d1c3f5
Reviewed-on: https://chromium-review.googlesource.com/1268684
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/state_test.go
[modify] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/test_test.go
[modify] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/stage_test.go
[add] https://crrev.com/4ba9826b4bc456a82eebba204a5b750769437391/src/chromiumos/tast/testing/output_test.go

Project Member

Comment 4 by bugdroid1@chromium.org, Oct 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/613225a68e7ef9ddc6678af826f9552a2e758a53

commit 613225a68e7ef9ddc6678af826f9552a2e758a53
Author: Daniel Erat <derat@chromium.org>
Date: Wed Oct 10 16:45:03 2018

tast: Add support for test preconditions.

Add a Pre field to the testing package's Test struct that
can be used to specify a precondition that must be satisfied
before the test is run.

Preconditions (represented by a new testing.Precondition
interface) contain setup code to get the DUT into a state
like "Chrome is logged in" or "ARC has booted". Performing
this setup once and then running all tests that require it
can greatly speed up testing.

This change also modifies the order in which tests are run:
tests are grouped by precondition before being sorted
lexicographically.

BUG=chromium:892009
TEST=added/updated unit tests; also manually verified in
     conjunction with a change in tast-tests that defines
     and uses preconditions for logging into chrome and
     booting android

Change-Id: I71975d15ff0d8594243065800fe952419443f22b
Reviewed-on: https://chromium-review.googlesource.com/1250030
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/bundle/bundle.go
[modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/testing/test_test.go
[modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/testing/test.go
[modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/testing/state.go
[modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/bundle/bundle_test.go
[modify] https://crrev.com/613225a68e7ef9ddc6678af826f9552a2e758a53/src/chromiumos/tast/bundle/args.go

Project Member

Comment 5 by bugdroid1@chromium.org, Oct 12

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/feb9a9ef7c4ca1f73cabeac98cea75f573b11682

commit feb9a9ef7c4ca1f73cabeac98cea75f573b11682
Author: Daniel Erat <derat@chromium.org>
Date: Fri Oct 12 04:34:42 2018

tast: Rename setup and cleanup functions.

Rename the "run setup", "run cleanup", "test setup", and
"test cleanup" functions to "pre-run", "post-run",
"pre-test", and "post-test". The new names hopefully
describe them more accurately (for example, a post-test
function can be used to collect system logs after failures).

Also make the post-test function run unconditionally. It
previously only ran if the test function was executed.

BUG=chromium:892009
TEST=unit tests pass; also verified that tests using
     preconditions still work

Change-Id: I619aed59e566fe2b4f9463521caa06c747839cde
Reviewed-on: https://chromium-review.googlesource.com/1275705
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/faillog/faillog.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/bundle.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/local.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/remote.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/testing/test_test.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/testing/state.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/testing/test.go
[modify] https://crrev.com/feb9a9ef7c4ca1f73cabeac98cea75f573b11682/src/chromiumos/tast/bundle/bundle_test.go

Blockedon: 915093

Sign in to add a comment