New issue
Advanced search Search tips

Issue 873556 link

Starred by 1 user
Status: Fixed
Owner:
derat@chromium.org
Closed: Aug 17
Cc:
nya@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Feature



Sign in to add a comment

Add "meta" tests for Tast

Project Member Reported by derat@chromium.org, Aug 13 
Right now, the only way to verify that changes to Tast don't break its end-to-end behavior is by manually running tests using "tast run" and verifying their output. I'm planning to add "meta" tests that themselves run the tast command and check its output.

There's a risk that a change to Tast could break it in such a way that it doesn't report these tests as failing, but I think that it's unlikely. (Also, it seems weird to write an Autotest test to verify that Tast works.)
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 14 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/e2feb63e480ee878d93933fb7c8d640c7450d9c5

commit e2feb63e480ee878d93933fb7c8d640c7450d9c5
Author: Daniel Erat <derat@chromium.org>
Date: Tue Aug 14 00:48:57 2018

tast: Update testing.NewState to take Test arg.

Make the testing.NewState function get timeouts from a
passed-in Test rather than as separate arguments. No
functional changes.

BUG= chromium:873556 
TEST=unit tests still pass and local tests still work

Change-Id: I97bfe8b1eb861c2945d9394c52d1f4b58a47fa5e
Reviewed-on: https://chromium-review.googlesource.com/1172951
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/e2feb63e480ee878d93933fb7c8d640c7450d9c5/src/chromiumos/tast/testing/test_test.go
[modify] https://crrev.com/e2feb63e480ee878d93933fb7c8d640c7450d9c5/src/chromiumos/tast/testing/state_test.go
[modify] https://crrev.com/e2feb63e480ee878d93933fb7c8d640c7450d9c5/src/chromiumos/tast/testing/state.go
[modify] https://crrev.com/e2feb63e480ee878d93933fb7c8d640c7450d9c5/src/chromiumos/tast/bundle/bundle.go
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 14 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/68154b5ee64f4609ef0821ef5a772009706c05e1

commit 68154b5ee64f4609ef0821ef5a772009706c05e1
Author: Daniel Erat <derat@chromium.org>
Date: Tue Aug 14 09:13:38 2018

tast: Add testing.Meta.

Give remote tests in the "meta" category access to a Meta
struct containing additional information describing how the
initial tast process was executed. This is needed to permit
a class of a tests that themselves run the "tast" command
and verify its output.

BUG= chromium:873556 
TEST=added unit test; also exercised with new tests

Change-Id: I171a70b9383a03ce8055ede5b7a296c1410bf991
Reviewed-on: https://chromium-review.googlesource.com/1172161
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/cmd/tast/run/remote.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/bundle/bundle.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/testing/test_test.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/testing/state.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/cmd/tast/run/remote_test.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/runner/args.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/testing/state_test.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/bundle/bundle_test.go
[modify] https://crrev.com/68154b5ee64f4609ef0821ef5a772009706c05e1/src/chromiumos/tast/bundle/args.go
Project Member

Comment 3 by bugdroid1@chromium.org, Aug 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/aaa6748fd404f3291e014bdddff6f0cfcb7acd54

commit aaa6748fd404f3291e014bdddff6f0cfcb7acd54
Author: Daniel Erat <derat@chromium.org>
Date: Wed Aug 15 11:42:33 2018

tast-tests: Add helper tests for meta tests.

Add three tests that will be called by meta tests that will
be added in a future change to perform end-to-end testing of
the tast command:

- meta.LocalFiles: reads internal and external data files
  and copies them to the output dir
- meta.LocalPanic: intentionally panics to verify that other
  tests are still run
- meta.RemoteFiles: reads a data file and copies it to an
  output dir

BUG= chromium:873556 
TEST=ran these tests via new meta tests
CQ-DEPEND=I171a70b9383a03ce8055ede5b7a296c1410bf991
CQ-DEPEND=Id9f0c42055ea1c7873e8db677c3325d0ce3a823a

Change-Id: Ia138300909874cce4c7b46a54bbdb28068fdb66c
Reviewed-on: https://chromium-review.googlesource.com/1173762
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[add] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/local/bundles/cros/meta/data/local_files_internal.txt
[modify] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/remote/bundles/cros/main.go
[add] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/remote/bundles/cros/meta/data/remote_files_data.txt
[add] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/local/bundles/cros/meta/local_files.go
[add] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/local/bundles/cros/meta/local_panic.go
[add] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/remote/bundles/cros/meta/remote_files.go
[modify] https://crrev.com/aaa6748fd404f3291e014bdddff6f0cfcb7acd54/src/chromiumos/tast/local/bundles/cros/main.go
Project Member

Comment 4 by bugdroid1@chromium.org, Aug 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/b5e2050bde25ae5107eb7581d3b721745ee1a9bd

commit b5e2050bde25ae5107eb7581d3b721745ee1a9bd
Author: Daniel Erat <derat@chromium.org>
Date: Wed Aug 15 11:42:34 2018

tast-tests: Add meta.ListTests and meta.RunTests.

Add meta tests that perform basic end-to-end verification of
the tast executable's "list" and "run" subcommands.

BUG= chromium:873556 
TEST=new tests pass

Change-Id: I419c8afee3ac3210baac067584518f46e5c6dd3f
Reviewed-on: https://chromium-review.googlesource.com/1173767
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[add] https://crrev.com/b5e2050bde25ae5107eb7581d3b721745ee1a9bd/src/chromiumos/tast/remote/bundles/cros/meta/run_tests.go
[add] https://crrev.com/b5e2050bde25ae5107eb7581d3b721745ee1a9bd/src/chromiumos/tast/remote/bundles/cros/meta/tastrun/run.go
[add] https://crrev.com/b5e2050bde25ae5107eb7581d3b721745ee1a9bd/src/chromiumos/tast/remote/bundles/cros/meta/list_tests.go
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 17 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/81f1af7fe0582cac535b0b71be54ccffcfe16f14

commit 81f1af7fe0582cac535b0b71be54ccffcfe16f14
Author: Daniel Erat <derat@chromium.org>
Date: Fri Aug 17 18:17:50 2018

tast-tests: Make meta tests non-informational.

Remove the "informational" attribute from meta.ListTests and
meta.RunTests so they'll run as part of the Tast BVT suites.

Also add comments documenting that helper tests need to be
built before building and running meta tests manually.

BUG= chromium:873556 
TEST=tests pass

Change-Id: I6391e4d7c75a6f10431832ad38436beff9045db1
Reviewed-on: https://chromium-review.googlesource.com/1177189
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/81f1af7fe0582cac535b0b71be54ccffcfe16f14/src/chromiumos/tast/remote/bundles/cros/meta/run_tests.go
[modify] https://crrev.com/81f1af7fe0582cac535b0b71be54ccffcfe16f14/src/chromiumos/tast/remote/bundles/cros/meta/list_tests.go

Comment 7 by derat@chromium.org, Aug 17

Status: Fixed (was: Started)
►

Sign in to add a comment