I'm about to upload a change to improve the flags situation for building and running external bundles.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/38db98f54646e370c15fd519d634fab4e5ef8f19

commit 38db98f54646e370c15fd519d634fab4e5ef8f19
Author: Daniel Erat <derat@chromium.org>
Date: Sat Sep 08 01:37:02 2018

tast: Rework and simplify build flags.

Try to improve the way that build parameters are passed to
the tast command. Previously, there was no way to specify an
additional location containing the test bundle code while
still including src/platform/tast-tests (which contains
packages like chrome and arc that will likely be necessary
to many non-public tests).

Remove the build package's flags (-buildarch, -buildoutdir,
-buildsysdir, -buildtestdir, -buildcommondir) and replace
build.Config's TestWorkspace, CommonWorkspace, and SysGopath
fields with a single Workspaces field that callers can set
to provide the location of the source code to be built.

Define -buildworkspace and -buildoutdir flags in the run
package instead. -buildworkspace can be used to specify an
additional workspace that should be used when building test
bundles.

Also give the build subcommand its own -arch and -workspaces
flags.

BUG=chromium:875141
TEST=verified that -build=true still works for local and
     remote tests. -buildlocaltestrunner still works. also
     created a new workspace containing an "ext" test bundle
     and was able to build and run it with the following:
     tast run -build=true -buildbundle=ext \
       -buildworkspace=/tmp/workspace \
       -checkbuilddeps=false <dut>

Change-Id: I6ef0373e222d34a2c3a255f79ee4a40ec115a743
Reviewed-on: https://chromium-review.googlesource.com/1205511
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[rename] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/paths.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/list_cmd.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run/remote_test.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run/remote.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run/config_test.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run_cmd.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run/local.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/build_cmd.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/build/config.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/build/build_test.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run/config.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run/local_test.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/build/build.go
[modify] https://crrev.com/38db98f54646e370c15fd519d634fab4e5ef8f19/src/chromiumos/cmd/tast/run_cmd_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/f5547b46863857f36c201eefc0c5dce7d039daf1

commit f5547b46863857f36c201eefc0c5dce7d039daf1
Author: Shuhei Takahashi <nya@chromium.org>
Date: Fri Sep 21 07:51:24 2018

tast: Prepare for non-cros bundles.

Make the default values of -buildworkspace and -overlaydir
derived from -buildbundle. This will allow building a non-standard
bundle only with -buildbundle.

BUG=chromium:875141
TEST=fast_build.sh -T
TEST=fast_build.sh && ~/go/bin/tast run DUT example.DataFiles

Change-Id: Ib722a92137b0e31bd62581b7abda014469810ddc
Reviewed-on: https://chromium-review.googlesource.com/1179722
Commit-Ready: Shuhei Takahashi <nya@chromium.org>
Tested-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/f5547b46863857f36c201eefc0c5dce7d039daf1/src/chromiumos/cmd/tast/list_cmd.go
[modify] https://crrev.com/f5547b46863857f36c201eefc0c5dce7d039daf1/src/chromiumos/cmd/tast/run/config.go
[add] https://crrev.com/f5547b46863857f36c201eefc0c5dce7d039daf1/src/chromiumos/cmd/tast/run/bundle.go
[modify] https://crrev.com/f5547b46863857f36c201eefc0c5dce7d039daf1/src/chromiumos/cmd/tast/run/config_test.go
[modify] https://crrev.com/f5547b46863857f36c201eefc0c5dce7d039daf1/src/chromiumos/cmd/tast/run_cmd.go

Question:

What do we want to name the private bundle?

Public bundle is named "cros". Usually we distingush public/private by Chromium OS v. Chrome OS, but cros fits both.

- crosprivate: Too long?
- crospriv: Okay but not fancy
- arc: If we create ARC-only bundle this might be suitable, but we're not sure yet
- cheets: Alias of arc

Do you have any idea?

Random idea;
- crosinternal: reflecting "internal" is also used in cros to refer private things.

Or, how about using one deeper package?
bundle/cros/private
bundle/cros/internal
bundle/{private,internal}/cros
etc.?

crosinternal (or crosint?) sounds a good candidate!

Bundle name is not just a problem about Go package name, but also about Portage package name. We today have chromeos-base/tast-local-tests-cros for the public bundle. Making bundle name hierarchical will complicate things, unfortunately.

"crosint" sounds good to me too. I wouldn't recommend nesting the packages.

I wonder if we should name the repository tast-tests-int or tast-tests-internal to match, instead of tast-tests-private.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/715aca19516329f90f133bb3d30bbb9bfc0ecfb2

commit 715aca19516329f90f133bb3d30bbb9bfc0ecfb2
Author: Shuhei Takahashi <nya@chromium.org>
Date: Wed Oct 31 15:17:29 2018

tast: Register the new crosint bundle.

Register crosint bundle to the known bundle list. Also,
tast-tests-private repo is added to fast_build.sh.

BUG=chromium:875141
TEST=fast_build.sh -T
TEST=~/go/bin/tast run -buildbundle crosint DUT arcapp.LINPACK

Change-Id: I542578d8c5dfff75a95464e2504f245cc65241fb
Reviewed-on: https://chromium-review.googlesource.com/1307198
Commit-Ready: Shuhei Takahashi <nya@chromium.org>
Tested-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>

[modify] https://crrev.com/715aca19516329f90f133bb3d30bbb9bfc0ecfb2/src/chromiumos/cmd/tast/run/bundle.go
[modify] https://crrev.com/715aca19516329f90f133bb3d30bbb9bfc0ecfb2/fast_build.sh

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/05c8ac36a2d67c19e76acfe5775b05b2c8dad8b7

commit 05c8ac36a2d67c19e76acfe5775b05b2c8dad8b7
Author: Shuhei Takahashi <nya@chromium.org>
Date: Wed Oct 31 15:17:31 2018

tast-tests: Move waitUntilReady to a shared package.

This function will be called from other local bundles too.

BUG=chromium:875141
TEST=tast run DUT example.Pass

Change-Id: I120e8e59bc6c2dfbb4c81f0d539cee7d034170b6
Reviewed-on: https://chromium-review.googlesource.com/1307177
Commit-Ready: Shuhei Takahashi <nya@chromium.org>
Tested-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>

[add] https://crrev.com/05c8ac36a2d67c19e76acfe5775b05b2c8dad8b7/src/chromiumos/tast/local/ready/ready.go
[modify] https://crrev.com/05c8ac36a2d67c19e76acfe5775b05b2c8dad8b7/src/chromiumos/tast/local/bundles/cros/main.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/c23c1a26bf6aca1a8bb49452f62a6141c8ef52ed

commit c23c1a26bf6aca1a8bb49452f62a6141c8ef52ed
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Fri Nov 02 11:14:23 2018

tast: Fix fast_build.sh.

We're adding new tast-test-private repository, but not completed.
So, there's no tast-test-private/src dir, and fast_build.sh fails.
This is the fix.

BUG=chromium:875141
TEST=Ran ./fast_build.sh -T.

Change-Id: I24eddff9df0c4fda1cca7e667cfe4900bf3e8fba
Reviewed-on: https://chromium-review.googlesource.com/1312519
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/c23c1a26bf6aca1a8bb49452f62a6141c8ef52ed/fast_build.sh

vapier@ pointed out test images are published to public.
https://groups.google.com/a/google.com/d/topic/chromeos-chatty-eng/BCSHPYXPP-g/discussion

So we can't install private bundles to test images. An alternative approach I can think of is to upload private bundles as an artifact archive, and download it on running tast autotest on shards. I found we include some tast binaries in autotest SSP archive:
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/commands.py?type=cs&q=package:chromeos+autotest_server_package.tar.bz2&g=0&l=2490

So maybe we can piggy-back it to install private bundles into shards, and then copying to DUTs. This may work, but is complicated, so I want your opinions.

Any thoughts?

Hmm. What's the approach used to install private Autotest tests? Do those somehow get pushed dynamically to DUTs rather than being included in test images?

I'm resuming this work in this quarter. I'll work on a prototype and write a design doc according to learnings.


> What's the approach used to install private Autotest tests? Do those somehow get pushed dynamically to DUTs rather than being included in test images?

My plan is to upload private bundles as build artifacts (just like gs://chromeos-image-archive/coral-release/R73-11547.0.0/autotest_packages.tar), and download it at run time. Devservers have access to the artifacts, so the earlier work to download data files at run time will help here.