New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 844765 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug
Build-Toolchain



Sign in to add a comment

tast: Make sure that compiler is installed when building tests for DUT

Project Member Reported by derat@chromium.org, May 18 2018

Issue description

To build test bundles for DUTs, the "tast" command currently runs "uname -m" on the DUT and then uses the result as a key into this hardcoded map (in https://chromium.googlesource.com/chromiumos/platform/tast/+/master/src/chromiumos/cmd/tast/build/build.go) to get the right compiler to use:

// archToCompiler maps from a machine name (or processor, see "uname -m") to the corresponding
// Go command that should be used for building.
// TODO(derat): What's the right way to get the toolchain name for a given board?
// "cros_setup_toolchains --show-board-cfg <board>" seems to print it, but it's very slow (700+ ms).
var archToCompiler map[string]string = map[string]string{
    "i686":    "i686-pc-linux-gnu-go",
    "x86_64":  "x86_64-cros-linux-gnu-go",
    "armv7l":  "armv7a-cros-linux-gnueabi-go",
    "aarch64": "armv7a-cros-linux-gnueabi-go",
}

Kevin mentioned that when he tried to use this code to build for either x86_64 or ARM today per https://chromium.googlesource.com/chromiumos/platform/tast/+/master/docs/quickstart.md, the resulting compiler wasn't found. It turns out that he hadn't built any test images in his chroot after setting it up; presumably doing so is necessary to get the compilers.

Rahul, here's what you told me earlier over email about the relationship between our Go toolchain packages:

---

cross-*/go ebuild is a symlink to dev-lang/go. But portage will build them
differently because of different {CATEGORY} string (e.g.
https://cs.corp.google.com/chromeos_public/src/third_party/chromiumos-overlay/eclass/cros-llvm.eclass?type=cs&q=%22$%7BCATEGORY%7D%22+package:%5Echromeos_public$&l=14)
.

dev-lang/go provides binaries to build go code for host chroot builds.
  cross-*/go provides binaries to build go code for target boards.

e.g. sudo emerge category/go-package will use the x86_64-pc-linux-gnu-go
compiler.
But emerge-daisy category/go-package uses armv7a-cros-linux-gnueabi-go
compiler.

Same story for any other cross-* package.

---

I suspect that the chromeos-base/tast-cmd package (which gets pulled in by virtual/target-chromium-os-sdk and installs the tast command) should RDEPEND on one or more other packages to install the Go compilers that it uses to cross-compile. Is that correct? If so, is this the correct list?

cross-armv7a-cros-linux-gnueabi/go
cross-x86_64-cros-linux-gnu/go
cross-i686-cros-linux-gnu/go
 
IIUC correctly, virtual/target-chromium-os-sdk  is a target package.
As a result, it can't depend on any cross-* package since they are host packages.

Do you really want to run the compiler on the DUT?

Comment 2 by derat@chromium.org, May 18 2018

Oh, and I think that tast-cmd is installed on moblab. I'm not sure if the additional Go packages would be a problem there in terms of file size -- they're only needed when running with -build=true (i.e. for developing tests, which probably doesn't happen on moblab devices).
Sorry, I think I mis-interpreted virtual/target-chromium-os-sdk. It is indeed a host package.

But, it is not possible to for it to have dependency on cross* packages because of the way SDK builder currently builds the toolchains.


To install the toolchains, setup_board must be called currently.
Cc: vapier@chromium.org
Building a test image isn't necessary, but running "setup_board" is.

Basically, "cross-x86_64-cros-linux-gnu/go" will get installed at the same time as "cross-x86_64-cros-linux-gnu/gcc", "cross-x86_64-cros-linux-gnu/binutils", etc.

Same for other categories of cross-toolchains (I removed support for "cross-i686-cros-linux-gnu/go" some time ago, and added support "cross-aarch64-cros-linux-gnu/go").

I don't think you can simply RDEPEND on these cross-* packages, as they don't exist when a fresh chroot is created. Running "crossdev" is required to create the necessary overlay/symlinks. Easiest way to run crossdev with the right options and then install the cross-* toolchains is to simply do a "setup_board".

Comment 5 by derat@chromium.org, May 18 2018

Thanks for the replies!

I'd like to support the use case where a developer wants to build/deploy/run tests on a DUT that's based on a board that they haven't run setup_board for previously. I think that this is common when running against lab devices to try to replicate test failures.

It sounds like this will already work if the developer has previously run setup_board for some other board that shares the same architecture as the DUT, but not if they haven't. It feels non-ideal, but is my best bet here to add a custom error message when the compiler isn't found that tells the developer to run "setup_board" for the DUT's board?

This isn't too bad if it's a one-time thing, but will I have problems with the compiler getting stale if the developer isn't actively doing development for that board? When do the compiler packages usually get updated -- just when running build_packages or re-running setup_board (maybe with --force)?
There are only a few toolchains so a setup_board for arm, amd64 and arm64 will get you all the toolchains.
The toolchains will get updated whenever update_chroot is called (update_chroot is automatically called in setup_board and build_packages).

Instead of calling setup_board separately for each board, another alternative is:
$ cros_setup_toolchains --targets=boards --include-boards=daisy,eve,kevin
Technically, even daisy isn't needed.
Only one board each representing amd64 and arm64 is all you need.
The 32-bit toolchains get installed automatically for the 64-bit targets.
I usually setup "chell" and "elm" to get all the toolchains.

Comment 8 by derat@chromium.org, May 19 2018

Status: Started (was: Assigned)
Thanks! I'll probably go with {amd,arm}64-generic in the command line since those seems less likely to make the user think that they need to replace "chell" and "elm" with other board names.
You probably want {arm,arm64,amd64}-generic since arm64-generic won't install arm32 toolchains.
Manoj covered things pretty well. packages cannot depend on cross-* by design(ish).

if devs aren't running build_packages or update_chroot periodically, but are running repo sync, then yes things will be stale and out of sync. tast should not attempt to support this scenario as that's the dev's problem.

if you want to make sure "all" the toolchains are installed, you could use the magic "sdk" target with setup_toolchains to pull them all in. or if you have the board name (which you could read out of the DUT), pass that to setup_toolchains.
Project Member

Comment 11 by bugdroid1@chromium.org, May 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/f5d75a93fba51aeae62306ec2fd9b863310dcfd8

commit f5d75a93fba51aeae62306ec2fd9b863310dcfd8
Author: Daniel Erat <derat@chromium.org>
Date: Tue May 22 04:17:54 2018

tast: Suggest installing toolchains for missing compiler.

Update the tast command to suggest running
cros_setup_toolchains if the required Go compiler isn't
found when building tests.

Also drop support for compiling tests for i686.

BUG= chromium:844765 
TEST=checked that "tast run -build=true" suggests
     cros_setup_toolchains command after removing
     cross-armv7a-cros-linux-gnueabi/go and
     cross-x86_64-cros-linux-gnu/go and that after running
     the command, tast is able to build tests again

Change-Id: I7ce59759072e9b95198b4d4e6cf00c296dd897b3
Reviewed-on: https://chromium-review.googlesource.com/1066981
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Rahul Chaudhry <rahulchaudhry@chromium.org>

[modify] https://crrev.com/f5d75a93fba51aeae62306ec2fd9b863310dcfd8/src/chromiumos/cmd/tast/build/build.go

Comment 12 by derat@chromium.org, May 22 2018

Status: Fixed (was: Started)

Sign in to add a comment