I THINK this was cause by issues related to bringing up GOMA. If so, it should already be fixed by hidehiko@. It also looks to be in the same area as a problem cmtice@ fixed earlier today.

If either of those was the cause, the build should succeed if it's rerun.

The issue here is portage not looking into right place to find binary packages.

I do not believe GOMA or cmtice@ fix will help here since there is no compilation issue.

Interestingly builder passes the SDKTest if  compiler-rt ebuild is upreved.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chromiumos-sdk/builds/2312

Another test job failed:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chromiumos-sdk/builds/2320

Looking at the code in chromite sdk_stages.py, cross-* packages are copied to new chroot. But portage still fails to find the compiler-rt package with the llvm-next use flag.

    pkgdir = os.path.join('var', 'lib', 'portage', 'pkgs')
    old_pkgdir = os.path.join(self._build_root, constants.DEFAULT_CHROOT_DIR,
                              pkgdir)
    new_pkgdir = os.path.join(self._build_root, new_chroot_dir, pkgdir)
    osutils.SafeMakedirs(new_pkgdir, sudo=True)
    cros_build_lib.SudoRunCommand(
        ['cp', '-r'] + glob.glob(os.path.join(old_pkgdir, 'cross-*')) +
        [new_pkgdir])

seems to me portage found the binpkg.  look at the exact error message:

!!! The following binary packages have been ignored due to non matching USE:

    =cross-armv7a-cros-linux-gnueabi/compiler-rt-5.0_pre300080-r4 -llvm-next

so i guess you're messing with "llvm-next" here.

vapier@ The test CL  https://chromium-review.googlesource.com/c/513590/5/profiles/default/linux/package.use
adds llvm-next use flag to cross-armv7a-cros-linux-gnueabi/compiler-rt. 
So while building the package, it should have been built with "llvm-next" flag.

But the package found by portage is missing "llvm-next".

From my understanding of build script, it creates a new  chroot and copies cross-* packages to the new chroot.
After copying, how does portage know the the USE flags cross-*compiler-rt was built with in older chroot. Is there is a metadata file that also needs to be copied over?

There is a /var/lib/portage/pkgs/Packages file with a section for crosss-arm*/compiler-rt package. 

Is my understanding correct that the "Packages" file also needs to be copied in addition to binaries to the new chroot?

BUILD_TIME: 1496264963
CPV: cross-armv7a-cros-linux-gnueabi/compiler-rt-5.0_pre300080-r4
DEFINED_PHASES: compile configure install prepare test unpack
DEPEND: sys-devel/llvm cross-armv7a-cros-linux-gnueabi/binutils cross-armv7a-cros-linux-gnueabi/gcc sys-devel/make >=dev-util/cmake-2.8.12 dev-vcs/git
DESC: Compiler runtime library for clang
EAPI: 5
IUSE: llvm-next
KEYWORDS: *
LICENSE: UoI-NCSA
MD5: 578f7b3605ef5462d0668f6e40d703e6
MTIME: 1496264964
SHA1: 5dfd571a6826936cfc5aef802ee9dd17051bd02c
SIZE: 2543829
USE: abi_x86_64 amd64 elibc_glibc kernel_linux userland_GNU
REPO: crossdev

it could be copied, or it could be generated on the fly.  my worry with a copy is that it might reference packages that don't exist in the new chroot.

the emaint command can be used to generate it.

Added some logging and Verified that BuildSDKToolchains does build compiler-rt with llvm-next USE flag.

18:41:36: INFO: RunCommand: /b/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpXBVPa8' -- equery u cross-armv7a-cros-linux-gnueabi/compiler-rt in /b/cbuild/repository
+llvm-next

18:41:38: INFO: RunCommand: /b/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpr82FKF' -- emerge -pv cross-armv7a-cros-linux-gnueabi/compiler-rt in /b/cbuild/repository

These are the packages that would be merged, in order:

Calculating dependencies  ... done!
[ebuild   R    ] cross-armv7a-cros-linux-gnueabi/compiler-rt-5.0_pre300080-r4::crossdev  USE="llvm-next" 0 KiB

18:41:41: INFO: RunCommand: sudo -n 'CROS_CACHEDIR=/b/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- grep -A6 compiler-rt /b/cbuild/repository/chroot/var/lib/portage/pkgs/Packages
CPV: cross-aarch64-cros-linux-gnu/compiler-rt-5.0_pre300080-r4
DEFINED_PHASES: compile configure install prepare test unpack
DEPEND: sys-devel/llvm cross-aarch64-cros-linux-gnu/binutils cross-aarch64-cros-linux-gnu/gcc sys-devel/make >=dev-util/cmake-2.8.12 dev-vcs/git
DESC: Compiler runtime library for clang
EAPI: 5
IUSE: llvm-next
KEYWORDS: *
--
CPV: cross-armv7a-cros-linux-gnueabi/compiler-rt-5.0_pre300080-r4
DEFINED_PHASES: compile configure install prepare test unpack
DEPEND: sys-devel/llvm cross-armv7a-cros-linux-gnueabi/binutils cross-armv7a-cros-linux-gnueabi/gcc sys-devel/make >=dev-util/cmake-2.8.12 dev-vcs/git
DESC: Compiler runtime library for clang
EAPI: 5
IUSE: llvm-next
KEYWORDS: *

Mike, How should emaint command be run?

Old chroot has packages other than cross-* so I assume running it won't have any effect.

In the new chroot, is it fine to just copy the Packages file and run "emaint binhost ".?
Note that the cross-* links are setup by crossdev and therefore ebuilds for cross-* packages are not yet available, not sure if that makes any difference to emaint. 

Running emaint boinhost -f didn't help.

I also checked the xpak file in the cross-*compiler-rt package and it does have llvm-next USE flag (Ran qxpak -x file.xpak USE and checked USE file) .

Since, the problem goes away if I uprev compiler-rt ebuild, I believe that crossdev is getting the binary packages from another prebuilt location and NOT from the /var/libs/portage/pkgs directory. Or they might be getting copied over by another set of prebuilts. 


Vapier@ What do you suggest?

who should own this?

I'll work on. Though, I'd need help from more people.
Let me start from short design doc.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c9e196d38eedf135382c083580d9c8305b93b3b5

commit c9e196d38eedf135382c083580d9c8305b93b3b5
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Thu Aug 31 13:50:33 2017

Upload SDK prebuilt earlier than test run.

Currently, on SDK updating, the prebuilt package is uploaded
after test run, followed by version uprev.
However, in this way, there is little time for goma to prepare
using the new SDK before it is used by other bots, which causes
that build slows down until goma gets ready.

With this CL, prebuilt binary will be uploaded before testing
and version uprev is committed after test completion.
Because test takes several hours, there's more chance for goma
to prepare.

BUG=chromium:729178
TEST=Ran trybot.

Change-Id: I4e98e38e91de7989c447632aa8505dcdd87115f2
Reviewed-on: https://chromium-review.googlesource.com/615224
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>

[modify] https://crrev.com/c9e196d38eedf135382c083580d9c8305b93b3b5/cbuildbot/stages/sdk_stages.py
[modify] https://crrev.com/c9e196d38eedf135382c083580d9c8305b93b3b5/cbuildbot/prebuilts.py
[modify] https://crrev.com/c9e196d38eedf135382c083580d9c8305b93b3b5/cbuildbot/builders/sdk_builders.py
[modify] https://crrev.com/c9e196d38eedf135382c083580d9c8305b93b3b5/scripts/upload_prebuilts.py

Chatted with Shinya, and handed over the remaining tasks (adding quick goma SDK availability tests) to him. Thanks!

[It appears that a bunch of old cros issues bulk-added the "Infra" component recently, but they should probably be "Infra>Client>ChromeOS".]

Issue 776869 has been merged into this issue.

 Issue 798617  has been merged into this issue.

+goma folks

Ping? 

Ah, sorry for late update.

Previously, we enabled goma local-burst mode (= when a compiler package is not available on goma server, goma uses local cpu as much as possible. in this mode, the compile performance is almost the same as non-goma env). So, I believe compile error (due to goma) won't occur even if goma is enabled.

The current problem I understand is that a compiler might not be available on goma just after uprev.
If I misunderstood the problem, correct me.

To improve the chromeos compiler deploy process to goma, we were arguing it in the Design Doc (https://docs.google.com/document/d/1K-_3aOJLhGdYwpwsqGxvUktqHlT94a65MuSfJ-3gz6o/edit)

According to the discussion, the work impact seems not so high, so I'll work this with low priority.
  1. We try to kick compiler update process immediately compiler upload happens.
  2. However, goma executor is updating, the process will be postponed. So we send PSA of goma server update to chromeos-toolchain@ to notice it.

If I should work this with high priority, please let me know.

+yunlian@ who is our sheriff next week. For some of our nightly tests, we have to manually update the workaround whenever there's a change to compiler-rt's ebuild.

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/521782

Manoj: is this workaround still needed?

Without an uprev of compiler-rt, llvm next testing in SDK builder used to fail. There is some hassle in keeping it updated but Compiler-rt ebuild is not updated frequently.