The main reason for these errors is the binutils version change (2.25.51 -> 2.27).

Before the upgrade, cross-armv7a-cros-linux-gnueabi/binutils-2.25.51 is installed in the chroot at "/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold".

After the upgrade, cross-armv7a-cros-linux-gnueabi/binutils-2.27 is installed in the chroot at "/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.27-gold".

The path change is also reflected in the standalone toolchain for "armv7a-cros-linux-gnueabi".
i.e. binutils-bin in the standalone toolchain tarball contains "2.27-gold" instead of "2.25.51-gold"

SimpleChromeWorkflow for daisy picks up the new toolchain tarball with the upgraded binutils.
However, when chrome build is setup, gn args contains hardcoded references to binutils-2.25.51.
From out_daisy/Release/gn.args:
cros_target_cc = "armv7a-cros-linux-gnueabi-clang -B.../.cros_cache/chrome-sdk/tarballs/daisy+9588.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -Wno-unknown-warning-option"
cros_target_cxx = "armv7a-cros-linux-gnueabi-clang++ -B.../.cros_cache/chrome-sdk/tarballs/daisy+9588.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -Wno-unknown-warning-option"
cros_target_ld = "armv7a-cros-linux-gnueabi-clang++ -B.../.cros_cache/chrome-sdk/tarballs/daisy+9588.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -Wno-unknown-warning-option"

During the build, clang/clang++ try to run the linker from ".../2.25.51-gold" directory in .cros_cache, which doesn't exist.
The compilers then fallback to "../../third_party/binutils/Linux_x64/Release/bin/" which contains binutils-2.26 for host.
This results in the above error messages.

"out_daisy/Release/gn.args" is getting its values for cros_target_{cc,cxx,ld} from ${GN_ARGS}, which is set by "cros chrome-sdk --board=daisy".
cros_chrome_sdk.py, in turn, computes the values of these variables from CC/CXX/LD environment variables.

The environment variables are loaded from ".cros_cache/chrome-sdk/tarballs/daisy+9588.0.0+environment_chromeos-base_chromeos-chrome.tar.xz/environment".

The basic problem, then, is the binutils version mismatch between "daisy+9588.0.0+target_toolchain" and "daisy+9588.0.0+environment_chromeos-base_chromeos-chrome.tar.xz" in ".cros_cache/chrome-sdk/tarballs/".

"daisy+9588.0.0+target_toolchain" contains the upgraded binutils at "usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.27-gold".

"daisy+9588.0.0+environment_chromeos-base_chromeos-chrome.tar.xz" is setting the environment variables to point to old binutils at "usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold", which no longer exists.


"daisy+9588.0.0+target_toolchain" is built by "~/trunk/chromite/bin/cros_setup_toolchains -t armv7a-cros-linux-gnueabi --create-packages" from inside a chroot.

I don't know how "daisy+9588.0.0+environment_chromeos-base_chromeos-chrome.tar.xz" is created or updated.
It should probably be updated in sync with the target_toolchain package on trybots that run TestSimpleChromeWorkflow.

seems like it'd be easiest to just kill USE=gold in the chromeos-chrome ebuild.  if a target is good enough to support gold in Chrome, then we should already be making that the default via cros_setup_toolchains.py which means the ebuild -B hack is redundant.

I'm not sure I fully understand what you're suggesting.

Are you saying that we should make it so

cros_target_cc = "armv7a-cros-linux-gnueabi-clang -B.../.cros_cache/chrome-sdk/tarballs/daisy+9588.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -Wno-unknown-warning-option"

in args.gn becomes:

cros_target_cc = "armv7a-cros-linux-gnueabi-clang -Wno-unknown-warning-option"

i.e. should we simply remove the -B argument instead of updating it to point to binutils-2.27?

If we do that, will "armv7a-cros-linux-gnueabi-clang" automatically pick up the correct binutils (binutils-2.27 in our case)?

And the question still remains: In SimpleChrome, these gn args are set from variables in "daisy+9588.0.0+environment_chromeos-base_chromeos-chrome.tar.xz", and I do not know how this file is created or updated.

i'm not suggesting you mess with anything in the SimpleChrome workflow.

look at this CL:
  https://chromium-review.googlesource.com/520502

In the chrome ebuild, we have

	# Set binutils path for goma.
	CC_host+=" -B$(get_binutils_path "${LD_host}")"
	CXX_host+=" -B$(get_binutils_path "${LD_host}")"

So probably we cannot remove "-B" is without testing gomma

If "environment_chromeos-base_chromeos-chrome.tar.xz" is created from the chromeos-base/chromeos-chrome ebuild, I'm still confused why we need to modify anything at all.

Note that emerge of chromeos-chrome happens fine in the chroot with new binutils-2.27 installed. Why is the environment (with paths to newer binutils) not getting carried forward to SimpleChromeWorkflow in the trybot?

Seems like the trybot is using the latest target_toolchain tarball with newer binutils from the chroot, but not using the "environment_chromeos-base_chromeos-chrome.tar.xz" from the latest chrome build in the chroot.

the environment.tar.xz file is only generated from the committed ebuild file. ie: your changes to ebuild will not reflect in this file if you are just doing a trybot. 
I dont remember how often these environment files are generated. I think daily??

we're talking about the target compiler settings, not the host.  we already clobber CC_host/CXX_host in chrome-sdk, so the ebuild settings there don't matter.

see: https://cs.corp.google.com/chromeos_public/chromite/cli/cros/cros_chrome_sdk.py?q=f:chromite+cc_host+package:%5Echromeos_public$&dr=C&l=620

As far as I can tell, this issue is not a blocker for the binutils-2.27 upgrade.

Sure, it would be nice for the trybots to use both the toolchain and the environment from the same build, but that's independent of the binutils upgrade.

Once the upgrade is done, SimpleChromeWorkflow will be broken until the builders catch-up with the upgrade and publish the new environment to go along with the new toolchain tarballs. Once they catch up, things will be working fine again. Please let me know if my understanding is incorrect.

On my local machine, I have verified that I can successfully build chrome for daisy using simple-chrome after manually modifying the environment file to fix the binutils path.

Here's the TestSimpleChromeWorkflow log from the latest daisy-release trybot run with binutils-2.27 upgrade: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/11749/steps/TestSimpleChromeWorkflow/logs/stdio

I was wrong about one thing: the trybot *is* using the environment from the same build as the standalone toolchain. In the log, it is clear that the binutils path is being set to "/tmp/cbuildbot-tmpDtkHQd/chrome-sdk-cacheJV3isB/cache/chrome-sdk/tarballs/daisy+__b__cbuild__trybot_archive__trybot-daisy-release__R61-9608.0.0-b11749+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.27-gold"

The compiler, however, does not find the tools there and ends up calling "../../third_party/binutils/Linux_x64/Release/bin/objcopy". I don't know why.

After some more investigation, it all makes sense (sort of):

1) The toolchain tarballs are created and uploaded by the chromiumos-sdk builders. These contain binutils-2.25.51 right now, and will contain binutils-2.27 after the upgrade.

2) The chrome-environment tarballs, however, are board specific. They are (I think) created and uploaded by the *-release builders for each board. This is the code that does it: https://chromium.googlesource.com/chromiumos/chromite/+/master/cbuildbot/stages/chrome_stages.py#177 (but I'm not entirely sure which builders run it).

3) During *-release builds, the "environment_chromeos-base_chromeos-chrome.tar.xz" file is created in the build artifacts directory, right next to the chromiumos image etc. It is presumably uploaded from there for use by simple-chrome workflow. The trybots also pick up this environment when running TestSimpleChromeWorkflow, so the simple-chrome workflow gets tested with the latest environment file from the chromiumos build that the trybot just completed.

4) However, the trybots do not build a fresh toolchain tarball in the chroot for TestSimpleChromeWorkflow. They use the prebuilt one uploaded by the chromiumos-sdk builder. This causes the version mismatch resulting in the issue here: the environment is pointing to binutils-2.27 which is the version in the chroot, but the toolchain tarball being used still has binutils-2.25.

The problem is transient. Once the binutils-2.27 upgrade CL is merged, chromiumos-sdk bot will pick it up, build new toolchain tarballs, and upload them to the official location. This takes ~16 hours. In the meanwhile, all *-release builders will fail in the TestSimpleChromeWorkflow stage since they would have picked up the binutils-2.27 upgrade CL (and hence generated the new environment file), but are not using the old toolchain tarball with binutils-2.25.51.

I see a few options here:

a) Lazy option: Do nothing. Upgrade binutils over some weekend. Let the sdk and release builders catch up by Monday.

b) Hack option 1: Temporarily disable TestSimpleChromeWorkflow stage in *-release builders, so they don't fail while the binutils upgrade is happening. Re-enable the stage after a successful chromiumos-sdk run.

c) Hack option 2: Modify the binutils ebuild so that both 2.25.51 and 2.27 paths are available (one as a symlink to the other). Do this before the upgrade. Let chromiumos-sdk publish new tarballs (with binutils 2.25.51). Then do the upgrade. Let chromiumos-sdk publish new tarballs (with binutils 2.27). Remove the symlink hack.

d) Long term option: Remove the binutils version id from the install path completely. We don't have multiple versions installed at the same time anyway. However, doing this will run into the same issue of temporary mismatch between environment and prebuilt toolchain tarball, so we'd need one of (a), (b), (c), or (e) to roll this out anyway.

e) Good idea option: Modify TestSimpleChromeWorkflow so it builds its own toolchain tarball in the chroot and uses that instead of downloading the prebuilt one. Any reason why we don't want to do this?

One question: Are ${board}-release builders run with --latest-toolchain flag?

If the answer is "no", then we may not have to do anything at all. The builders will effectively ignore the binutils upgrade CL and use the prebuilt toolchains both inside the chroot and for simple-chrome (there will be no binutils version mismatch between environment tarball generated in the chroot and the toolchain tarball used for simple-chrome).

I believe release builders do not use latest-toolchain. My last llvm upgrade to release builders was effective only after chromiumos-sdk published new SDK versions.

Only chell-chrome-pfq (and toolchain builders) seems to use latest_toolchain according to cbuildbot/config_dump.json.

*-release builders do not use --latest-toolchain. Only us (toolchain-team) use this flag.

but, isn't the environment modified as soon as uprev the ebuild? 
will the ebuild only uprev once the chromiums-sdk succeeds? (I dont remember)

I ran two trybot runs for daisy-release, both with binutils-2.27 upgrade CL (https://chromium-review.googlesource.com/#/c/486144). They were launched at the same time:

* With --latest-toolchain flag: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/11820
  + This failed in TestSimpleChromeWorkflow: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/11820/steps/TestSimpleChromeWorkflow/logs/stdio
  + Gn variables in the log indicate simple-chrome trying to use linker from "...+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.27-gold", which doesn't exist in the downloaded prebuilt toolchain (this issue).

* Without --latest-toolchain flag: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/11821
  + This run was successful. Log from TestSimpleChromeWorkflow: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/11821/steps/TestSimpleChromeWorkflow/logs/stdio
  + Gn variable in the log indicate simple-chrome trying to use linker from "...+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold", which is present in the downloaded prebuilt toolchain.
  + This implies that without --latest-toolchain, binutils didn't get updated in the chroot at all. The setup_board/build_packages steps downloaded and used the toolchain prebuilts.


So, this issue will only affect builders that
a) Run with --latest-toolchain flag,
   *and*
b) Run TestSimpleChromeWorkflow.

If *-release builders do not run with --latest-toolchain flag, they will be unaffected.


I think the "Do nothing" option is fine under these conditions. Only chell-chrome-pfq and toolchain builders will fail after the binutils upgrade CL is merged, and they'll recover after the next successful chromiumos-sdk run.

Marking this as WontFix, feel free to re-open if you disagree.