Further observations from the log file at https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chromiumos-sdk/builds/2197/steps/InitSDK/logs/stdio:

-) The builds for sys-devel/gcc and sys-devel/binutils seem to have started in parallel:
> Started sys-devel/binutils-2.27 (logged in /tmp/binutils-2.27-dRsMOR)
> Started sys-devel/gcc-4.9.2-r153 (logged in /tmp/gcc-4.9.2-r153-opkq0O)
> Pending 4/54, Building 2/2, [Time 16:17:26 | Elapsed 12m58.1s | Load 13.13 4.93 2.5]

-) During the gcc build, it decides to use the old version of binutils:
> gcc-4.9.2-r153: checking what linker to use... /usr/x86_64-pc-linux-gnu/binutils-bin/2.23.2/ld

-) Later, the gcc build aborts because it could not find the linker:
> gcc-4.9.2-r153: collect2: fatal error: cannot find 'ld'

-) Right after gcc build fails, we see that binutils emerge succeeded:
> Failed sys-devel/gcc-4.9.2-r153 (in 1m35.8s), retrying later.
> Pending 5/54, Building 1/1, Retrying 1, [Time 16:18:01 | Elapsed 13m33.9s | Load 18.18 7.39 3.43]
> Completed sys-devel/binutils-2.27 (in 1m41.8s)

-) When gcc build is re-tried, it succeeds:
> Retrying emerge of sys-devel/gcc-4.9.2-r153.
> Started sys-devel/gcc-4.9.2-r153 (logged in /tmp/gcc-4.9.2-r153-4nmVIZ)
> ...
> Completed sys-devel/gcc-4.9.2-r153 (in 15m55.1s)

This indicates that the previous failure was a race during parallel emerge of sys-devel/gcc and sys-devel/binutils. While sys-devel/gcc was still building, sys-devel/binutils completed and upgraded the linker. The linker that the gcc ebuild was using (/usr/x86_64-pc-linux-gnu/binutils-bin/2.23.2/ld) is no longer available. Re-trying the sys-devel/gcc ebuild worked, since it used the new linker from the very start.


Later from the log file:
> WARNING: The following packages failed once or more,
> but succeeded upon retry. This might indicate incorrect
> dependencies.
>   sys-devel/gcc-4.9.2-r153

Indeed.

probably: https://bugs.gentoo.org/563614

 Issue 722439  has been merged into this issue.

 Issue 747353  has been merged into this issue.

This is annoyance but I dont see anyone doing anything about this. 
So, I will archive it. 

 Issue 799509  has been merged into this issue.

vapier@ : Will adding a RDEPEND >=${CATEGORY}/binutils-2.27 to gcc ebuild fix the parallel binutils update issue in the builder?

I think a DEPEND= would fix it for now, but it will potentially re-break whenever we upgrade binutils again.

right, everytime binutils updates, it'll run into it

 Issue 879005  has been merged into this issue.

I feel bad people keep reporting this issue. can we try DEPEND solution? even if it is "temporary"

i'm fairly certain gcc just happens to trip it more frequently because it's a larger build, and because we're updating it around the same time.  however, any package that happens to be compiling when binutils is installed might run into the same failure.

i've looked at the binutils-config code and don't see where we're missing the atomic update.

 Issue 915737  has been merged into this issue.

https://chromium-review.googlesource.com/1387715 might mitigate this

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/af95b61ea18e9f223c01c3a81578e4a49620f890

commit af95b61ea18e9f223c01c3a81578e4a49620f890
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Dec 21 19:33:34 2018

make_chroot: manually build host toolchain packages in order

To workaround race conditions where we build & install a package that
is actively being used by other packages we're building (e.g. we run
the assembler while building gcc while also building & installing the
assembler in parallel), pull some of the critical packages out so we
can run them in a specific order.

For now we build binutils & the C library in serial.  The other libs
and compilers don't directly depend on each other, so they can still
build in parallel.

This might slow the build down overall slightly, but we probably don't
run as slow in practice because we don't see large packages (e.g. gcc)
failing to build and then needing to be retried.

BUG= chromium:715788 
TEST=bootstrapping sdk still works

Change-Id: I5d52d1660662957daf10f4ba6b0d116a788dff21
Reviewed-on: https://chromium-review.googlesource.com/c/1387715
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/af95b61ea18e9f223c01c3a81578e4a49620f890/sdk_lib/make_chroot.sh