It also has a warning message in the log in the 2 failed runs, not sure if it's related. 

[9128/43783] ACTION //chromeos/assistant/internal:build_libassistant(//build/toolchain/cros:target)
WARNING at the command-line "--args":1:12: Build argument has no effect.
arm_arch = "armv8-a+crc"
           ^------------
The variable "arm_arch" was set as a build argument
but never appeared in a declare_args() block in any buildfile.

To view all possible args, run "gn args --list <out_dir>"

The build continued as if that argument was unspecified.

@xdai, can you make the kevin64 builder experimental again for now if this is the only issue blocking chrome uprev.

Yes, I think currently it's the only issue blocking chrome uprev. (caroline-chrome-pfq is also failing right now because of  Issue 916594 , but due to previous experiences in other boards, I think it will recover itself automatically in next run). 

dgarrett@, can you help mark kevin64 experimental again so that we can (hopefully) get a successful uprev tomorrow?

Update: the kevin64 was marked as experimental in https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1387869.

If kevin64 is marked experimental, why did Chrome PFQ still fail at 11448.0.0-rc1 and 11450.0.0-rc1?

That was the only builder to fail in those cases.

The failure is in UpdateChromeosLKGM stage in master.

Yes, current master PFQ's failure is tracked in bug 917623.

vapier@ Any ideas what's going on here. aarch64-cros-linux-gnu-nm in the standalone toolchain can be broken  depending on SDK builder version.

Examples:

Download Dec 22 standalone AArch64 toolchain (https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926424535264797152)

$ bin/aarch64-cros-linux-gnu-nm --help
Usage: /usr/local/google/home/manojgupta/aarch64_3/bin/../usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/binutils-bin/2.27.0-gold/../../../../../usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/binutils-bin/2.27.0/nm.elf [option(s)] [file(s)]
 List symbols in [file(s)] (a.out by default).
 The options are:
<snip>

However if I download the Dec 25' AArch64 toolchain, aarch64-cros-linux-gnu-nm is broken.

$ bin/aarch64-cros-linux-gnu-nm --help
/usr/local/google/home/manojgupta/aarch64_2/bin/../usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/binutils-bin/2.27.0-gold/../../../../../usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/binutils-bin/2.27.0/nm.elf: can't set BFD default target to `aarch64-cros-linux-gnu': Invalid bfd target

seems to be non-determinism in which libbfd.so we're copying over

from a working one:
$ strings ./aarch64-cros-linux-gnu-2018.12.24.142651/lib/libbfd-2.27.0.20170315.so \
  | grep linux-gnu | sort -u
/usr/lib64/binutils/aarch64-cros-linux-gnu/2.27.0/debug

from a failing one:
$ strings ./aarch64-cros-linux-gnu-2018.12.25.072926/lib/libbfd-2.27.0.20170315.so \
  | grep linux-gnu | sort -u
/usr/lib64/binutils/x86_64-pc-linux-gnu/2.27.0/debug

scanning back through history, it looks like this has been happening for a long time.  looking at all the 2018 files:
01 (Jan): all pass
02 (Feb): all pass
03 (Mar): all pass
04 (Apr): all pass
05 (May): all pass
06 (Jun): ~50% fail
07 (Jul): ~50% fail
08 (Aug): ~50% fail
09 (Sep): ~50% fail
10 (Oct): ~50% fail
11 (Nov): ~50% fail
12 (Dec): ~50% fail

the first fail i have is 2018.06.04.200410.  so maybe something changed in chromite around that time ?  or some compiler roll ?

i'd believe the bug has existed for even longer, but something else changed to trigger the non-deterministic aspect.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/2ab07a927b2363848a981448c50675abe362b0c9

commit 2ab07a927b2363848a981448c50675abe362b0c9
Author: Manoj Gupta <manojgupta@google.com>
Date: Thu Dec 27 21:26:57 2018

chromeos-chrome: Use cros readelf instead of Chrome's readelf.

Fixes for SimpleChrome fails on kevin64.

BUG=chromium:917193
TEST=kevin64 builds.

Change-Id: Ia8e9581980db1eb5f47efdbd0f8c347ef787b73c
Reviewed-on: https://chromium-review.googlesource.com/1390397
Commit-Ready: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/2ab07a927b2363848a981448c50675abe362b0c9/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/b8f2c069d723f7ac9bbc94a812ffd1b94140e5db

commit b8f2c069d723f7ac9bbc94a812ffd1b94140e5db
Author: Manoj Gupta <manojgupta@google.com>
Date: Fri Dec 28 23:17:35 2018

chromeos-chrome: Use llvm-nm instead of GNU binutils nm.

binutils nm seems to be randomly broken for AArch64.
Switch to llvm-nm instead.

BUG=chromium:917193
TEST=chromeos-chrome builds.

Change-Id: I45e68f1b90ab617e6211b0ac284092834dfe823d
Reviewed-on: https://chromium-review.googlesource.com/c/1390399
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Commit-Queue: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/b8f2c069d723f7ac9bbc94a812ffd1b94140e5db/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8d6203a906fe4e29aee04e72271b92f3b2dda93d

commit 8d6203a906fe4e29aee04e72271b92f3b2dda93d
Author: Manoj Gupta <manojgupta@google.com>
Date: Tue Jan 01 00:34:48 2019

cros_chrome_sdk: Use cros readelf for target builds.

Use cros's readelf instead of Chrome's bundled readelf for target
builds.
Keep using bundled readelf for host builds.

BUG=chromium:917193
TEST=Chrome builds on kevin64.

Change-Id: I324580587c4c7102f87eef702f7a67c3801120ad
Reviewed-on: https://chromium-review.googlesource.com/1391328
Commit-Ready: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/8d6203a906fe4e29aee04e72271b92f3b2dda93d/cli/cros/cros_chrome_sdk.py
[modify] https://crrev.com/8d6203a906fe4e29aee04e72271b92f3b2dda93d/cli/cros/cros_chrome_sdk_unittest.py

Mike, would be it possible for you to take a look at the lddtree issues when copying aarch64 toolchain? 
strip binary showed the same problem in simplechromeworkflow in kevin64-full builder.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924711731334881136

chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /b/swarming/w/ir/tmp/t/cbuildbot-tmpaP4hhP/chrome-sdk-cache4uN1XP/cache/chrome-sdk/tarballs/kevin64+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__kevin64-full__R73-11558.0.0-rc1+target_toolchain/bin/aarch64-cros-linux-gnu-strip --strip-unneeded -o /b/swarming/w/ir/tmp/t/cbuildbot-tmpaP4hhP/chrome-sdk-stagefA0NPB/chrome /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_kevin64/Release/chrome
/b/swarming/w/ir/tmp/t/cbuildbot-tmpaP4hhP/chrome-sdk-cache4uN1XP/cache/chrome-sdk/tarballs/kevin64+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__kevin64-full__R73-11558.0.0-rc1+target_toolchain/bin/../usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/binutils-bin/2.27.0-gold/../../../../../usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/binutils-bin/2.27.0/strip.elf: can't set BFD default target to `aarch64-cros-linux-gnu': Invalid bfd target

if the wrong bfd lib is copied over, then every binutils tool is going to be broken because every one relies on libbfd to provide the backend

Right but I can't really make kevin64 as important in chromite unless binutils issues in SimpleChromeWorkflow are fixed.

This issue has an owner, a component and a priority, but is still listed as untriaged or unconfirmed. By definition, this bug is triaged. Changing status to "assigned". Please reach out to me if you disagree with how I've done this.

This has just failed again on the experimental kevin64-chrome-pfq: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924338769594160256

The error says: can't set BFD default target to `aarch64-cros-linux-gnu': Invalid bfd target

Another instance of this failure: kevin64-chrome-pfq: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8923957964232627920

Yes, the failures will randomly happen till the libbfd.so copy in binutils package is not fixed. It should not impact the chrome pfq though.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a940d56cde4eac222835db2287970e4cac31d01c

commit a940d56cde4eac222835db2287970e4cac31d01c
Author: Manoj Gupta <manojgupta@google.com>
Date: Tue Jan 22 17:25:02 2019

chromeos_config: Change kevin64 builders for TestSimpleChromeWorkFlow.

Remove TestSimpleChromeWorkFlow from kevin64-chrome-pfq and kevin64-full.
Instead let it run on kevin64-release.
There are very few devs targeting kevin64 so not running on
kevin64-chrome-pfq should not be a blocker.

This will ensure that kevin64-chrome-pfq will pass and should be
able to make it important later.

BUG=chromium:917193
TEST=kevin64-chrome-pfq and kevin64-full do not run TestSimpleChromeWorkFlow.
TEST=kevin64-release runs TestSimpleChromeWorkFlow.

Change-Id: Icb6f56e58d1a16d46070c0080d8ea5c0e605a0ed
Reviewed-on: https://chromium-review.googlesource.com/1426070
Commit-Ready: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/a940d56cde4eac222835db2287970e4cac31d01c/config/chromeos_config.py
[modify] https://crrev.com/a940d56cde4eac222835db2287970e4cac31d01c/config/config_dump.json