Hi stevenjb@, do you know what's the reason for this? hana-chrome-pfq keeps failure because of this.

I believe that hana is still experimental, which is why it is greyed out in Legoland. The failures go back to at least 9/19:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2960230

It was probably broken by this change:
https://chromium-review.googlesource.com/1232194

I'm not sure why that broke hana specifically.

Yunlian found the issue yesterday and had a fix below.  It seems to be caused by Chrome build not setting "use_gold=true" gn flag although it is actually using gold.  This caused libassistant build to use bfd linker instead which does not work with some of the linker flags on arm platform.

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1240816

also happened in peppy-chrome-pfq
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8934399751426913264

Thanks for the fix.

The fix has been reverted in  issue 889584 .

This is still happening on hana-pfq.  :

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933212433232216864

  [2653/2653] SOLINK ./libassistant.so
  FAILED: libassistant.so libassistant.so.TOC 
  python "../../../../../../chromeos/assistant/libassistant/src/build/toolchain/gcc_solink_wrapper.py" --readelf="readelf" --nm="nm" --sofile="./libassistant.so" --tocfile="./libassistant.so.TOC" --output="./libassistant.so"  -- armv7a-cros-linux-gnueabihf-clang++ -B/b/swarming/wDPSaed/ir/tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabihf/binutils-bin/2.27.0-gold -Wno-unknown-warning-option -shared -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -Wl,--as-needed -fuse-ld=bfd --target=arm-linux-gnueabihf -Wl,-O2 -Wl,--gc-sections --sysroot=../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz -L../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/lib -Wl,-rpath-link=../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/lib -L../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/lib -Wl,-rpath-link=../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/lib -L../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/local/lib -Wl,-rpath-link=../../../../../../../../../../../../../../tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/local/lib -static-libgcc -Wl,-O2 -Wl,--as-needed -Wl,--fix-cortex-a53-843419 -stdlib=libc++ -o "./libassistant.so" -Wl,-soname="libassistant.so" @"./libassistant.so.rsp"
  /b/swarming/wDPSaed/ir/tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabihf/binutils-bin/2.27.0/ld.bfd.real.elf: unrecognized option '--fix-cortex-a53-843419'
  /b/swarming/wDPSaed/ir/tmp/t/cbuildbot-tmpVUzLq9/chrome-sdk-cacheg66hx7/cache/chrome-sdk/tarballs/hana+__b__swarming__wDPSaed__ir__cache__cbuild__repository__buildbot_archive__hana-chrome-pfq__R71-11138.0.0-rc1+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabihf/binutils-bin/2.27.0/ld.bfd.real.elf: use the --help option for usage information
  clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "../../chromeos/assistant/internal/build_libassistant.py", line 172, in <module>
      sys.exit(main(sys.argv[1:]))
    File "../../chromeos/assistant/internal/build_libassistant.py", line 163, in main
      subprocess.check_call(args)
    File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['ninja', '-C', 'obj/chromeos/assistant/internal', 'libassistant.so']' returned non-zero exit status 1

Issue 893060 has been merged into this issue.

This is now failing on daisy_skate-chrome-pfq as well. Any progress here?

Can we escalate since we haven't had a good Chrome PFQ in quite some time?  thanks

Daisy_skate-chrome-pfq just passed, but hana is failing for the same reason.

Is there a way to get source code of chrome from the chrome-pfq builder so that I can try to reproduce it locally?

I tried to repro inside chroot and emerge-hana chromeos-chrome with local chrome source, but everything works fine. I also tried simple chrome flow with hana and builds fine.

TestSimpleChromeWorkflow seems special. We need some help from experts on this.
stevenjb@, do you know how we can run TestSimpleChromeWorkflow locally?

that stage simply does:
  https://chromium.googlesource.com/chromiumos/docs/+/master/simple_chrome_workflow.md

I think the problem here is how to get the fresh chromium code.
fetch chromium might only get the one that passed pfq test?

mmm, how you sync the code locally is entirely up to you.  the default sync is to grab tot, not to grab a lkgm.  so the default already matches what the chrome pfq bots are doing.

if you want to replicate the SDK/sysroot usage, the default chrome-sdk will use lkgm for that (which is independent of the `gclient sync` for chromium source code), but you can easily specify the version info yourself.

if you look at the bot log for RunCommand lines, it shows the exact commands the bot is using.  the very first one is the `cros chrome-sdk` it uses.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/590633f6b80bdb51ffdb8968a7dc95a92c4647c5

commit 590633f6b80bdb51ffdb8968a7dc95a92c4647c5
Author: Manoj Gupta <manojgupta@google.com>
Date: Wed Oct 10 02:36:35 2018

chromeos-chrome: Filter unsupported flags.

Filter unsupported arm64 flags from arm32 builds.
"-Wl,--fix-cortex-a53-843419" is an arm64 option
that is not needed for arm32.

BUG= chromium:889079 
TEST=hana-chrome-pfq tryjob works.

Change-Id: Ib84b9aff2ba482a379c7fe100ee91a516a956698
Reviewed-on: https://chromium-review.googlesource.com/c/1272163
Commit-Queue: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Trybot-Ready: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Caroline Tice <cmtice@chromium.org>

[modify] https://crrev.com/590633f6b80bdb51ffdb8968a7dc95a92c4647c5/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

hana chrome pfq is green now (and so is master pfq) https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/Prod/b8933078610905641776

 Issue 888710  has been merged into this issue.