First failure on each:
https://ci.chromium.org/buildbot/chromium.android/Android%20Cronet%20Builder%20Asan/8583
https://ci.chromium.org/buildbot/internal.client.clank/clang-release-builder-x86/18819
https://ci.chromium.org/buildbot/internal.client.clank/clang-release-builder/17247

Has large overlapping CL range:
http://test-results.appspot.com/revision_range?start=618013&end=618065 (53 commits)

Error looks like:
FAILED: libwebviewchromium.so libwebviewchromium.so.TOC lib.unstripped/libwebviewchromium.so 
python "../../build/toolchain/gcc_solink_wrapper.py" --readelf="../../third_party/android_ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-readelf" --nm="../../third_party/android_ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-nm" --strip=../../third_party/eu-strip/bin/eu-strip --sofile="./lib.unstripped/libwebviewchromium.so" --tocfile="./libwebviewchromium.so.TOC" --output="./libwebviewchromium.so" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro --gcc-toolchain=../../third_party/android_ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -fuse-ld=lld -Wl,--color-diagnostics -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a --target=arm-linux-androideabi -Werror -Wl,--warn-shared-textrel -Wl,-O2 -Wl,--gc-sections -rdynamic --sysroot=../../third_party/android_ndk/platforms/android-16/arch-arm -nostdlib -Wl,--warn-shared-textrel -fsanitize=address -fsanitize-address-use-after-scope -Wl,--version-script=../../build/android/android_only_jni_exports.lst -Wl,--dynamic-linker,/system/bin/linker -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -L../../third_party/android_ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o "./lib.unstripped/libwebviewchromium.so" -Wl,-soname="libwebviewchromium.so" @"./libwebviewchromium.so.rsp"
ld.lld: error: undefined symbol: __cxa_pure_virtual
>>> referenced by android_protocol_handler.cc
>>>               obj/android_webview/common/android_protocol_handler.o:(vtable for (anonymous namespace)::AndroidRequestInterceptorBase)

ld.lld: error: undefined symbol: __cxa_guard_acquire
>>> referenced by aw_contents.cc:122 (../../android_webview/browser/aw_contents.cc:122)
>>>               obj/android_webview/common/aw_contents.o:(android_webview::(anonymous namespace)::g_locale())

ld.lld: error: undefined symbol: __cxa_guard_release
>>> referenced by aw_contents.cc:122 (../../android_webview/browser/aw_contents.cc:122)
>>>               obj/android_webview/common/aw_contents.o:(android_webview::(anonymous namespace)::g_locale())

ld.lld: error: undefined symbol: __cxa_guard_acquire
>>> referenced by aw_contents.cc:127 (../../android_webview/browser/aw_contents.cc:127)
>>>               obj/android_webview/common/aw_contents.o:(android_webview::(anonymous namespace)::g_locale_list())

ld.lld: error: undefined symbol: __cxa_guard_release
>>> referenced by aw_contents.cc:127 (../../android_webview/browser/aw_contents.cc:127)
>>>               obj/android_webview/common/aw_contents.o:(android_webview::(anonymous namespace)::g_locale_list())

ld.lld: error: undefined symbol: __cxa_pure_virtual
>>> referenced by aw_contents.cc
>>>               obj/android_webview/common/aw_contents.o:(vtable for android_webview::FindHelper::Listener)
[...]
https://logs.chromium.org/logs/chrome/bb/internal.client.clank/clang-release-builder/17247/+/recipes/steps/compile/0/stdout

+hans, this appears to be different from  issue 916951  as it started failing earlier than r618177, but I'm at loss at what could be the cause.

All three builders fail with different fail with different targets, different undefined symbols, and referenced by different bits of code. The only thing in common appears to be "ld.lld" ?

Any clues on what could be the cause?

Looking at https://ci.chromium.org/buildbot/chromium.android/Android%20Cronet%20Builder%20Asan/8583, that's using Chromium #618078, which is before the Clang roll that landed in #618154 so it's not related to that.

I can bisect it locally unless you beat me to it.

Got it, I'll start reverting:

---
commit 1a874fe60ccb87401867a7334b8eb11e5e4b1552
Author: Tom Anderson <thomasanderson@chromium.org>
Date:   Thu Dec 20 00:37:25 2018 +0000

    Roll buildtools and enable in-tree libc++ on Android
    
    Buildtools changelog:
    https://chromium.googlesource.com/chromium/buildtools/+log/7d88270de197ebe8b439ab5eb57a4a2a0bb810e0..0e1cbc4eab6861b0c84bf2ed9a3c4b7aa2063819
    
    BUG= 767901 
    R=thakis
    
    Change-Id: I96fdafca5e228cbfe7a0909170c0235e7845bbde
    Reviewed-on: https://chromium-review.googlesource.com/c/1385021
    Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
    Commit-Queue: Nico Weber <thakis@chromium.org>
    Reviewed-by: Nico Weber <thakis@chromium.org>
    Reviewed-by: Dirk Pranke <dpranke@chromium.org>
    Cr-Commit-Position: refs/heads/master@{#618035}
---

Revert: https://chromium-review.googlesource.com/c/chromium/src/+/1386434

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0139283eabe99b3d50e65b74c878519a90c1c8e5

commit 0139283eabe99b3d50e65b74c878519a90c1c8e5
Author: Hans Wennborg <hans@chromium.org>
Date: Thu Dec 20 13:22:33 2018

Revert "Roll buildtools and enable in-tree libc++ on Android"

This reverts commit 1a874fe60ccb87401867a7334b8eb11e5e4b1552.

Reason for revert:
This broke the build, see bug.

Original change's description:
> Roll buildtools and enable in-tree libc++ on Android
> 
> Buildtools changelog:
> https://chromium.googlesource.com/chromium/buildtools/+log/7d88270de197ebe8b439ab5eb57a4a2a0bb810e0..0e1cbc4eab6861b0c84bf2ed9a3c4b7aa2063819
> 
> BUG= 767901 
> R=​thakis
> 
> Change-Id: I96fdafca5e228cbfe7a0909170c0235e7845bbde
> Reviewed-on: https://chromium-review.googlesource.com/c/1385021
> Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#618035}

TBR=thakis@chromium.org,dpranke@chromium.org,thomasanderson@chromium.org

Change-Id: Ia54846c3ad7ed2fd03cb3c52bd1d0c18a1461a91
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  767901 , 916962 
Reviewed-on: https://chromium-review.googlesource.com/c/1386434
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618194}
[modify] https://crrev.com/0139283eabe99b3d50e65b74c878519a90c1c8e5/DEPS
[modify] https://crrev.com/0139283eabe99b3d50e65b74c878519a90c1c8e5/build/config/c++/c++.gni

Thanks for the quick look into it!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/acb62c29f149da5137ccf33762e16e53a47c511f

commit acb62c29f149da5137ccf33762e16e53a47c511f
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Thu Dec 20 21:41:42 2018

Remove -Wl,--no-undefined from default Android config

This is needed to enable in-tree libc++ on Android.  Otherwise, we'll see errors
that look like this:

FAILED: libwebviewchromium.so libwebviewchromium.so.TOC lib.unstripped/libwebviewchromium.so
ld.lld: error: undefined symbol: __cxa_guard_acquire
>>> referenced by aw_contents.cc:122 (../../android_webview/browser/aw_contents.cc:122)
>>>               obj/android_webview/common/aw_contents.o:(android_webview::(anonymous namespace)::g_locale())

The issue is that libc++abi is not a dependency of libc++ in asan builds --
instead it's exported from executables to prevent multiple definitions of
libc++abi symbols.

-Wl,-z,-defs is normally added only for non-sanitizer builds [1] on Linux,
Android, and Fuchsia.  But the Android-specific config adds -Wl,--no-undefined
(which is identical to -Wl,-z,-defs) all the time.

The solution is to simply remove the Android-specific -Wl,--no-undefined and
rely on the -Wl,-z,defs added by //build/config/compiler:compiler.

[1] https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?rcl=c6dd1fda26ff8dddd92ba7830e8696afe3b031f5&l=398

BUG= 916962 
R=thakis

Change-Id: If6887c96a0eaeeac3c831511e3e4519dac3532b5
Reviewed-on: https://chromium-review.googlesource.com/c/1387399
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618335}
[modify] https://crrev.com/acb62c29f149da5137ccf33762e16e53a47c511f/build/config/android/BUILD.gn