Specifically, all 62 tests are crashing.  This seems to have first surfaced in https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2431 .  The crashes all have this stack in them:

  000294e4  __sanitizer::Abort()                                                                             ??:0:0
  00027e70  __sanitizer::Die()                                                                               ??:0:0
  000bab38  __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long)   ??:0:0
  00027f1c  __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long)  ??:0:0
  000a424c  InitializeCommonInterceptors()                                                                   ??:0:0
  0009e2ec  __asan::InitializeAsanInterceptors()                                                             ??:0:0
  000ba554  __asan::AsanInitInternal()                                                                       ??:0:0

Still looking...

Note, it's possible this is related to https://bugs.chromium.org/p/chromium/issues/detail?id=872721 , i.e. there's some kind of hardware failure that in these tests manifests this way.

Nico, could this possibly be https://chromium-review.googlesource.com/c/chromium/src/+/1167465 ?

Taking out of sheriff queue for the moment, please place back in if this needs re-triage

Issue 872721 has been merged into this issue.

This is also responsible for chrome_public_test_apk failing on the same builder. Here's the stack trace that's causing it to fail:

08-09 18:44:12.370   496   496 F DEBUG   :     #01 pc 0003ff81  /system/lib/libc.so (pthread_kill+32)
08-09 18:44:12.370   496   496 F DEBUG   :     #02 pc 0001c73f  /system/lib/libc.so (raise+10)
08-09 18:44:12.370   496   496 F DEBUG   :     #03 pc 000198f1  /system/lib/libc.so (__libc_android_abort+34)
08-09 18:44:12.371   496   496 F DEBUG   :     #04 pc 000174b0  /system/lib/libc.so (abort+4)
08-09 18:44:12.371   496   496 F DEBUG   :     #05 pc 000294e4  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (__sanitizer::Abort()+68)
08-09 18:44:12.371   496   496 F DEBUG   :     #06 pc 00027e70  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (__sanitizer::Die()+192)
08-09 18:44:12.371   496   496 F DEBUG   :     #07 pc 000bab38  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (__asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long)+340)
08-09 18:44:12.371   496   496 F DEBUG   :     #08 pc 00027f1c  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (__sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long)+136)
08-09 18:44:12.371   496   496 F DEBUG   :     #09 pc 000a424c  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (InitializeCommonInterceptors()+21972)
08-09 18:44:12.371   496   496 F DEBUG   :     #10 pc 0009e2ec  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (__asan::InitializeAsanInterceptors()+36)
08-09 18:44:12.371   496   496 F DEBUG   :     #11 pc 000ba554  /data/app/org.chromium.chrome-1/lib/arm/libclang_rt.asan-arm-android.so (__asan::AsanInitInternal()+296)
08-09 18:44:12.371   496   496 F DEBUG   :     #12 pc 00008b49  /data/app/org.chromium.chrome-1/lib/arm/libchromium_android_linker.so (offset 0x4000)
08-09 18:44:12.372   496   496 F DEBUG   :     #13 pc 000073bd  /data/app/org.chromium.chrome-1/lib/arm/libchromium_android_linker.so (offset 0x4000)
08-09 18:44:12.372   496   496 F DEBUG   :     #14 pc 0000512f  /data/app/org.chromium.chrome-1/lib/arm/libchromium_android_linker.so (offset 0x4000)
08-09 18:44:12.372   496   496 F DEBUG   :     #15 pc 00004745  /data/app/org.chromium.chrome-1/lib/arm/libchromium_android_linker.so (offset 0x4000) (Java_org_chromium_base_library_1loader_Linker_nativeLoadLibrary+72)
08-09 18:44:12.372   496   496 F DEBUG   :     #16 pc 000ea6b9  /system/lib/libart.so (art_quick_generic_jni_trampoline+40)
08-09 18:44:12.372   496   496 F DEBUG   :     #17 pc 000e5fc1  /system/lib/libart.so (art_quick_invoke_stub_internal+64)
08-09 18:44:12.372   496   496 F DEBUG   :     #18 pc 003e8911  /system/lib/libart.so (art_quick_invoke_static_stub+172)
08-09 18:44:12.372   496   496 F DEBUG   :     #19 pc 007fa564  [stack]


And attached is the tombstone it generates if it helps

Deleted: tombstone.txt 229 KB

tombstone.txt 229 KB View Download

This is also responsible for the chrome_public_test_vr_apk failures on that builder.

 Issue 872709  has been merged into this issue.

I'd say https://chromium-review.googlesource.com/c/chromium/src/+/1167465 causing this is unlikely, but reverts are cheap, so I'd recommend trying that. It's definitely a CL about android and asan.

Issue 872898 has been merged into this issue.

(i'd create and land a revert, but the repo is closed. since i'll be out for the day when the repo works again, please someone speculatively revert. if the change is sitll in tomorrow morning est, i'll revert then.)

Reverting https://chromium-review.googlesource.com/c/chromium/src/+/1167465 now in https://crrev.com/c/1169933 . 

Bot looks happier now after the revert. (At least, the previous asan crashes are gone.)

> Bot looks happier now after the revert. (At least, the previous asan crashes are gone.)

The revert of https://chromium-review.googlesource.com/c/chromium/src/+/1167465 landed in 
#582012 and was first picked up by the bot in https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2446

But that build also contains #582026, the revert of the most recent Clang roll. Maybe it's more likely that that was the cause...

However the first reported failing build, https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2431, didn't have the roll.

Instead, the first build with that roll was https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2340 and that was completely green.



Okay, maybe it was https://chromium-review.googlesource.com/c/chromium/src/+/1167465 after all. We still need to understand how that broke anything.