Yeah the logs aren't very helpful. Yuly or Geoff could you help? I'm traveling and won't have access to a device to try and repro.

I've logged in to the bot to get the following logcat:
I/ActivityManager(  779): Start proc 4566:org.chromium.native_test:test_process/u0a85 for activity org.chromium.native_test/.NativeUnitTestNativeActivity
I/art     ( 4566): Late-enabling -Xcheck:jni
W/ResourcesManager( 4566): Asset path '/system/framework/android.test.runner.jar' does not exist or contains no resources.
I/MultiDex( 4566): VM with version 2.1.0 has multidex support
I/MultiDex( 4566): install
I/MultiDex( 4566): VM has multidex support, MultiDex support library is disabled.
I/cr_base_multidex( 4566): Completed multidex installation.
I/cr_NativeTest( 4566): loading: GLESv2
I/cr_NativeTest( 4566): loaded: GLESv2
I/cr_NativeTest( 4566): loading: _angle_perftests__library
W/linker  ( 4566): lib_angle_perftests__library.so: unused DT entry: type 0x6ffffef5 arg 0x18694
W/linker  ( 4566): lib_angle_perftests__library.so: unused DT entry: type 0x6ffffffe arg 0x18634
W/linker  ( 4566): lib_angle_perftests__library.so: unused DT entry: type 0x6fffffff arg 0x3
E/art     ( 4566): dlopen("/data/app/org.chromium.native_test-1/lib/arm/lib_angle_perftests__library.so", RTLD_LAZY) failed: dlopen failed: cannot locate symbol "__cxa_pure_virtual" referenced by "lib_angle_perftests__library.so"...
D/AndroidRuntime( 4566): Shutting down VM
E/AndroidRuntime( 4566): FATAL EXCEPTION: main
E/AndroidRuntime( 4566): Process: org.chromium.native_test:test_process, PID: 4566
E/AndroidRuntime( 4566): java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__cxa_pure_virtual" referenced by "lib_angle_perftests__library.so"...
E/AndroidRuntime( 4566):        at java.lang.Runtime.loadLibrary(Runtime.java:371)
E/AndroidRuntime( 4566):        at java.lang.System.loadLibrary(System.java:988)
E/AndroidRuntime( 4566):        at org.chromium.native_test.NativeUnitTest.loadLibraries(NativeUnitTest.java:50)
E/AndroidRuntime( 4566):        at org.chromium.native_test.NativeUnitTest.preCreate(NativeUnitTest.java:43)
E/AndroidRuntime( 4566):        at org.chromium.native_test.NativeUnitTestNativeActivity.onCreate(NativeUnitTestNativeActivity.java:19)
E/AndroidRuntime( 4566):        at android.app.Activity.performCreate(Activity.java:5990)
E/AndroidRuntime( 4566):        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1106)
E/AndroidRuntime( 4566):        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2278)
E/AndroidRuntime( 4566):        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2387)
E/AndroidRuntime( 4566):        at android.app.ActivityThread.access$800(ActivityThread.java:151)
E/AndroidRuntime( 4566):        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1303)
E/AndroidRuntime( 4566):        at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime( 4566):        at android.os.Looper.loop(Looper.java:135)
E/AndroidRuntime( 4566):        at android.app.ActivityThread.main(ActivityThread.java:5254)
E/AndroidRuntime( 4566):        at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime( 4566):        at java.lang.reflect.Method.invoke(Method.java:372)
E/AndroidRuntime( 4566):        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
E/AndroidRuntime( 4566):        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)
W/ActivityManager(  779):   Force finishing activity 1 org.chromium.native_test/.NativeUnitTestNativeActivity

OK, thanks. This is surprising considering it only seems to fail on the N5. But maybe this SO post can shed some light:

https://stackoverflow.com/questions/19533014/undefined-symbol-cxa-pure-virtual-error-when-loading-library-from-java

Maybe we can try adding some of those flags.

I suspect the issue is that ANGLEPerfTest::step() doesn't have any implementation now, after moving all the implementing files into whitebox tests.
Maybe we can just add a dummy implementation somewhere?

Looking at our BUILD.gn, it could be a side-effect of making libANGLE an indirect rather than direct dependency. I see

config("angle_common_config") {
  include_dirs = [ "src/common/third_party/base" ]
  if (is_android) {
    libs = [ "log" ]
  }
}

and in libANGLE_config:

  if (is_android && (angle_enable_gl || angle_enable_vulkan)) {
    if (ndk_supports_nativewindow) {
      libs += [ "nativewindow" ]
    } else {
      libs += [ "android" ]
    }
  }

It might be one of those missing libs defines the virtual handler. We could make a build with those and try it on a remote N5. I might even be able to try this using Chrome-oting. Or you guys could help if you have a local device.

Your theory makes more sense. I think we should probably just make a default implementation for step. I'll submit a CL with that. Good thought.

Cool, you can test your CL on manual nexus 5 trybot to verify that this is indeed the issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/12a15e47679ddf16c99aeb4374e836b96f857b0f

commit 12a15e47679ddf16c99aeb4374e836b96f857b0f
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Dec 18 16:37:34 2018

Roll src/third_party/angle 0d218da49519..720ca449767c (1 commits)

https://chromium.googlesource.com/angle/angle.git/+log/0d218da49519..720ca449767c


git log 0d218da49519..720ca449767c --date=short --no-merges --format='%ad %ae %s'
2018-12-18 jmadill@chromium.org Pass GN header visibility check.


Created with:
  gclient setdep -r src/third_party/angle@720ca449767c

The AutoRoll server is located here: https://autoroll.skia.org/r/angle-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:915429
TBR=geofflang@chromium.org

Change-Id: I571025ec199c13f4c5c65ffa0cd172a94519d08c
Reviewed-on: https://chromium-review.googlesource.com/c/1382595
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#617526}
[modify] https://crrev.com/12a15e47679ddf16c99aeb4374e836b96f857b0f/DEPS

So, I've checked with nm and lib_angle_perftests__library.so has __cxa_pure_virtual undefined (even after crrev.com/c/1380431, I guess there are more pure virtual methods somewhere, like in gtest), but libGLESv2.so has it defined.
I guess what happened before is that lib_angle_perftests__library.so was getting this symbol defined by libGLESv2_static.
I'll try to see what differs in libGLESv2 compilation that it gets this symbol defined.

So far I was able to figure out that changing libGLESv2 dep to libGLESv2_static causes __cxa_pure_virtual to be defined. However, this symbol does not come from "libGLESv2_static.a". But, changing the dep causes 56 targets to be rebuilt, so it must be some configuration inherited from libGLESv2_static causing this symbol to be defined. No luck determining exactly what's the difference so far. Compilation and link flags look similar in both cases.

 Issue 916824  has been merged into this issue.

My observation is that if lib_angle_perftests__library.so is linked with either libEGL.so, libGLESv2.so or libangle_util.so, then __cxa_pure_virtual becomes undefined.
The only difference in link flags is the libraries above appear in list of objects being linked.

My theory of what's happening is:
1. __cxa_pure_virtual is defined in each of libEGL.so, libGLESv2.so or libangle_util.so.
2. If lib_angle_perftests__library.so is not linked with the above libraries, linker understands that it needs to add a definition of __cxa_pure_virtual, I guess from c++abi
3. But, if lib_angle_perftests__library.so is linked with the above libraries, the linker thinks: "Oh, it can just use the symbol that is already defined in another library, the loader will load the dependency before lib_angle_perftests__library.so anyways, so the symbol will be found in memory when needed.
4. Nexus 5 loader does something that the linker didn't expect and fails to find the symbol (note that Nexus 5X is able to load lib_angle_perftests__library.so just fine).

Going to workaround this by defining __cxa_pure_virtual in lib_angle_perftests__library.so directly.

thakis@, you are a compiler expert, what do you think about the theory above?

If your 1 is true and the symbol is visible, then your library link flag setup is broken.

If you have several .sos then you either need to make sure you link against a c++ (abi) library that's also in a .so, or the c++ (abi) library needs to be linked statically into each .so but with hidden symbols.

If that's not the case, then you need to change your build to move toward that world.

(+thomasanderson because of the recent change to use use_custom_libcxx on Android makes this potentially easier to fix.)

Sorry, thakis@, I don't know what flags I need to implement your suggestion.
Could you please look at the attached .rsp files and point to what is wrong in them?
All of them have "-lc++abi", I think this means that they "link against a c++ (abi) library that's also in a .so"?

Deleted: libangle_util.so.rsp 1.2 KB

libangle_util.so.rsp 1.2 KB Download

Deleted: libEGL.so.rsp 675 bytes

libEGL.so.rsp 675 bytes Download

Deleted: libGLESv2.so.rsp 1.4 KB

libGLESv2.so.rsp 1.4 KB Download

Deleted: lib_angle_perftests__library.so.rsp 5.0 KB

lib_angle_perftests__library.so.rsp 5.0 KB Download

"It depends", see e.g. https://cs.chromium.org/search/?q=export_libcxxabi_from_executables&sq=package:chromium&type=cs

Actually, objdump -x lib_angle_perftests__library.so gives me:

Dynamic Section:
  NEEDED               libEGL.so
  NEEDED               libGLESv2.so
  NEEDED               libc.so
  NEEDED               libdl.so
  NEEDED               libm.so
  NEEDED               libandroid.so
  NEEDED               liblog.so
  NEEDED               libvulkan.so
  SONAME               lib_angle_perftests__library.so

I guess c++abi is linked statically, then?
What do I need to hide the symbols? Or to link dynamically?

Looking at the Cl in comment 0 sounds like this might be like  issue 744567 . Can you read that issue and check if it's the same general issue? If so, then it's possible that the CL in comment 0 just isn't a workable approach.


(Sorry, like everyone else I'm trying to wrap my stuff up for the year and I don't have time to look into this in depth.)

It looks a bit different from  issue 744567 . There they had 2 copies of base, here we have 0 copies of c++abi.

So, after https://chromium-review.googlesource.com/c/1385021 lib_angle_perftests__library.so links with obj/buildtools/third_party/libc++abi/libc++abi/cxa_virtual.o instead of c++abi, which solves __cxa_pure_virtual issue.
However, now it fails to find unw_getcontext, so it still fails.

Does it need a dependency on libunwind, too?

Don't know, something is very weird here.
The bot says:

I/ActivityManager(  776): Start proc 4494:org.chromium.native_test:test_process/u0a85 for activity org.chromium.native_test/.NativeUnitTestNativeActivity
W/ResourcesManager( 4494): Asset path '/system/framework/android.test.runner.jar' does not exist or contains no resources.
I/MultiDex( 4494): VM with version 2.1.0 has multidex support
I/MultiDex( 4494): install
I/MultiDex( 4494): VM has multidex support, MultiDex support library is disabled.
I/cr_base_multidex( 4494): Completed multidex installation.
I/cr_NativeTest( 4494): loading: EGL
I/cr_NativeTest( 4494): loaded: EGL
I/cr_NativeTest( 4494): loading: _angle_perftests__library
W/linker  ( 4494): lib_angle_perftests__library.so: unused DT entry: type 0x6ffffef5 arg 0x193e8
W/linker  ( 4494): lib_angle_perftests__library.so: unused DT entry: type 0x6ffffffe arg 0x19388
W/linker  ( 4494): lib_angle_perftests__library.so: unused DT entry: type 0x6fffffff arg 0x3
E/art     ( 4494): dlopen("/data/app/org.chromium.native_test-1/lib/arm/lib_angle_perftests__library.so", RTLD_LAZY) failed: dlopen failed: cannot locate symbol "unw_getcontext" referenced by "lib_angle_perftests__library.so"...
D/AndroidRuntime( 4494): Shutting down VM
E/AndroidRuntime( 4494): FATAL EXCEPTION: main
E/AndroidRuntime( 4494): Process: org.chromium.native_test:test_process, PID: 4494
E/AndroidRuntime( 4494): java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "unw_getcontext" referenced by "lib_angle_perftests__library.so"...
E/AndroidRuntime( 4494):        at java.lang.Runtime.loadLibrary(Runtime.java:371)
E/AndroidRuntime( 4494):        at java.lang.System.loadLibrary(System.java:988)
E/AndroidRuntime( 4494):        at org.chromium.native_test.NativeUnitTest.loadLibraries(NativeUnitTest.java:50)
E/AndroidRuntime( 4494):        at org.chromium.native_test.NativeUnitTest.preCreate(NativeUnitTest.java:43)
E/AndroidRuntime( 4494):        at org.chromium.native_test.NativeUnitTestNativeActivity.onCreate(NativeUnitTestNativeActivity.java:19)
E/AndroidRuntime( 4494):        at android.app.Activity.performCreate(Activity.java:5990)
E/AndroidRuntime( 4494):        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1106)
E/AndroidRuntime( 4494):        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2278)
E/AndroidRuntime( 4494):        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2387)
E/AndroidRuntime( 4494):        at android.app.ActivityThread.access$800(ActivityThread.java:151)
E/AndroidRuntime( 4494):        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1303)
E/AndroidRuntime( 4494):        at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime( 4494):        at android.os.Looper.loop(Looper.java:135)
E/AndroidRuntime( 4494):        at android.app.ActivityThread.main(ActivityThread.java:5254)
E/AndroidRuntime( 4494):        at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime( 4494):        at java.lang.reflect.Method.invoke(Method.java:372)
E/AndroidRuntime( 4494):        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
E/AndroidRuntime( 4494):        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)
W/ActivityManager(  776):   Force finishing activity 1 org.chromium.native_test/.NativeUnitTestNativeActivity

However, when I list lib_angle_perftests__library.so symbols with nm or objdump, "unw_getcontext" doesn't appear there.

Aha, "unw_getcontext" doesn't appear when I build locally, but it appears as undefined on the bot. Guess my local environment differs from the bot environment somehow.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/04baee6d1b9538141a9465f514cf042b511459dd

commit 04baee6d1b9538141a9465f514cf042b511459dd
Author: Yuly Novikov <ynovikov@chromium.org>
Date: Fri Dec 21 21:22:21 2018

Disable angle_perftests on Nexus 5 GPU.FYI bot

Broken by http://crrev/c/1359515,
which is more important than running them on this bot.

TBR=kbr@chromium.org

Bug: 915429
Change-Id: I837f7b2a8fe029647fb5e36639d196346565b148
Reviewed-on: https://chromium-review.googlesource.com/c/1388653
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618604}
[modify] https://crrev.com/04baee6d1b9538141a9465f514cf042b511459dd/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/04baee6d1b9538141a9465f514cf042b511459dd/testing/buildbot/test_suite_exceptions.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd1da8c13f676843ae4ab9fd015803ba10b8a899

commit bd1da8c13f676843ae4ab9fd015803ba10b8a899
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Fri Dec 21 23:31:20 2018

Disable angle_perftests on Nexus5.

It has been failing for several days.

To reenable, just re-add the lines I removed to perf_data_generator.py
and then rerun tools/perf/generate_perf_data
TBR=jbudorick@chromium.org

Bug:  916824 , 915429
Change-Id: I46bb0cd44b1ebf68ccbd6eae79a718c4cf49ee73
Reviewed-on: https://chromium-review.googlesource.com/c/1387832
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618668}
[modify] https://crrev.com/bd1da8c13f676843ae4ab9fd015803ba10b8a899/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/bd1da8c13f676843ae4ab9fd015803ba10b8a899/tools/perf/core/perf_data_generator.py

I hope  issue 767901  solves the linking / loading problem.
Lowering the priority since the test is disabled now.

Switching blocked/blocked on so I can close  issue angleproject:2995  while keeping this issue associated.