I did not know android manifest is processed like that in incremental build.

Thanks for reporting and looking into the specific reasons! I think an adhoc way around it right now is to also check if SecondInstrumentation is in the manifest. This might be problematic if there exist a test apk with two instrumentation but none of which is for org.chromium.base.test.BaseChromiumAndroidJUnitRunner

This will also get resolved once all tests are converted to JUnit4 and we only use class runner to fetch tests, then that can resolve this issue too

Actually just checked with +agrieve, there is an easy work around this issue since the original instrumentation class is stored in a meta-data tag.

Hey Yoland, did you by chance make any progress on this? Thanks!

This is more complicated than I thought, the script side change (https://codereview.chromium.org/3019573002) can figure out that the real instrumentation is BaseChromiumAndroidJUnitRunner instead of android.app.Instrumentation. However, Java side would cause failure since it would look for BaseChromiumAndroidJUnitRunner in manifest and come back with failure:

Unable to find instrumentation info for: ComponentInfo{org.chromium.chrome.tests/org.chromium.base.test.BaseChromiumAndroidJUnitRunner}

Unless we also override "getComponentName" in java side to provide "org.chromium.chrome.tests/android.app.Instrumentation" during incremental build.

I'll have a look at this today.

Reviews underway :)
https://chromium-review.googlesource.com/c/chromium/src/+/688715
https://codereview.chromium.org/3014693002/

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/6c4c84e978fc0b5fa8176b57baea822f4827c2c3

commit 6c4c84e978fc0b5fa8176b57baea822f4827c2c3
Author: Andrew Grieve <agrieve@chromium.org>
Date: Tue Oct 03 15:24:02 2017

Expose <meta-data> in apk_helper.py

Need to use this in test runner to inspect incremental install <meta-data>.

Bug:  chromium:762024 
Change-Id: I6f2f80d14e499e1d79564579f657454da9b5b8e8
Reviewed-on: https://chromium-review.googlesource.com/697908
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>

[modify] https://crrev.com/6c4c84e978fc0b5fa8176b57baea822f4827c2c3/devil/devil/android/apk_helper_test.py
[modify] https://crrev.com/6c4c84e978fc0b5fa8176b57baea822f4827c2c3/devil/devil/android/apk_helper.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2

commit ca127560828fdaa1f72d81d7b81c2eabe61b2cd2
Author: Andrew Grieve <agrieve@chromium.org>
Date: Wed Oct 04 03:50:15 2017

Android: Fix running junit4 tests with incremental install

Was broken in several ways:
1. Permissions were not being granted upon install.
2. Needed custom logic to look at instrumentation name in meta-data.
3. BootstrapApplication logic needed to use the correct component name.
   This logic has always been wrong, but never an issue until now.
4. BaseChromiumAndroidJUnitRunner needs to tell the Junit TestRequestBuilder
   the location of all incremental dex files. This would actually also
   be an issue when using AndroidJUnitRunner directly, except that
   test_runner.py passes an explicit list of classes gleaned via
   proguard, so no "find all classes" logic is needed in this case.

Bug:  762024 
Change-Id: Ibc3880fdff14829d74926d1c576114b8551a5c9f
Reviewed-on: https://chromium-review.googlesource.com/688715
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Yoland Yan <yolandyan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506296}
[modify] https://crrev.com/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2/base/test/android/javatests/src/org/chromium/base/test/BaseChromiumAndroidJUnitRunner.java
[modify] https://crrev.com/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2/build/android/incremental_install/java/org/chromium/incrementalinstall/BootstrapApplication.java
[modify] https://crrev.com/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2/build/android/incremental_install/java/org/chromium/incrementalinstall/ClassLoaderPatcher.java
[modify] https://crrev.com/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2/build/android/pylib/instrumentation/instrumentation_test_instance.py
[modify] https://crrev.com/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2/build/android/pylib/local/device/local_device_gtest_run.py
[modify] https://crrev.com/ca127560828fdaa1f72d81d7b81c2eabe61b2cd2/build/android/pylib/local/device/local_device_instrumentation_test_run.py