The failing bot: https://ci.chromium.org/buildbot/chromium.clang/ToTAndroid/
The CQ trybot it's mimicking? https://ci.chromium.org/buildbot/tryserver.chromium.android/linux_android_rel_ng/

Is there anything different in the build config?


The ToT bot's gn args:

clang_use_chrome_plugins = false
ffmpeg_branding = "Chrome"
is_clang = true
is_component_build = false
is_debug = false
llvm_force_head_revision = true
proprietary_codecs = true
strip_debug_info = true
symbol_level = 1
target_os = "android"


The CQ trybot's gn args:

dcheck_always_on = true
ffmpeg_branding = "Chrome"
goma_dir = "/b/c/goma_client"
is_component_build = false
is_debug = false
proprietary_codecs = true
strip_absolute_paths_from_debug_symbols = true
strip_debug_info = true
symbol_level = 1
target_os = "android"
use_goma = true


Diff between the ToT and CQ bots:

-clang_use_chrome_plugins = false
+dcheck_always_on = true
 ffmpeg_branding = "Chrome"
-is_clang = true
+goma_dir = "/b/c/goma_client"
 is_component_build = false
 is_debug = false
-llvm_force_head_revision = true
 proprietary_codecs = true
+strip_absolute_paths_from_debug_symbols = true
 strip_debug_info = true
 symbol_level = 1
 target_os = "android"
+use_goma = true


Clang and goma stuff aside, looks like our bot is missing:

dcheck_always_on = true
strip_absolute_paths_from_debug_symbols = true

Do any of those make any difference on the tests? It's easy to test with swarming :-) Trying gin_unittests:

$ tools/mb/mb.py isolate out/release gin_unittests
$ tools/swarming_client/isolate.py archive -I https://isolateserver.appspot.com -i out/release/gin_unittests.isolate -s out/release/gin_unittests.isolated
$ export hash=a128ebe40a91ab21981b07775c0fabb541b01404
$ tools/swarming_client/swarming.py trigger --swarming https://chromium-swarm.appspot.com --isolate-server https://isolateserver.appspot.com --dimension device_os KTU84P --dimension device_type hammerhead --dimension os Android --dimension pool Chrome --named-cache swarming_module_cache_vpython .swarming_module_cache/vpython --idempotent --user hans@chromium.org --cipd-package 'bin:infra/tools/luci/logdog/butler/${platform}:git_revision:ff387eadf445b24c935f1cf7d6ddd279f8a6b04c' --cipd-package '.swarming_module:infra/python/cpython/${platform}:version:2.7.14.chromium14' --cipd-package '.swarming_module:infra/tools/luci/logdog/butler/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c' --cipd-package '.swarming_module:infra/tools/luci/vpython-native/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c' --cipd-package '.swarming_module:infra/tools/luci/vpython/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c' --env-prefix PATH .swarming_module --env-prefix PATH .swarming_module/bin --env-prefix VPYTHON_VIRTUALENV_ROOT .swarming_module_cache/vpython --isolated $hash



dcheck_always_on doesn't seem to help.
strip_absolute_paths_from_debug_symbols also doesn't help.

Maybe there's something different in how the tests are run?

For reference, the tests were enabled here:
https://chromium-review.googlesource.com/c/chromium/src/+/968243
https://chromium-review.googlesource.com/c/chromium/src/+/992234

Hmm, I was looking at gin_unittests, but I can't find any runs of that on linux_android_rel_ng :-/

The CQ bot is this one:

      'KitKat Phone Tester (rel)': {
        'additional_compile_targets': [
          'cronet_test_instrumentation_apk',
          'monochrome_static_initializers',
        ],
        'test_suites': {
          'gtest_tests': 'chromium_android_gtests',
          'junit_tests': 'chromium_junit_tests',
          'isolated_scripts': 'monochrome_apk_checker_isolated_script',
        },
        'swarming': {
          'dimension_sets': [
            {
              'device_os': 'KTU84P',
              'device_type': 'hammerhead',
            },
          ],
        },
        'os_type': 'android',
      },


The ToT bot:

      'ToTAndroid': {
        'additional_compile_targets': [
          'all',
        ],
        # This mirrors 'KitKat Phone Tester (rel)'.
        'test_suites': {
          'gtest_tests': 'chromium_android_gtests',
          'junit_tests': 'chromium_junit_tests',
          'isolated_scripts': 'monochrome_apk_checker_isolated_script',
        },
        'swarming': {
          'dimension_sets': [
            {
              'device_os': 'KTU84P',
              'device_type': 'hammerhead',
            },
          ],
        },
        'os_type': 'android',
      },


These look the same to me.

Does testing/buildbot/*exceptions.py list the try not but not the test bot for gin_unittests? Do the boys have the same test lists in the json file in that dir?

Yes, that's it. gin_unittests is excluded from 'KitKat Phone Tester (rel)', so we should do that too. (Maintaining this doesn't seem very tractable though...)

Aha, and it's the same for the other tests that we fail too.

I'll see if I can sync up the lists of tests somehow..

Patch: https://chromium-review.googlesource.com/c/chromium/src/+/1049969

Looking at the failures from https://ci.chromium.org/buildbot/chromium.clang/ToTAndroid/

- gin_unittests 27/29 tests fail, so basically all of them "Couldn't mmap v8 natives data file"
- jingle_unittests 3/67 tests fail, all of them from LibjingleLogTest
- media_blink_unittests 19/1135 tests fail, all of them from KeySystemConfigSelectorTest
- remoting_unittests; actually most pass, but running on android doesn't make sense?
- service_manager_unittests; nothing starts, and it times out after an hour
- webview_instrumentation_test_apk; sounds like something that should work, but doesn't

gin_unittests: https://chromium-review.googlesource.com/#/c/chromium/src/+/1059514

jingle_unittests: https://chromium-review.googlesource.com/c/chromium/src/+/1059613 and https://chromium-review.googlesource.com/#/c/chromium/src/+/1059614

media_blink_unittests: https://chromium-review.googlesource.com/c/chromium/src/+/1059655 and https://chromium-review.googlesource.com/#/c/chromium/src/+/1059656

remoting_unittests: https://chromium-review.googlesource.com/#/c/chromium/src/+/1059777

service_manager_unittests: https://chromium-review.googlesource.com/#/c/chromium/src/+/1059780

webview_instrumentation_test_apk: https://chromium-review.googlesource.com/#/c/chromium/src/+/1059622

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8f3f9b74ea46af2cded08c530cc67c52efd48cf

commit c8f3f9b74ea46af2cded08c530cc67c52efd48cf
Author: Hans Wennborg <hans@chromium.org>
Date: Tue May 15 15:24:39 2018

testing: Move gin_unittests to non_android_chromium_gtests

It's already excluded manually from almost all Android bots in
test_suite_exceptions.pyl, and fails if we try to run it on [1].

It seems this test simply does not reliably work on Android, and
so excluding it by default makes more sense.

For some reason, this move also enables it on some chromium.clang
bots. I'm not sure whether disabling it for those bots was intentional,
but we'll find out.

 [1] https://ci.chromium.org/buildbot/chromium.clang/ToTAndroid/3343

Bug:  840756 ,  842698 ,  843124 
Change-Id: I4c40bbfe66943ad77e6f970f4ca3fa1a218166d3
Reviewed-on: https://chromium-review.googlesource.com/1059514
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558709}
[modify] https://crrev.com/c8f3f9b74ea46af2cded08c530cc67c52efd48cf/testing/buildbot/chromium.android.json
[modify] https://crrev.com/c8f3f9b74ea46af2cded08c530cc67c52efd48cf/testing/buildbot/chromium.clang.json
[modify] https://crrev.com/c8f3f9b74ea46af2cded08c530cc67c52efd48cf/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/c8f3f9b74ea46af2cded08c530cc67c52efd48cf/testing/buildbot/chromium.memory.json
[modify] https://crrev.com/c8f3f9b74ea46af2cded08c530cc67c52efd48cf/testing/buildbot/test_suite_exceptions.pyl
[modify] https://crrev.com/c8f3f9b74ea46af2cded08c530cc67c52efd48cf/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5ba8e28179646b2c24edf2a01a6da9364d331c87

commit 5ba8e28179646b2c24edf2a01a6da9364d331c87
Author: Hans Wennborg <hans@chromium.org>
Date: Tue May 15 15:40:57 2018

testing: Move service_manager_unittests to non_android_chromium_gtests

It's already excluded manually from almost all Android bots.

For some reason this also enables it on some chromium.clang and
chromium.memory bots. If it turns out to fail there, we can add
exclusions.

Bug:  842698 ,  840756 , 843134
Change-Id: I6c119b5272163a41676f1a0f8b7a2081631c9564
Reviewed-on: https://chromium-review.googlesource.com/1059780
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558716}
[modify] https://crrev.com/5ba8e28179646b2c24edf2a01a6da9364d331c87/testing/buildbot/chromium.android.json
[modify] https://crrev.com/5ba8e28179646b2c24edf2a01a6da9364d331c87/testing/buildbot/chromium.clang.json
[modify] https://crrev.com/5ba8e28179646b2c24edf2a01a6da9364d331c87/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/5ba8e28179646b2c24edf2a01a6da9364d331c87/testing/buildbot/chromium.memory.json
[modify] https://crrev.com/5ba8e28179646b2c24edf2a01a6da9364d331c87/testing/buildbot/test_suite_exceptions.pyl
[modify] https://crrev.com/5ba8e28179646b2c24edf2a01a6da9364d331c87/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a0b11ebee78a6807db48426d6ff014513b76a3b1

commit a0b11ebee78a6807db48426d6ff014513b76a3b1
Author: Hans Wennborg <hans@chromium.org>
Date: Tue May 15 16:26:50 2018

testing: Exclude webview_instrumentation_test_apk

It seems to pass on some Android bots, but not ours, so adding
it to the list *shrug*.

Bug:  840756 
Change-Id: I153923ee91fba5df8762fdff8682a58041cf40c7
Reviewed-on: https://chromium-review.googlesource.com/1059622
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558737}
[modify] https://crrev.com/a0b11ebee78a6807db48426d6ff014513b76a3b1/testing/buildbot/chromium.clang.json
[modify] https://crrev.com/a0b11ebee78a6807db48426d6ff014513b76a3b1/testing/buildbot/test_suite_exceptions.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0c5ac9192f7f5f23c41f51fcf918326920ddc2ac

commit 0c5ac9192f7f5f23c41f51fcf918326920ddc2ac
Author: Hans Wennborg <hans@chromium.org>
Date: Tue May 15 17:02:53 2018

media_blink_unittests: Exclude KeySystemConfigSelectorTest on Android

Bug:  608541 ,  840756 ,  842698 
Change-Id: Iaaaf79dc775a6738cff55037ee9c6ff354530e19
Reviewed-on: https://chromium-review.googlesource.com/1059655
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Frank Liberato <liberato@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558755}
[modify] https://crrev.com/0c5ac9192f7f5f23c41f51fcf918326920ddc2ac/media/blink/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e0b1c7971f025f4065eef93326691a570c4c000f

commit e0b1c7971f025f4065eef93326691a570c4c000f
Author: Hans Wennborg <hans@chromium.org>
Date: Wed May 16 09:30:03 2018

testing: Move remoting_unittests to non_android_chromium_gtests

The tests don't pass, and they were previously excluded using
test_suite_exceptions.pyl.

Bug:  842698 ,  840756 , 492768
Change-Id: I040acf080b55b0e1400204f9dd30afc21d38d5de
Reviewed-on: https://chromium-review.googlesource.com/1059777
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Lambros Lambrou <lambroslambrou@chromium.org>
Cr-Commit-Position: refs/heads/master@{#559021}
[modify] https://crrev.com/e0b1c7971f025f4065eef93326691a570c4c000f/testing/buildbot/chromium.android.fyi.json
[modify] https://crrev.com/e0b1c7971f025f4065eef93326691a570c4c000f/testing/buildbot/chromium.android.json
[modify] https://crrev.com/e0b1c7971f025f4065eef93326691a570c4c000f/testing/buildbot/chromium.clang.json
[modify] https://crrev.com/e0b1c7971f025f4065eef93326691a570c4c000f/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/e0b1c7971f025f4065eef93326691a570c4c000f/testing/buildbot/test_suite_exceptions.pyl
[modify] https://crrev.com/e0b1c7971f025f4065eef93326691a570c4c000f/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6ec6070736f62073f74478aeda1052a2589865b2

commit 6ec6070736f62073f74478aeda1052a2589865b2
Author: Hans Wennborg <hans@chromium.org>
Date: Wed May 16 20:30:30 2018

jingle_unittests: Don't run LibjingleLogTest on Android

Those tests don't pass, but the rest of jingle_unittests seems to do so.

Bug: 843104,  840756 ,  842698 
Change-Id: I9895fa88042ec5decdc70e33fceb362d14638f1d
Reviewed-on: https://chromium-review.googlesource.com/1059613
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Nicolas Zea <zea@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#559266}
[modify] https://crrev.com/6ec6070736f62073f74478aeda1052a2589865b2/jingle/BUILD.gn

So close.. now only components_unittests is failing, and I think that's new: https://ci.chromium.org/buildbot/chromium.clang/ToTAndroid/3364

It's looking good now.