Last 14 builds are red, from
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2423
to
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2436

If we count failing targets we get:

      6 Failure chrome_sync_shell_test_apk on Android device Nexus 5X 
      4 Failure chrome_public_test_apk on Android device Nexus 5X
      2 Failure chrome_public_test_vr_apk on Android device Nexus 5X
      2 Failure viz_content_browsertests on Android device Nexus 5X 
      2 Failure components_browsertests on Android device Nexus 5X 
      1 Failure vr_pixeltests on Android device Nexus 5X 
      1 Failure url_unittests on Android device Nexus 5X
      1 Failure unit_tests on Android device Nexus 5X
      1 Failure surface_sync_content_browsertests on Android device Nexus 5X
      1 Failure content_browsertests on Android device Nexus 5X 

chrome_sync_shell_test_apk failed in last 6 builds, starting from https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2431

I don't see anything obviously related in https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI/2431.

Adding the sync team triager, in case they have seen something similar before.

 Issue 872710  has been merged into this issue.

I filled  issue 872714  specifically for chrome_sync_shell_test_apk. All tests are failing, so disabling them makes no sense and I don't know how to proceed.

Meanwhile let's have a look at chrome_public_test_apk.

It failed in 2423 and 2424 and then 2431-2436.

in 2423 org.chromium.chrome.browser.fullscreen.FullscreenManagerTest#testHideBrowserControlsAfterFlingBoosting failed.

in 2424 it was an infra failure:

Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/test_runner.py", line 1040, in main
    return RunTestsCommand(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 743, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 875, in RunTestsInPlatformMode
    raw_results = test_run.RunTests()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_test_run.py", line 58, in RunTests
    tests = self._GetTests()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 334, in _GetTests
    raw_tests = self._GetTestsFromRunner()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 692, in _GetTestsFromRunner
    raw_test_lists = self._env.parallel_devices.pMap(list_tests).pGet(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap
    r.pFinish(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish
    self._objs.JoinAll()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 690, in list_tests
    return crash_handler.RetryOnSystemCrash(_run, d)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash
    return f(device)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 680, in _run
    target, extras=extras, retries=0, timeout=timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1317, in StartInstrumentation
    large_output=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1136, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1101, in handle_large_output
    handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1087, in handle_large_command
    return handle_check_return(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1078, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1074, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 534, in Shell
    command, output, status=None, device_serial=self._device_serial)

2431 - similar infra failure:

I  195.119s tear_down_device(01e3a056239c9955)  Wrote device cache: /b/swarming/w/ir/out/Release/device_cache_01e3a056239c9955.json
I  195.122s Main  Generated json results file at /b/swarming/w/itJOs4QW/tmpqsgxOe
I  195.122s Main  Finishing archiving output.
E  195.123s Main  Error occurred.
Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/test_runner.py", line 1040, in main
    return RunTestsCommand(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 743, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 875, in RunTestsInPlatformMode
    raw_results = test_run.RunTests()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_test_run.py", line 58, in RunTests
    tests = self._GetTests()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 334, in _GetTests
    raw_tests = self._GetTestsFromRunner()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 692, in _GetTestsFromRunner
    raw_test_lists = self._env.parallel_devices.pMap(list_tests).pGet(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap
    r.pFinish(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish
    self._objs.JoinAll()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 690, in list_tests
    return crash_handler.RetryOnSystemCrash(_run, d)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash
    return f(device)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 687, in _run
    dev.PullFile(dev_test_list_json.name, host_file)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1872, in PullFile
    self.adb.Pull(device_path, host_path)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 478, in Pull
    self._RunDeviceAdbCmd(cmd, timeout, retries)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 305, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 285, in _RunAdbCmd
    args, output, status, device_serial)
AdbCommandFailedError: (device: 01e3a056239c9955) adb pull /sdcard/temp_file-58e7c3a1f7f7d.json /b/swarming/w/itJOs4QW/tmpzGw9Nu/list_tests.json: failed with exit status 1 and output:
- adb: error: remote object '/sdcard/temp_file-58e7c3a1f7f7d.json' does not exist

so all chrome_public_test_apk failures look like such an infra failure.

I've filled issue 872721 for this.

I believe all the consistent redness here is due to the ASAN issue being tracked in  issue 872714 .