Link to most recently failing build:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/944

Looks like we are unable to find the browser which is causing several shards to skip running most of the benchmarks, resulting in a couple of thousand benchmarks without data. 

Link to logs of a failed shard: 
https://chrome-swarming.appspot.com/task?id=410d0ff4170cf410&refresh=10&show_raw=1

(CRITICAL) 2018-11-08 17:17:12,415 story_runner.RunBenchmark:384  Benchmark execution interrupted by a fatal exception: <class 'telemetry.internal.browser.browser_finder_exceptions.BrowserFinderException'>(Cannot find browser of type android-chromium. 

Available browsers:
android-chrome
android-system-chrome
reference
)

Has something changed recently here in regards to android?

Looks like the same thing is happeneing for speedometer on android go but they are the only benchamrks on the shard so nothing else is failing: 

https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-go-perf?limit=200

shard 0:
https://chrome-swarming.appspot.com/task?id=410d51bacda07710&refresh=10&show_raw=1

Traceback (most recent call last):
  RunBenchmark at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py:375
    expectations=expectations, max_num_values=benchmark.MAX_NUM_VALUES)
  Run at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py:223
    test, finder_options.Copy(), story_set)
  traced_function at /b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  __init__ at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py:55
    self._test, finder_options)
  traced_function at /b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  _GetPossibleBrowser at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py:103
    possible_browser = self._FindBrowser(finder_options)
  traced_function at /b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  _FindBrowser at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py:98
    finder_options))))
BrowserFinderException: Cannot find browser of type android-chromium. 

Available browsers:
android-chrome
android-system-chrome
reference

Locals:
  finder_options   : [('android_blacklist_file', None), ('assert_gpu_compositing', None), ('browser_executable', None), ('browser_options', [('_browser_startup_timeout', 60), ('_extra_browser_args', set(['--disable-gpu-process-for-dx12-vulkan-info-collection', '--enable-features=TracingPerfettoBackend'])), ('assert_gpu_compositing', False), ('browser_type', 'android-chromium'), ('browser_user_agent_type', None), ('clear_sytem_cache_for_browser_and_profile_on_start', False), ('compatibility_mode', []), ('disable_background_ne ... 478b10>), ('reset_results', None), ('results_label', None), ('run_disabled_tests', False), ('show_stdout', None), ('smoke_test_mode', None), ('story_filter', None), ('story_filter_exclude', None), ('story_shard_begin_index', None), ('story_shard_end_index', None), ('story_tag_filter', None), ('story_tag_filter_exclude', None), ('suite', None), ('suppress_gtest_report', False), ('upload_bucket', 'output'), ('upload_results', True), ('use_live_sites', None), ('verbosity', 1), ('webview_embedder_apk', None)] (truncated)
  possible_browser : None

Looks related to https://chromium-review.googlesource.com/c/chromium/src/+/1315992

But why is that change applying to bots which should not have (yet) enable_chrome_android_internal?

jbudorick, any clues?

This looks pretty bad. Pinpoint jobs also are not being able to find apks any more, so I think the problem is quite widespread. Not sure why the CL in #3 is applying to a lot more bots than were intended?

John, should the CL be reverted?

 Issue 903811  has been merged into this issue.

 Issue 903551  has been merged into this issue.

#5: yes, it should.

enable_chrome_android_internal gets set to true if //clank exists.

Reverted in https://chromium-review.googlesource.com/c/chromium/src/+/1330843 & perf bots appear to be happier.