There are a *lot* of related bugs like this that could use some triage/consolidation:
https://bugs.chromium.org/p/chromium/issues/list?q=SystemHealthBenchmarkSmokeTest

Here are a few recent flaky failures in telemetry_perf_unittests (memory_mobile and memory_desktop):

benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_mobile.load:news:reddit
https://chromium-swarm.appspot.com/task?id=3cedcd700e982710&refresh=10&show_raw=1&wide_logs=true

benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_mobile.load:tools:dropbox
https://chromium-swarm.appspot.com/task?id=3ce929c99b443d10&refresh=10&show_raw=1&wide_logs=true

benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:news:reddit
https://chromium-swarm.appspot.com/task?id=3ce8e20eca13b510&refresh=10&show_raw=1&wide_logs=true

moving this to perezju (although he's traveling this week, and so are the other benchmark owners I know)

Hmhh, not sure why this test is never flaky on perf bots: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=system_health.memory_mobile&tests=system_health.memory_mobile%2Fload%3Atools%3Adropbox


How is android_n5x_swarming_rel different from Android Nexus 5X in perf waterfall?

Detected 4 new flakes for test/step "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_mobile.load:tools:dropbox". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyfAsSBUZsYWtlInFiZW5jaG1hcmtzLnN5c3RlbV9oZWFsdGhfc21va2VfdGVzdC5TeXN0ZW1IZWFsdGhCZW5jaG1hcmtTbW9rZVRlc3Quc3lzdGVtX2hlYWx0aC5tZW1vcnlfbW9iaWxlLmxvYWQ6dG9vbHM6ZHJvcGJveAw. This message was posted automatically by the chromium-try-flakes app.

load:tools:dropbox flaked again on CQ and still no flakes on perf bots.

Not sure what to do. Disable anyway on N5X? 

This is a Chrome crash with good stack trace:
Stack Trace:
  ********************************************************************************
  	UI dump
  	- (no package):
  	  - (no id)
  	- android:
  	  - (no id)
  	  - android:id/button1['OK']
  	  - android:id/buttonPanel
  	  - android:id/content
  	  - android:id/contentPanel
  	  - android:id/message['Unfortunately, Chromium has stopped.']
  	  - android:id/parentPanel
  	  - android:id/scrollView
  	********************************************************************************
  	Logcat

....
  	04-18 22:18:27.198  7314  7365 E chromium: [ERROR:shell_integration_android.cc(32)] Not implemented reached in shell_integration::DefaultWebClientState shell_integration::GetDefaultBrowser()
....


Full stack can be found in https://chromium-swarm.appspot.com/task?id=3cf3a23b0f3bb810&refresh=10&show_raw=1 (Search for "Found Minidump: True")


robertshield@: can you take a look at this bug since you seem to be the one who added that CHECK

nednguyen: Is it possible to make it more easy to tell by the output whether it's a proper crash or some timeout or flake? When I ran into a similar problem the other week, I just interpreted the output as noise and figured the test was flaky.

#8: hans@ I can tell that it's a Chrome crash by the UI dump above. 

I don't believe I'm the right person to debug this. The stack trace in the log isn't useful (since it doesn't provide a trace of the useful part of the stack showing where the not implemented function is called from).

I can suggest a place to start:

Looks like there are two NOTREACHED()s being hit in shell_integration code:

04-18 22:18:27.058  7314  7314 E chromium: [ERROR:shell_integration_android.cc(22)] Not implemented reached in shell_integration::DefaultWebClientSetPermission shell_integration::GetDefaultWebClientSetPermission()

04-18 22:18:27.198  7314  7365 E chromium: [ERROR:shell_integration_android.cc(32)] Not implemented reached in shell_integration::DefaultWebClientState shell_integration::GetDefaultBrowser()

I would guess something has changed to cause ShellIntegration functions to be called on Android, but I don't see any usable call stacks in the log that would suggest where they are being called from.

GetDefaultWebClientSetPermission is called less often than GetDefaultBrowser (only a couple of interesting call sites: https://codesearch.chromium.org/search/?q=GetDefaultWebClientSetPermission&sq=package:chromium&type=cs), so hunting down calls to that during test / Chrome startup would be where I'd start. 

If anyone on this bug has an Android dev environment, probably a breakpoint set there would show why the shell_integration functions are being called quite quickly.