Chatted offline, the simplest way to do this is add an extra flag to "perf/run_tests" script, s.t like "--disable-sh-smoke-test" & add that flag for android_chromium_rel_ng.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/883442d271d1257505fb99d0802b5a1a0c201d51

commit 883442d271d1257505fb99d0802b5a1a0c201d51
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Fri Sep 23 01:28:56 2016

Roll src/third_party/catapult/ b803018ac..7bd10eda4 (9 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/b803018ac776..7bd10eda47f1

$ git log b803018ac..7bd10eda4 --date=short --no-merges --format='%ad %ae %s'
2016-09-22 nednguyen Make use_live_traffic in FakeNetworkController default to False
2016-09-22 jbudorick [Android] Attempt to grab the forwarder daemon logs on map failure.
2016-09-22 aiolos Remove warning when a ref build is set as monitored.
2016-09-22 charliea [trace model] Add .range accessor for Event
2016-09-22 nednguyen [Telemetry] Enable typ's discovery flags for telemetry's unittest_runner framework
2016-09-22 sullivan Add ability to query for test patterns of length 8.
2016-09-22 bccheng Explicitly initialize the network controller
2016-09-22 nednguyen Add logging to _FileLock to debug race condition when multiple processes download a same file
2016-09-22 nednguyen [Telemetry] Start ts_proxy_server with host=None when --use-live-site flag is enabled

BUG= 634052 , 643649 ,647340,643320

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2365803002
Cr-Commit-Position: refs/heads/master@{#420534}

[modify] https://crrev.com/883442d271d1257505fb99d0802b5a1a0c201d51/DEPS

Mike, now we can add these flags:

+To disable system health smoke tests on Nexus5, add arguments: 
['--skip=benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.*']

+To enable only system health smoke tests on Nexus 5x, add arguments:
['benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.'] 

Ping Mike, if you can give the code pointer, I can also make the CL

Ping Mike again.

Sorry for the delay. I've created https://codereview.chromium.org/2446883002/. This CL simply disables the system health smoke tests on N5, and lets the N5X run the full suite.

If we want, I believe we can try running telemetry_perf_unittests twice, one with only the smoke tests. I don't think that's helpful, but we can look into it if you think that is better.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/463b55a16e6303c9efd32da390b5a8fcbc4d5c8a

commit 463b55a16e6303c9efd32da390b5a8fcbc4d5c8a
Author: stip <stip@chromium.org>
Date: Tue Oct 25 23:05:52 2016

Disable system health smoke tests on Nexus5.

BUG= 643649 

Review-Url: https://codereview.chromium.org/2446883002
Cr-Commit-Position: refs/heads/master@{#427525}

[modify] https://crrev.com/463b55a16e6303c9efd32da390b5a8fcbc4d5c8a/testing/buildbot/chromium.android.json
[modify] https://crrev.com/463b55a16e6303c9efd32da390b5a8fcbc4d5c8a/testing/buildbot/chromium.linux.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/26a92ee5fdfde64e4c4ef29b4859114560c9195d

commit 26a92ee5fdfde64e4c4ef29b4859114560c9195d
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Wed Oct 26 05:38:02 2016

Roll src/third_party/catapult/ bea54e53a..8d60f90b4 (2 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/bea54e53a27a..8d60f90b4c84

$ git log bea54e53a..8d60f90b4 --date=short --no-merges --format='%ad %ae %s'
2016-10-25 nednguyen [Telemetry] Actually implement the filtering logic for --skip flag
2016-10-25 benjhayden Reduce usage of Histogram.centralBins.

BUG= 643649 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2444383005
Cr-Commit-Position: refs/heads/master@{#427598}

[modify] https://crrev.com/26a92ee5fdfde64e4c4ef29b4859114560c9195d/DEPS

Ooops, sorry for unclear request. With you CL in https://codereview.chromium.org/2446883002, we disabled the system health smoke tests on Nexus5, however, we still run everything (other perf unittests + system health smoke) on Nexus 5X. I rememeber that we also consider running only system health smoke tests on Nexus 5x.

To do that, you would add a positional arg to the command line on Nexus 5x:
tools/perf/run_tests .... benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest

We have the capacity to run the full suite on the N5X, is there a reason why we wouldn't?

Ah, I just worry that we don't have the capacity to run the full suite on N5X. Of course, running more test on more platform is just better for us :P

I verified that the system_health_smoke_test are skipped in https://chromium-swarm.appspot.com/user/task/322632096e568e10