Now that system health stories are indepdent units, I think we shouldn't feel shy about only disabling stories that fail.  

However, we do need to make sure that we monitor the metrics on a per story level, not per benchmark level.

We report values for individual stories (e.g. load:social:facebook) and groups (e.g. load:social). However, we only monitor (alerts) the former. Therefore, I don't think this should be a problem.

The question is how we should do that. As I said before, I think it would be good to reuse the @benchmark.Disabled/Enabled notation. To be able to use it, we would need to pass |possible_browser| at some point to the SH story set (at creation time?) so that it could call decorators.ShouldSkip (https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/decorators.py?l=191). The problem with this is that |possible_browser| is not reachable from CreateStorySet (https://cs.chromium.org/chromium/src/tools/perf/benchmarks/system_health.py?l=40).

Alternatively, instead of doing this at the System Health story set level, we could generalize the functionality and add support for disabling arbitrary stories. This would essentially mean adding an extra if statement to story_runner (https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py?l=213).

I agree with Ned that if you're monitoring on a benchmark/story set level, you shouldn't disable on a story level.

You can't accurately compare the same benchmark on two different story sets (even if one is a subset of the other) and expect to have a useful metric. You would run a risk of masking real regressions, and at the very least get spurious regressions/improvements each time you disabled/enabled a story.

#3: I completely agree. However, we monitor on a story level in the SH benchmarks (since ShouldTearDownStateAfterEachStoryRun is overriden to return True). Therefore, I think that the feature would make sense and be very useful.

Petr: I think the easiest way to do this is to reuse the CanRunStory(..) structure in SharedStoryState (..).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/709cafa59a08f71bded1f42a110f0d0d7d999bb8

commit 709cafa59a08f71bded1f42a110f0d0d7d999bb8
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Thu Sep 08 17:26:35 2016

Roll src/third_party/catapult/ fa4f08391..328391fbc (2 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/fa4f083913a8..328391fbc1e3

$ git log fa4f08391..328391fbc --date=short --no-merges --format='%ad %ae %s'
2016-09-08 petrcermak [telemetry] Export SharedPageState.possible_browser
2016-09-08 andy Allow symlinks to tracing binaries

BUG= 634331 

TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2326563002
Cr-Commit-Position: refs/heads/master@{#417324}

[modify] https://crrev.com/709cafa59a08f71bded1f42a110f0d0d7d999bb8/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0d3d85c5f3a772dd65b1eae53c890d7251dee0ab

commit 0d3d85c5f3a772dd65b1eae53c890d7251dee0ab
Author: petrcermak <petrcermak@chromium.org>
Date: Fri Sep 09 10:41:50 2016

[system-health] Add support for disabling individual stories on individual platforms

This patch adds support for enabling/disabling individual stories on individual
platforms using the same approaches as for benchmarks:

  1. Disabled/Enabled decorator:

     @decorators.Disabled('win')
     class Story(system_health_story.SystemHealthStory):
        ...

  2. ShouldDisable method:

     class Story(system_health_story.SystemHealthStory):
        ...

        @classmethod
        def ShouldDisable(cls, possible_browser):
          return possible_browser.platform.GetOSName() == 'win'

Note that this patch also enables search:portal:google on all desktop
platforms except for Windows.

BUG= 634331 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq

Review-Url: https://codereview.chromium.org/2228103002
Cr-Commit-Position: refs/heads/master@{#417552}

[modify] https://crrev.com/0d3d85c5f3a772dd65b1eae53c890d7251dee0ab/tools/perf/page_sets/system_health/browsing_stories.py
[modify] https://crrev.com/0d3d85c5f3a772dd65b1eae53c890d7251dee0ab/tools/perf/page_sets/system_health/loading_stories.py
[modify] https://crrev.com/0d3d85c5f3a772dd65b1eae53c890d7251dee0ab/tools/perf/page_sets/system_health/platforms.py
[modify] https://crrev.com/0d3d85c5f3a772dd65b1eae53c890d7251dee0ab/tools/perf/page_sets/system_health/searching_stories.py
[modify] https://crrev.com/0d3d85c5f3a772dd65b1eae53c890d7251dee0ab/tools/perf/page_sets/system_health/system_health_story.py