Maybe also include which stories are disabled an on which platforms.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b087b7731c2efa0c446d4b52ea0839487cce4e1d

commit b087b7731c2efa0c446d4b52ea0839487cce4e1d
Author: nednguyen <nednguyen@google.com>
Date: Fri May 19 16:52:22 2017

Modify list_system_health_stories to generate_system_health_csv

This CL also check in system_health_stories.csv file, which will be used to later
replace https://bit.ly/csh-stories

BUG=723636

Review-Url: https://codereview.chromium.org/2888133002
Cr-Commit-Position: refs/heads/master@{#473213}

[add] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/generate_system_health_csv
[delete] https://crrev.com/e309521b316083148b2b2073fb589a086e010bc7/tools/perf/list_system_health_stories
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/background_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/browsing_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/loading_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/long_running_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/media_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/multi_tab_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/searching_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/system_health_story.py
[add] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/system_health_stories.csv

I am not actively working on this.

Actually, I might take over this now. May be useful to also help keeping track of which stories are disabled and where.

Actually, may not have time to work on this soon after all. :)

I have a CL up for returning a json file of what benchmarks have what stories disabled on what platforms up here:
https://chromium-review.googlesource.com/c/chromium/src/+/655361

requires this cl as well:
https://codereview.chromium.org/3008203003/

assign this to Ashley to triage/prioritize

Lori is going to add presubmit checks to ensure that the generated csv file is kept up-to-date and then, using the changes listed in comment #8, update the csv generation to include a column that lists what platforms the story is disabled on (if any)

I don't think we want both (1) keep the csv checked-in and (2) add disabled stories to it.

It's going to create similar issues/conflicts to the ones we already have on the benchmark_sharding_map.json, but now for all CLs trying to disable/enable stories.

Instead of keeping the csv in the repo, could we periodically run a script that updates the list of stories to, say, a Google Sheets document?

I agree that it would be nice to setup a cron-job type script to update those csv files (benchmark_sharding_map.json still need to be keep in synced with the commits).

Though my long term vision for this is the data should be tracked by a dashboard (perf/flakiness/test tracking?), hence it may not worth investing too much if this is not a big problem. 

Also notes that the benchmarks scheduling JSON file often caused conflict because changing benchmarks reshuffle the whole file, while that's not the case with the .csv files.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/158a7a01ceb96fd019e996e3501d2fd2995bebb2

commit 158a7a01ceb96fd019e996e3501d2fd2995bebb2
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Mon Sep 11 23:09:59 2017

Roll src/third_party/catapult/ 99ec81878..c59db25d8 (5 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/99ec8187805e..c59db25d8116

$ git log 99ec81878..c59db25d8 --date=short --no-merges --format='%ad %ae %s'
2017-09-11 eakuefner [Tracing] Add add_shared_diagnostics.AddValueDiagnostics
2017-09-11 benjhayden Export {raw,merged} {csv,json} from results.html.
2017-09-11 dtu [pinpoint] Re-layout job page header.
2017-09-11 xunjieli [wpr-go] Use wprgo for test_page_sets data.
2017-09-11 rnephew [Telemetry] Expose disable_stories in StoryExpectations api.

Created with:
  roll-dep src/third_party/catapult
BUG= 753948 ,723636


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I759eca1f159ddd866b45992789ab2f39dcf8fbca
Reviewed-on: https://chromium-review.googlesource.com/660896
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501080}
[modify] https://crrev.com/158a7a01ceb96fd019e996e3501d2fd2995bebb2/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7194c04130998e5f665cb420c23e50b9e9ca9643

commit 7194c04130998e5f665cb420c23e50b9e9ca9643
Author: Lori Oliver <loridoesitbest@gmail.com>
Date: Mon Sep 25 20:32:10 2017

Move logic of generate_system_health_csv to .py file

Move logic of generate_system_health_csv to generate_system_health_csv.py in order to allow for imports.
Move into tools/perf/core

Bug: chromium:723636
Change-Id: Ied8df78b51200110f8baa61b0e7cba8a617e2b72
Reviewed-on: https://chromium-review.googlesource.com/675667
Commit-Queue: Lori Oliver <loloangela@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#504153}
[add] https://crrev.com/7194c04130998e5f665cb420c23e50b9e9ca9643/tools/perf/core/system_health_csv_generator.py
[modify] https://crrev.com/7194c04130998e5f665cb420c23e50b9e9ca9643/tools/perf/generate_system_health_csv
[modify] https://crrev.com/7194c04130998e5f665cb420c23e50b9e9ca9643/tools/perf/system_health_stories.csv

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4ced119ff5eb82dd557015f53437fe96ed6924ad

commit 4ced119ff5eb82dd557015f53437fe96ed6924ad
Author: Lori Oliver <loridoesitbest@gmail.com>
Date: Wed Sep 27 18:07:48 2017

Add disabled platforms column to go/csh-stories

- PopulateExpectations accepts a dictionary:
    all_expectations = {story_name: [[conditions], reason]}
  Returns disables dictionary:
    disables = {story_name: "Disabled Platforms"}

- generate_system_health_csv_unittest.py tests the new
PopulateExpectations function.

Bug: chromium:723636
Change-Id: Ic94b08f53c004bab5aa813399da34dd7d4b09279
Reviewed-on: https://chromium-review.googlesource.com/674099
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Lori Oliver <loloangela@google.com>
Cr-Commit-Position: refs/heads/master@{#504729}
[add] https://crrev.com/4ced119ff5eb82dd557015f53437fe96ed6924ad/tools/perf/core/generate_system_health_csv_unittest.py
[modify] https://crrev.com/4ced119ff5eb82dd557015f53437fe96ed6924ad/tools/perf/core/system_health_csv_generator.py
[modify] https://crrev.com/4ced119ff5eb82dd557015f53437fe96ed6924ad/tools/perf/system_health_stories.csv

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/461bf1f9a1d8e9e79bf6adcced3fc2e898135dc1

commit 461bf1f9a1d8e9e79bf6adcced3fc2e898135dc1
Author: rnephew <rnephew@chromium.org>
Date: Thu Oct 19 18:38:48 2017

[Telemetry] Add ability to print disabled stories to story_expectation_validator

Requires https://codereview.chromium.org/3008203003/

Bug: 723636
Change-Id: I3bda0037af48c78fd7b80912f4a84d0542a782e8
Reviewed-on: https://chromium-review.googlesource.com/655361
Commit-Queue: rnephew <rnephew@chromium.org>
Reviewed-by: rnephew <rnephew@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#510147}
[modify] https://crrev.com/461bf1f9a1d8e9e79bf6adcced3fc2e898135dc1/tools/perf/core/story_expectation_validator.py