Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 3 users
Status: Assigned
Owner:
Last visit 22 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment
Autogenerate tracking document for system health stories
Project Member Reported by nedngu...@google.com, May 17 Back to list
We generally don't do a good job of manually update tracking sheet (e.g: benchmarks, system health stories).

So I would want to auto-generate system health stories tracking sheet for https://bit.ly/csh-stories similar to what we did for chrome benchmarks sheet.

Ideas of what we would auto generate:
1) Story name
2) Which platform it run on
3) Description (pulled out from pydoc by default or by overriding GetStoryDescription() for things like loading stories)
4) Its tags (probably one per column so people can easily sort/filter which areas are covered).
 
Cc: rnep...@chromium.org
Owner: nedngu...@google.com
Status: Started
Maybe also include which stories are disabled an on which platforms.

Project Member Comment 3 by bugdroid1@chromium.org, May 19
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b087b7731c2efa0c446d4b52ea0839487cce4e1d

commit b087b7731c2efa0c446d4b52ea0839487cce4e1d
Author: nednguyen <nednguyen@google.com>
Date: Fri May 19 16:52:22 2017

Modify list_system_health_stories to generate_system_health_csv

This CL also check in system_health_stories.csv file, which will be used to later
replace https://bit.ly/csh-stories

BUG=723636

Review-Url: https://codereview.chromium.org/2888133002
Cr-Commit-Position: refs/heads/master@{#473213}

[add] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/generate_system_health_csv
[delete] https://crrev.com/e309521b316083148b2b2073fb589a086e010bc7/tools/perf/list_system_health_stories
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/background_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/browsing_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/loading_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/long_running_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/media_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/multi_tab_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/searching_stories.py
[modify] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/page_sets/system_health/system_health_story.py
[add] https://crrev.com/b087b7731c2efa0c446d4b52ea0839487cce4e1d/tools/perf/system_health_stories.csv

Project Member Comment 4 by sheriffbot@chromium.org, Jul 17
Labels: Hotlist-Google
Owner: ----
Status: Available
I am not actively working on this.
Owner: perezju@chromium.org
Status: Started
Actually, I might take over this now. May be useful to also help keeping track of which stories are disabled and where.
Owner: ----
Status: Available
Actually, may not have time to work on this soon after all. :)
I have a CL up for returning a json file of what benchmarks have what stories disabled on what platforms up here:
https://chromium-review.googlesource.com/c/chromium/src/+/655361

requires this cl as well:
https://codereview.chromium.org/3008203003/
Owner: ashleymarie@chromium.org
Status: Assigned
assign this to Ashley to triage/prioritize
Owner: loloangela@google.com
Lori is going to add presubmit checks to ensure that the generated csv file is kept up-to-date and then, using the changes listed in comment #8, update the csv generation to include a column that lists what platforms the story is disabled on (if any)
I don't think we want both (1) keep the csv checked-in and (2) add disabled stories to it.

It's going to create similar issues/conflicts to the ones we already have on the benchmark_sharding_map.json, but now for all CLs trying to disable/enable stories.

Instead of keeping the csv in the repo, could we periodically run a script that updates the list of stories to, say, a Google Sheets document?
Cc: sullivan@chromium.org
I agree that it would be nice to setup a cron-job type script to update those csv files (benchmark_sharding_map.json still need to be keep in synced with the commits).

Though my long term vision for this is the data should be tracked by a dashboard (perf/flakiness/test tracking?), hence it may not worth investing too much if this is not a big problem. 

Also notes that the benchmarks scheduling JSON file often caused conflict because changing benchmarks reshuffle the whole file, while that's not the case with the .csv files.
Project Member Comment 14 by bugdroid1@chromium.org, Sep 11
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/158a7a01ceb96fd019e996e3501d2fd2995bebb2

commit 158a7a01ceb96fd019e996e3501d2fd2995bebb2
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Mon Sep 11 23:09:59 2017

Roll src/third_party/catapult/ 99ec81878..c59db25d8 (5 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/99ec8187805e..c59db25d8116

$ git log 99ec81878..c59db25d8 --date=short --no-merges --format='%ad %ae %s'
2017-09-11 eakuefner [Tracing] Add add_shared_diagnostics.AddValueDiagnostics
2017-09-11 benjhayden Export {raw,merged} {csv,json} from results.html.
2017-09-11 dtu [pinpoint] Re-layout job page header.
2017-09-11 xunjieli [wpr-go] Use wprgo for test_page_sets data.
2017-09-11 rnephew [Telemetry] Expose disable_stories in StoryExpectations api.

Created with:
  roll-dep src/third_party/catapult
BUG= 753948 ,723636


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I759eca1f159ddd866b45992789ab2f39dcf8fbca
Reviewed-on: https://chromium-review.googlesource.com/660896
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501080}
[modify] https://crrev.com/158a7a01ceb96fd019e996e3501d2fd2995bebb2/DEPS

Project Member Comment 15 by bugdroid1@chromium.org, Sep 25
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7194c04130998e5f665cb420c23e50b9e9ca9643

commit 7194c04130998e5f665cb420c23e50b9e9ca9643
Author: Lori Oliver <loridoesitbest@gmail.com>
Date: Mon Sep 25 20:32:10 2017

Move logic of generate_system_health_csv to .py file

Move logic of generate_system_health_csv to generate_system_health_csv.py in order to allow for imports.
Move into tools/perf/core

Bug: chromium:723636
Change-Id: Ied8df78b51200110f8baa61b0e7cba8a617e2b72
Reviewed-on: https://chromium-review.googlesource.com/675667
Commit-Queue: Lori Oliver <loloangela@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#504153}
[add] https://crrev.com/7194c04130998e5f665cb420c23e50b9e9ca9643/tools/perf/core/system_health_csv_generator.py
[modify] https://crrev.com/7194c04130998e5f665cb420c23e50b9e9ca9643/tools/perf/generate_system_health_csv
[modify] https://crrev.com/7194c04130998e5f665cb420c23e50b9e9ca9643/tools/perf/system_health_stories.csv

Project Member Comment 16 by bugdroid1@chromium.org, Sep 27
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4ced119ff5eb82dd557015f53437fe96ed6924ad

commit 4ced119ff5eb82dd557015f53437fe96ed6924ad
Author: Lori Oliver <loridoesitbest@gmail.com>
Date: Wed Sep 27 18:07:48 2017

Add disabled platforms column to go/csh-stories

- PopulateExpectations accepts a dictionary:
    all_expectations = {story_name: [[conditions], reason]}
  Returns disables dictionary:
    disables = {story_name: "Disabled Platforms"}

- generate_system_health_csv_unittest.py tests the new
PopulateExpectations function.

Bug: chromium:723636
Change-Id: Ic94b08f53c004bab5aa813399da34dd7d4b09279
Reviewed-on: https://chromium-review.googlesource.com/674099
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Lori Oliver <loloangela@google.com>
Cr-Commit-Position: refs/heads/master@{#504729}
[add] https://crrev.com/4ced119ff5eb82dd557015f53437fe96ed6924ad/tools/perf/core/generate_system_health_csv_unittest.py
[modify] https://crrev.com/4ced119ff5eb82dd557015f53437fe96ed6924ad/tools/perf/core/system_health_csv_generator.py
[modify] https://crrev.com/4ced119ff5eb82dd557015f53437fe96ed6924ad/tools/perf/system_health_stories.csv

Project Member Comment 17 by bugdroid1@chromium.org, Yesterday (39 hours ago)
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/461bf1f9a1d8e9e79bf6adcced3fc2e898135dc1

commit 461bf1f9a1d8e9e79bf6adcced3fc2e898135dc1
Author: rnephew <rnephew@chromium.org>
Date: Thu Oct 19 18:38:48 2017

[Telemetry] Add ability to print disabled stories to story_expectation_validator

Requires https://codereview.chromium.org/3008203003/

Bug: 723636
Change-Id: I3bda0037af48c78fd7b80912f4a84d0542a782e8
Reviewed-on: https://chromium-review.googlesource.com/655361
Commit-Queue: rnephew <rnephew@chromium.org>
Reviewed-by: rnephew <rnephew@chromium.org>
Reviewed-by: Juan Antonio Navarro PĂ©rez <perezju@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#510147}
[modify] https://crrev.com/461bf1f9a1d8e9e79bf6adcced3fc2e898135dc1/tools/perf/core/story_expectation_validator.py

Sign in to add a comment