Sorry, when I said "isn't running", I meant, "isn't working" or "isn't producing results". All values on the dashboard are zero.

Would like to keep it since it's the only set of tests telling us how we're doing in an increasingly mobile world. These tests have always been too big to monitor manually, they instead were expected to generate automated alerts when things went wrong, but test harnesses keep changing :/

crouleau@: do you know anything about these? Looks like the cns server code is in our repo in media/tools/. Is it supposed to be running somewhere?

I agree with Dale. We should fix these tests and re-enable them. Maybe we can get a Q1 KR for this. Since chromeperf alerts were turned off for a long time before I joined (we still don't know how they got turned off), we probably never got notified that the tests stopped working. As Dale said, the value is in the alerts, not in people manually checking the results.

+John Pallett for OKR prioritization. 

I don't know anything about these tests or the cns server code. That said, I would be happy to dig into it at some point. Q1 is getting a bit full for me, but I'll see if I can figure out something. I'm setting this to available in case someone else wants to pick it up. I don't personally have the cycles for it until sometime in late Q1 (unless we decide to prioritize it above other things). I'll see if I can get someone else to work on this.

It's also worth figuring out whether telemetry supports this use case natively yet, so we don't have to maintain our own service.

I'm disabling these now. Then we can work to reenable them later, after John migrates tough_video_cases to TBMv2.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4207948af983aac15311f65b7d8b7b23c69614e9

commit 4207948af983aac15311f65b7d8b7b23c69614e9
Author: crouleau <crouleau@chromium.org>
Date: Wed Apr 26 20:04:23 2017

Disable media_cns_cases, add owners and components to all media benchmarks.

BUG= 676345 , 710253 

Review-Url: https://codereview.chromium.org/2842223002
Cr-Commit-Position: refs/heads/master@{#467422}

[modify] https://crrev.com/4207948af983aac15311f65b7d8b7b23c69614e9/tools/perf/benchmarks/media.py

I am actively working on this. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/adc394765fa9c04984a96cd3a147305174dd6c71

commit adc394765fa9c04984a96cd3a147305174dd6c71
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Wed Nov 01 18:20:38 2017

Add simple constrained network case to media Telemetry tests.

Remove legacy broken media_cns_cases. Also remove media measurements
since media_cns_cases was the only user of it.

Bug:  676345 
Change-Id: If3659e3ba0d57f6ef1a73d55ef4022a3f5bf797f
Reviewed-on: https://chromium-review.googlesource.com/747820
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Caleb Rouleau <crouleau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#513181}
[modify] https://crrev.com/adc394765fa9c04984a96cd3a147305174dd6c71/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/adc394765fa9c04984a96cd3a147305174dd6c71/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/adc394765fa9c04984a96cd3a147305174dd6c71/tools/perf/benchmark.csv
[modify] https://crrev.com/adc394765fa9c04984a96cd3a147305174dd6c71/tools/perf/benchmarks/media.py
[delete] https://crrev.com/ed19739505b46b654f916cf1ae7f0ff28127cd58/tools/perf/measurements/media.py
[delete] https://crrev.com/ed19739505b46b654f916cf1ae7f0ff28127cd58/tools/perf/page_sets/media_cns_cases.py
[modify] https://crrev.com/adc394765fa9c04984a96cd3a147305174dd6c71/tools/perf/page_sets/tough_video_cases.py
[modify] https://crrev.com/adc394765fa9c04984a96cd3a147305174dd6c71/tools/perf/page_sets/tough_video_cases/video.html

Done. There is a single CNS case added to media telemetry cases and the old media_cns_cases benchmark is removed.

Can you share a link to the new benchmark?

There isn't a new benchmark. We just added this test case (Called a "Page" or "Story" in Telemetry vernacular) to the existing media.desktop and media.mobile TBMv2 benchmarks.

Here's a graph: https://chromeperf.appspot.com/report?sid=9170f93c7142745d89e4d99bf58e5b27e80e242e4e3e3f719119b4986e263a12
You can select different metrics, bots for that same page.

Looks like Telemetry team inserts a subtest folder called "Regular-3G" for these tests, which I totally didn't expect, but the test is still usable... +ned

To run the test use the following command in a chromium checkout

$ src/tools/perf/run_benchmark media.desktop --browser=list

then pick the browser you want from the list. I chose "dev"

$ src/tools/perf/run_benchmark media.desktop --browser=dev --story-tag-filter=cns

You can also run it on android, on a trybot, and in a bisect.  

The code for it is in https://cs.chromium.org/chromium/src/tools/perf/page_sets/tough_video_cases.py (Note this code is moving soon, if the link doesn't work I can send you a new one). It was a trivial change since Telemetry now has constrained network support built in. 

(The media.desktop and media.mobile benchmarks are the only media Telemetry benchmarks now per issue 713335. We had to merge media.mse_cases, media.media_cns_cases, media.tough_video_cases, media.android.tough_video_cases, media.tough_video_cases_extra, and media.android.tough_video_cases_extra benchmarks all into those two. media.desktop and media.mobile run the same test cases, but we add a couple flags to the media.mobile benchmark so autoplay video works.)

Did we only add this for the tulip2.vpm_webm test? That's the only one I see.

Yep. I just added it for that one as a proof of concept and so we can have basic coverage. I figured we would want to discuss which cases to add in detail before I add a bunch (or convert some of the normal tests into CNS tests since we don't want to overload the perf waterfall.)

 issue 781899  should have fixed the problem where a subtest folder is inserted for the new constrained network test case. Check back tomorrow to see if the new data is there.