Hi laszio@, 

I am on the Chrome Video Stack team. We would like to track our performance on ChromeOS machines as well. Do you know where I could go to see the data from media.tough_video_cases? Is this still being run continuously? It looks to be enabled here: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/media.py?l=157

+sosa@, do you know who knows about media.tough_video_cases performance tests on ChromeOS?

CC'ing Chrome OS Media folks.

Thanks for adding appropriate folks. I noticed https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py?q=tough_video_cases&dr=C&l=65

Looks like media.tough_video_cases is disabled for cros because they were taking too long? We are improving them to take less time, and then maybe they would be worth re-adding. 

Another related bug is crbug/676188: media_perftests are much less flaky than tough_video_cases, and we would like to enable them for cros.

maybe of interest to some of the ChromeOS graphics people. Adding them. 

You can add a new ChromeOS autotest to log tough_video_cases results.

Why we cannot enable tough_video_cases for CrOS like Windows or Mac? Because there's no perfbot for CrOS?

What does media.tough_video_cases do?

Correct, there is no perf bot for CrOS: https://build.chromium.org/p/chromium.perf/buildslaves

tough_video_cases plays video and audio and measures things like time_to_play, seek time, cpu usage, and memory usage.

+Ned,

do you know any context here around the lack of chromeos perf bots. Is there a standard solution to this problem of wanting coverage on CrOS?

+Annie who have better context about this than me.

+dpranke who may have more context than me.

CrOS has their own perf labs, they select which benchmarks to run in them and they don't use the perf waterfall. bccheng might have more context on which people on CrOS side are interested in media benchmarks?

We, CrOS video team, do want chromeos perf bots in perf waterfall. Otherwise we need to add perf autotests and select every benchmark manually. I think CrOS perf labs are suitable for perf test that we want to run on every ChromeOS device. For generic media benchmarks (like tough_video_cases), it will be great if there's at least one or two chromeos perf bots in perf waterfall.

We are working on adding perf bots for Chrome on ChromeOS. Here is the WIP design doc: https://docs.google.com/a/google.com/document/d/1UYMVx3URISEIpxmCNx2os7_6sFMtndXwgDncb4RBn74/edit?usp=sharing

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c52544194f2bf97e72cf9215f2c3d40928929036

commit c52544194f2bf97e72cf9215f2c3d40928929036
Author: Caleb Rouleau <crouleau@google.com>
Date: Tue May 09 17:51:57 2017

Merge tough_video_cases_extra into tough_video_cases.

Fix duplicate display name conflict by adding &seek to end of every url
for a seeking Page.

Delete ChromeOS-specific benchmarks as well since they are not being
used anymore (crbug/709161). I would recommend CrOS team to simply use
our normal media.tough_video_cases benchmark in the future instead of
creating their own. This is more reasonable now because of the huge
reduction in runtime of tough_video_cases (crbug/710253).

Original codereview is at https://codereview.chromium.org/2866703004/

Add tag for normal_play.

Bug: 709161, 713335,  565180 
Change-Id: I2f665a2e3d8e64c46ecc38860b0e27cf6251d984
Reviewed-on: https://chromium-review.googlesource.com/499989
Commit-Queue: Caleb Rouleau <crouleau@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#470375}
[modify] https://crrev.com/c52544194f2bf97e72cf9215f2c3d40928929036/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/c52544194f2bf97e72cf9215f2c3d40928929036/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/c52544194f2bf97e72cf9215f2c3d40928929036/tools/perf/benchmark.csv
[modify] https://crrev.com/c52544194f2bf97e72cf9215f2c3d40928929036/tools/perf/benchmarks/media.py
[modify] https://crrev.com/c52544194f2bf97e72cf9215f2c3d40928929036/tools/perf/page_sets/tough_video_cases.py

I'm not working on this. Maybe bccheng@chromium.org can take it?

We need the waterfall to use the simplechrome flow to build Chrome for ChromeOS first.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Bhaskar, can you triage this or dupe it to whatever existing bug you have? It seems pretty important to have performance testing coverage of ChromeOS. Telemetry already has a ton of test cases that would apply to ChromeOS. If I'm reading the code correctly, these cases are not being run right now.

Assign to CY, whose team does performance testing for ChromeOS currently, they have a good idea of what tests are not being run. 

From Issue 814935, you should be able to talk to Ben (bpastene@) to see if there is any cycle for running chromium.perf tests on those kevin DUTs in the lab. For running the test on all models in CrOS lab, could you provide how long it will take to run the tough_video_cases? Based on the runtime, we might be able to run it with nightly/perbuild suite accordingly.

This doc contains runtimes for the various benchmarks: https://docs.google.com/spreadsheets/d/1Vi37tr3kmhWQ07LYogpaAcvzxrOxoUuHNOplQEfFdFc/edit#gid=2092096275

media.tough_video_cases benchmark has been renamed to "media.desktop". That particular benchmark takes 684 seconds to run on Windows (and ChromeOS runtime may be similar). It would be great to run *all* the benchmarks on ChromeOS if possible. We already run all of them on Windows, Mac, Linux, and Android.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/17391ebbf9ab8c7bcc34bad45cb69cb65c7fbd20

commit 17391ebbf9ab8c7bcc34bad45cb69cb65c7fbd20
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Fri Jan 04 17:08:05 2019

Adding telemetry_Benchmarks.media.desktop

BUG=chromium:709161
TEST=run this telemetry benchmark

Change-Id: I81815b9eccc7bdb64c6530afd2aa919676fbf333
Reviewed-on: https://chromium-review.googlesource.com/1394603
Commit-Ready: Vovo Yang <vovoy@chromium.org>
Tested-by: Vovo Yang <vovoy@chromium.org>
Reviewed-by: Chung-yih Wang <cywang@chromium.org>

[add] https://crrev.com/17391ebbf9ab8c7bcc34bad45cb69cb65c7fbd20/server/site_tests/telemetry_Benchmarks/control.media.desktop
[modify] https://crrev.com/17391ebbf9ab8c7bcc34bad45cb69cb65c7fbd20/server/site_tests/telemetry_Benchmarks/generate_controlfiles.py

CrOS telemetry_Benchmarks.media.desktop test results:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/testDetails?testName=telemetry_Benchmarks.media.desktop