This is a Telemetry bug, please help to move it

+telemetry folk

If you need any more information for this issue, please feel free to comment.
This issue block us to measure the performance of playing 4K videos in Telemetry

Ned, can you triage?

All your browser flags look fine.

How often are you calling updateStats? I don't see any reference to it other than its declaration. It looks like at lot of the code you have written is missing from your post. If you are calling updateStat frequently, and that function is causing updates to the page, then I could see that being an issue.

We have media.tough_video_cases benchmark already, which measures dropped frame count. If you could use that benchmark instead, it would be better since we already do the work to verify and maintain that.

Feel free to re-assign to me if you have further questions.

var updateInterval = 2000;
var videoStarted = function() {
        if (startTime == 0){
            startTime = getCurrentTime();
            updateIntervalId = setInterval(updateStats, updateInterval);
        }
    };

We invoke the method with 2 seconds interval.
I am not sure if it is too frequently.

Could you share more information about media.tough_video_cases. I see it in the telemetry benchmark list. does it measure the FPS for playing 1080p-VP9 and 4K-VP9 steaming

I don't have enough information to reproduce this bug, so it's really hard for me to say what's going wrong. However, I wonder if we could take a step back and understand what you are trying to measure so I can recommend the most straight-forward way of doing that.

I'm confused about measuring FPS. Whenever possible, Chrome tries to play at the rate that the video requires. So it plays a 30 fps video at 30 fps. If it starts to have trouble decoding and presenting the video quickly enough, it will start dropping frames. So usually we measure dropped frame count rather than FPS, since FPS should be constant.

However, there is something called "clockless_playback" which decodes the video as quickly as possible so you can see how fast decode will work. We have tests at chromium/src/media/test/pipeline_integration_perftest.cc which measure this. I think it would be pretty simple to just add whatever videos you want to that test case and then run it. The advantage of those tests is that they are not Telemetry tests, so they are much less flaky. Here's the graph of one of them: https://chromeperf.appspot.com/report?sid=45aa0b5f2dcfb929ddca79492f1f5a8290b055c19ca4b1f594e0736669203596 . Please let me know if you need help running these tests.

It seems like you're probably interested in decode rates for VP9? +jaikk@ is PM for VP9/WebM so he could probably point you to people who have helpful tests for this.

+chcunningham@ is working on media capabilities API, which surfaces expected decode rate information to websites. Maybe this could be helpful as well?

+ericde@ does lots of communication with Intel related to video decode rates.

If you're confident that this is the method that you want to use and you just need to fix this bug, then please provide step-by-step instructions for reproduction.

media.tough_video_cases measures dropped_frame_count, but it doesn't measure FPS (since this would always be the same value). You can run it using these instructions: https://sites.google.com/a/chromium.org/dev/developers/telemetry/run_locally

I have forwarded your comments to our PnP benchmark designer. Let's wait for their comments about the benchmark "media.tough_video_cases". 
For the step-by-step instructions for reproduction, I am still checking with our workload server maintainer. not sure if the server can be shared or not.
If it can not be shared, I will check if I can share a brief one for easily reproduction.

Sorry for so late to response. 
I don't get the response from our PnP benchmark designer.

And for reproducing this issue, I create a nodejs server and will send it to you in mail.

It is a server based on nodejs. please follow the README to deploy it.
and the workload for VP94k 30 and 60 FPS streaming is provide on 7000 port.

And the updated benchmark script is also included. 
Please use it to reproduce this issue. and this issue still happen on R62 image.

Thanks for sending me the code. I'm busy with another bug right now, but I'll try to take a crack at this later this week.

Did the benchmark designer explain the idea of "decode rate"? What about using chromium/src/media/test/pipeline_integration_perftest.c instead of making your own tests? Or you could use media.tough_video_cases_tbmv2 instead?

One concern is that you seem to be using the deprecated Telemetry measurement system. The new Telemetry tests gather data from Chrome Trace events rather than by querying JS APIs. +ned may be able to point you to documentation for the new tests.

Thanks for working with us on this!

Thanks a lot for checking this bug.
The benchmark script has been used for a long while (more than 2 years). and we still need to run it for measuring if a new regression happen on some old device with the latest image (some device is released 1 or 2 years ago).
and of course, we are interested in the new Telemetry measurement system. Really appreciate if you can give me an example for the new measurement. 
About "decode rate" and the benchmark "media.tough_video_cases_tbmv2", I don't get the response from our designer yet, I will raise this issue in our next pnp sync meeting.

Ah, that makes sense that this is an existing benchmark. This is still on my radar, and I will try to look into it soon.

give to caleb(crouleau@) to keep monitoring it.

1. You say "In manual testing, the FPS is normal (on my device, the FPS is 30). In telemetry testing, the FPS is only 9. and we can see a lot of lag during playing the 4K video." but the id listed in the benchmark is "130328" which links to a 60fps video.

2. Does this also break for you on other OSes? linux, windows?

I have the server up and running though :)

3. You're using the same device for the manual testing and the automated testing, right?

- unrelated people
+ cc bug author.

My CrOS device drops roughly half the frames for id=130328 for a manual test. I'm trying to figure out a set up for a Telemetry test, but I'm having some trouble trying to figure out how to use a remote chromebook since I have never done it.

+ ned is there a guide to using Telemetry for ChromeOS?

+achuith can help with running Telemetry on ChromeOS device

Sorry, just see your comments. Yes, we use the same device in Auto and manual testing

>>> Does this also break for you on other OSes? linux, windows?
We only run Telemetry for ChromeOS. I don't known if the issue happen on other platform

>>>>My CrOS device drops roughly half the frames for id=130328 for a manual test. I'm trying to figure out a set up for a Telemetry test, but I'm having some trouble trying to figure out how to use a remote chromebook since I have never done it.

You can deploy the benchmark script into <Chromium>/src/tools/perf/benchmarks/. and execute it via the command as below

./run_benchmark --remote=IP_OF_THE_DEVICE --browser=cros-chrome <Benchmark_Name>

That command doesn't work for me because of Google Corporate network restrictions. I may need to do some work to set up a linux box in a test network... It would be really helpful if you could try to reproduce the issue on Linux or Mac or Windows since that would be much easier for me to test on.

I'm blocked waiting for help from achuith@ or getting feedback from shaofeng@ for what happens when reproing on non-cros device.

Sorry, I never run Telemetry on other platform before.

just do

./run_benchmark <Benchmark_Name> --browser=list

That will list all the browsers you can use. Pick one. (I picked "stable" in below command).

./run_benchmark <Benchmark_Name> --browser=stable

This assumes you have Chrome installed on the machine with the Telemetry checkout.

Hi,

Sorry for the delayed response.

As suggested, I tried a couple of things and details are as:
1. I tried to execute ./run_benchmark --remote=IP_OF_THE_DEVICE --browser=cros-chrome streamingVP94K, as a result I always see an error.
"EvaluateException: UncaughtTypeError:
TypeError: Cannot read property 'innerHTML' of null"
Browser fails to load the URL.

2. I also tried ./run_benchmark streamingVP94K --browser=stable and the result is the same as above.

As a result can we conclude that streaming test in telemetry is not working with any of the platform?

Please let me know if you need any more information from my end.
I have also attached result file generated for reference.

Note: I am testing on R64 Beta.

Deleted: Vp9_4k.txt 3.4 KB

Vp9_4k.txt 3.4 KB View Download

Deleted: results.html 1.1 MB

results.html 1.1 MB View Download

Ned, can you please help with Telemetry issues or delegate to someone on your team?

I pass this to to cros folks

Rachna inform me, she has another way to measure this KPI now. and this bug can be closed now. Thanks a lot to all of your effort.

The bug can be closed now , since there is another way to measure the KPI.