emircan@, can you point me to a detailed summary of what the bug is about? What are the gritty technical details here? If we assume the dropped-call metrics measure correctly (which I doubt), would a 720p 45-second WebRTC loopback call in a single tab, with the spinning green ball fake video, be enough to detect the regression?

The original CL that caused the regression -https://codereview.chromium.org/1737253002- had an impact because it marked video layer as non-opaque for rendering in order to accomodate for a new feature. It impacted all OSs, but the effects wouldn't be equal on all as it is based on rendering performance. It does not relate to decoder or camera. It relates to how remote video streams are rendered on the page, so it makes sense that it scales with the number of connections. My mistake here was to assume that this change wouldn't have a performance impact as camera streams always have opaque content. Although camera's stream content is opaque, marking the layer as non-opaque makes a difference in rendering according to the metrics. 

We asked marcheu@ about why the effects would be different on platforms to understand why the current tests didn't catch it. He said that on high end devices the impact isn't visible since there is enough bandwidth and GPU horsepower to hide this cost. So far, we know that this only caused difference in panther CrOS device in kiosk mode. Since panther was already running nearly with the full cpu usage; this dropped some extra frames, caused cpu adaptation, dropped frame rate/resolution and made it more visible. Clear problem here is that we do not have any automated testing on CrOS devices with ~10 connections in kiosk mode. But even if we had a test like you described above, simply checking cpu usage or dropped metrics might not be the clear as adaptation and HW performance would interfere. For instance, if panther had more room for cpu usage, we might not see any dropped frames.

Your description above suggests there were small changes visible on all platforms, but that their compounded effect happened to show in low-powered CrOS devices. 

What if we measured CPU and GPU usage for a WebRTC call, would we be able to pick this difference up? Is it so miniscule we need several video tags rendering at the same time to be able to spot a difference? Is it CPU/GPU we should even measure, or something else? What if we have one peer connection rendering to 100 video tags in a demo page, will that compound the effect?

I wonder if we can detect those small differences. The goal here is to construct the simplest test possible that would have caught this issue.

Yes, compounded effect happened to show on CrOS panther. According to logs in bisect, the dropped frame metric scales with number of connections. I am not sure about having one peer connection with multiple video tags in case they have rendering optimizations that we aren't aware of. We can test and see.

I cannot think of any other metric than CPU/GPU usage for measuring performance here. We need to be aware of CPU adaptation kicking in, but then frame rate or resolution would change. All these combined should cover it.

Ok, so maybe let's try CPU/GPU and dropped frame with a bunch of peer connections in a test page. Can I simply revert the fixes in  https://crbug.com/647886  to reproduce the regression locally? Do I need to do anything else?

Yes, there is only one Cl for fix actually. Reverting https://codereview.chromium.org/2348903003 would be enough.

Edward is working on a test now.

For a first attempt, he has created a page which creates n peerconnections and n*2 video tags. This currently gives a nice signal pre-bug compared to post-bug Chrome: about 10% CPU usage. Edward, can you elaborate? What's the GPU usage difference by the way?

I also have an idea to create a similar page with one pc, but n streams and n video tags. This is more similar to a real load in our "downstream product".

So, in a test page with 10 peerconnections and 20 video tags running a videocall for 45s, I got a CPU usage of ~44% and a GPU usage of ~11% with the bug, and of ~12% CPU usage and ~6% GPU usage without the bug.
I ran it three times with and without the bug yesterday, and three times with and without the bug today, and the results seem to be stable.

The 10% difference was obtained with a different number of peerconnections (which I don't remember).

Sweet! Ship it!

Like we discussed offline, let's get this page in the demo github repo (Christoffer can help you), pull it into the Telemetry page set, and start running it in our telemetry tests.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d38116383dc36f6be57077b86e3deaaae6cba20

commit 1d38116383dc36f6be57077b86e3deaaae6cba20
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Wed Nov 02 09:31:00 2016

Add a new telemetry benchmark to stress-test WebRTC.

BUG= 651820 
NOTRY=True
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq

Review-Url: https://codereview.chromium.org/2463013003
Cr-Commit-Position: refs/heads/master@{#429244}

[modify] https://crrev.com/1d38116383dc36f6be57077b86e3deaaae6cba20/tools/perf/benchmarks/webrtc.py
[modify] https://crrev.com/1d38116383dc36f6be57077b86e3deaaae6cba20/tools/perf/measurements/webrtc.py
[add] https://crrev.com/1d38116383dc36f6be57077b86e3deaaae6cba20/tools/perf/page_sets/data/webrtc_stresstest_cases.json
[add] https://crrev.com/1d38116383dc36f6be57077b86e3deaaae6cba20/tools/perf/page_sets/data/webrtc_stresstest_cases_000.wpr.sha1
[modify] https://crrev.com/1d38116383dc36f6be57077b86e3deaaae6cba20/tools/perf/page_sets/webrtc_cases.py

Cool! Here are some preliminary results:
https://chromeperf.appspot.com/report?sid=18bfbd3c9e3bc99a2bed0ecc8e038d8057fcc961774da2b2f7d3e0c70c85de53

We can see the stress test (45s) uses 12x more CPU compare to the 720p 45s call for the renderer process. It also uses a lot more GPU. For the browser process the stress test uses less for some reason. Well, the renderer is the interesting thing here anyway.

I'm going to get the new test running on CrOS devices as well, maybe we can see an even more pronounced effect there.

Here are some requirements we should aim for:

* It should be possible to disable CPU adaptation
* Input streams should be realistic; movement and colors. Ideally a recording should be used
* We need to be able to track test results over time on a dashboard

What to monitor in the test:
* FPS sent/received
* Resolution
* Dropped frames

I think we have everything in the current incarnation of the stress test, except 1) disabling CPU adaptation and 2) realistic input video. I think the former is a simple change: we need to pass the peerconnection constraint googCpuOveruseDetection and set it to false. We can add a checkbox to the demo page and make sure the test unchecks it before starting. The latter is harder though.

Edward, can you add googCpuOveruseDetection support to the test page? Looks like you need something like in this page: https://test.webrtc.org/manual/peer2peer/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/967024f84d0fc6ada6db5bace52e7b6ca027669b

commit 967024f84d0fc6ada6db5bace52e7b6ca027669b
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Tue Nov 08 17:21:55 2016

Update telemetry benchmark to stress-test WebRTC.

Added support for disabling googCpuOveruseDetection

R=phoglund@chromium.org
BUG= 651820 

Review-Url: https://codereview.chromium.org/2483233002
Cr-Commit-Position: refs/heads/master@{#430643}

[modify] https://crrev.com/967024f84d0fc6ada6db5bace52e7b6ca027669b/tools/perf/page_sets/data/webrtc_stresstest_cases.json
[add] https://crrev.com/967024f84d0fc6ada6db5bace52e7b6ca027669b/tools/perf/page_sets/data/webrtc_stresstest_cases_001.wpr.sha1
[modify] https://crrev.com/967024f84d0fc6ada6db5bace52e7b6ca027669b/tools/perf/page_sets/webrtc_cases.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/938f798e4a5c5a55c893d8a6c8068ba54c7cb6cf

commit 938f798e4a5c5a55c893d8a6c8068ba54c7cb6cf
Author: Patrik Höglund <phoglund@chromium.org>
Date: Wed Nov 02 16:30:03 2016

Add webrtc.stress telemetry tests.

BUG= chromium:651820 
TEST=locally on link laptop

Change-Id: I1a12af107e5b9bb94ceb2af57ae2064a34cc0fd7
Reviewed-on: https://chromium-review.googlesource.com/409416
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>
Reviewed-by: Patrik Höglund <phoglund@chromium.org>
Tested-by: Rohit Makasana <rohitbm@chromium.org>

[add] https://crrev.com/938f798e4a5c5a55c893d8a6c8068ba54c7cb6cf/server/site_tests/telemetry_Benchmarks/control.webrtc.stress
[modify] https://crrev.com/938f798e4a5c5a55c893d8a6c8068ba54c7cb6cf/tko/perf_upload/perf_dashboard_config.json
[modify] https://crrev.com/938f798e4a5c5a55c893d8a6c8068ba54c7cb6cf/server/site_tests/telemetry_Benchmarks/generate_controlfiles.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/2ef2e178bd89e509251bce9a8de627726d8e6866

commit 2ef2e178bd89e509251bce9a8de627726d8e6866
Author: Patrik Höglund <phoglund@chromium.org>
Date: Wed Nov 16 09:07:29 2016

Add webrtc.stress to whitelist.

BUG= chromium:651820 
TEST=can't test, config change

Change-Id: Ie811095628ddba6a1fe3348c88534b51c73b072c
Reviewed-on: https://chromium-review.googlesource.com/411881
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>
Commit-Queue: Rohit Makasana <rohitbm@chromium.org>
Tested-by: Rohit Makasana <rohitbm@chromium.org>

[modify] https://crrev.com/2ef2e178bd89e509251bce9a8de627726d8e6866/server/cros/telemetry_runner.py

Ok! The test is now running on CrOS devices. It's especially interesting on panther and buddy, which are Chromebox-for-meetings devices. Edward, did you verify the test still would have caught the issue, after the revisions we made? 

If so, the only thing remaining is to get tests under monitoring. I think we might want to monitor only buddy, panther and a couple more boards here, to limit the impact of false positives.

After verifying that the webrtc.stress test would have caught the issue, we've got both webrtc.peerconnection and webrtc.stress metrics under monitoring on the sheriff rotation for windows 7, mac retina, nexus 5x, linux, buddy, panther and auron_paine.