Very good idea, cpaulin@ wrote a document [1] and might have
done some work in this area.

[1] https://docs.google.com/document/d/1gp1lgO_lZcTRlGTSNA_32pLH87VpgTHHZ0afq-Oc89E/edit

Will start working on this shortly, just need to order myself a CrOS laptop and get that set up for testing...

Re #3: Yeah, I'll mostly expand on and tweak his tests.

My dev environment is now set up after some struggles.

For this one, I might write some new tests or just invoke the tests from 628630, which should now fail a whole lot less. How do I get a test into the CQ? Is it a special test suite?

Yes. suite:bvt-cq. First a test needs to run in some other suite (e.g. bvt-perbuild) to prove its stability.

Ok. I think we should just use the existing site tests. They will fail if there are direct gUM failures. The perf values they report should not be plotted in any dashboard though, since those aren't usable unless the device has good lighting.

https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=video_WebRtcPeerConnectionWithCamera&releases=56 looks really stable; just one failure* the last couple hundred executions. That failure looks like a timeout, which is quite strange given that it's capped to 20s for the main test, and it got killed after 101s. There's some setup and teardown but those shouldn't take more than a few seconds each. 

The cam test looks all green: https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=video_WebRtcPeerConnectionWithCamera&releases=56

One problem is I'll need to find some way to make the tests only report perf if they're on an av_webcam machine, since there's no way to say "upload perf only when the test is run from suite xyz" in tko/perf_upload/perf_dashboard_config.json, AFAIK. The best thing would be if I could somehow say if on_av_webcam_suite(): report_perf() in the test. I could also duplicate the test code and make a newer small test. Maybe that's actually better, since it's good if CQ tests are simple (and there's not too much duplication, at least).

* https://wmatrix.googleplex.com/testrun/unfiltered?test_ids=377385824

Yeah, I agree with the idea of creating a new simpler test for CQ. It'll avoid a bunch of special-casing and adding pool dependencies to the test code. This will probably add some redundancy to the test suite, but I'm ok with it in this case. @rohit, let us know if you feel otherwise. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/dde68e40149c41f071437cc4b8748097c2aa23f6

commit dde68e40149c41f071437cc4b8748097c2aa23f6
Author: Patrik Höglund <phoglund@chromium.org>
Date: Fri Nov 11 09:58:47 2016

Adding WebRTC CQ test.

This test is extremely basic and basically just checks that
the gUM-backed stream starts playing. The test uses fake
devices and does not require onboard webcams.

We could make the test use real webcams as well, but I think
that sounds risky for a commit queue.

BUG= chromium:628633 
TEST=locally on a link laptop

Change-Id: I67aafe0ad99dcef7b9f0232534ff0803da51ee5e
Reviewed-on: https://chromium-review.googlesource.com/410661
Commit-Ready: Patrik Höglund <phoglund@chromium.org>
Tested-by: Rohit Makasana <rohitbm@chromium.org>
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>

[add] https://crrev.com/dde68e40149c41f071437cc4b8748097c2aa23f6/client/site_tests/video_WebRtcSanity/control
[add] https://crrev.com/dde68e40149c41f071437cc4b8748097c2aa23f6/client/site_tests/video_WebRtcSanity/video_WebRtcSanity.py
[add] https://crrev.com/dde68e40149c41f071437cc4b8748097c2aa23f6/client/site_tests/video_WebRtcSanity/getusermedia.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/dde68e40149c41f071437cc4b8748097c2aa23f6

commit dde68e40149c41f071437cc4b8748097c2aa23f6
Author: Patrik Höglund <phoglund@chromium.org>
Date: Fri Nov 11 09:58:47 2016

Adding WebRTC CQ test.

This test is extremely basic and basically just checks that
the gUM-backed stream starts playing. The test uses fake
devices and does not require onboard webcams.

We could make the test use real webcams as well, but I think
that sounds risky for a commit queue.

BUG= chromium:628633 
TEST=locally on a link laptop

Change-Id: I67aafe0ad99dcef7b9f0232534ff0803da51ee5e
Reviewed-on: https://chromium-review.googlesource.com/410661
Commit-Ready: Patrik Höglund <phoglund@chromium.org>
Tested-by: Rohit Makasana <rohitbm@chromium.org>
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>

[add] https://crrev.com/dde68e40149c41f071437cc4b8748097c2aa23f6/client/site_tests/video_WebRtcSanity/control
[add] https://crrev.com/dde68e40149c41f071437cc4b8748097c2aa23f6/client/site_tests/video_WebRtcSanity/video_WebRtcSanity.py
[add] https://crrev.com/dde68e40149c41f071437cc4b8748097c2aa23f6/client/site_tests/video_WebRtcSanity/getusermedia.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a0149436d611c3c0fe6bd28bee82e3a29ff5c351

commit a0149436d611c3c0fe6bd28bee82e3a29ff5c351
Author: Patrik Höglund <phoglund@chromium.org>
Date: Thu Nov 10 12:53:17 2016

Clean up incorrect terminology, etc, from webrtc camera test.

This is the first step before creating a new CQ-style test out
of this test. I thought I'd clean up the original test first.

Fixes bad error handling and a bug where VGA cam errors did
not even get checked (!)

BUG= chromium:628633 
TEST=locally on a link laptop.

Change-Id: I32634173dcce62af58c2de68b3b43cde0c1f89fd
Reviewed-on: https://chromium-review.googlesource.com/410680
Commit-Ready: Patrik Höglund <phoglund@chromium.org>
Tested-by: Rohit Makasana <rohitbm@chromium.org>
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>

[modify] https://crrev.com/a0149436d611c3c0fe6bd28bee82e3a29ff5c351/client/site_tests/video_WebRtcCamera/control
[modify] https://crrev.com/a0149436d611c3c0fe6bd28bee82e3a29ff5c351/client/site_tests/video_WebRtcCamera/video_WebRtcCamera.py
[rename] https://crrev.com/a0149436d611c3c0fe6bd28bee82e3a29ff5c351/client/site_tests/video_WebRtcCamera/getusermedia.html