We can use vivid as a video capture device on betty by the following steps:

1. Add vivid as kernel module for betty.
2. Build and boot ChromeOS by following https://chromium.googlesource.com/chromiumos/docs/+/master/cros_vm.md#launch-a-locally-built-vm-from-within-the-chroot
3. SSH into the VM and run `modprobe vivid n_devs=1 node_types=0x1`

Then, ChromeOS regards /dev/video0 as a normal video input device. (See screenshot)

For the first step, I used kernelconfig scripts described in http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/kernel-configuration . But, it just adds 'CONFIG_V4L2_MEM2MEM_DEV=m', 'CONFIG_VIDEO_VIM2M=m', 'CONFIG_VIDEO_VIVID=m' and 'CONFIG_VIDEO_VIVID_MAX_DEVS=64' into third_party/kernel/v4.4/chromeos/config/x86_64/chromiumos-x86_64.flavour.config .
(CONFIG_V4L2_MEM2MEM_DEV and CONFIG_VIDEO_VIM2M might be unnecessary)

How can we add automatic tests using vivid?

Deleted: ktnA4pcH2ti.png 175 KB

	ktnA4pcH2ti.png 175 KB View Download

I confirmed that vivid works well for camera_V4L2, video_WebRtcCamera, and video_WebRtcPeerConnectionWithCamera.vp8.

I executed autotests using USB camera by the following steps:
1. Set up VM by following the comment #4
2. In VM, run `echo -e "# camera\n- usb_camera" > /usr/local/etc/autotest-capability/00-betty-capabilities.yaml`
3. Run `test_that` from chroot

To run autotest automatically without manual operation like the above, I think we need to modify overlay for betty like the following:
https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-betty-private/+/643548
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1109803
(but, these are incomplete yet)

To run the VM test on the chromium waterfall we'll need to set up a special config file for vivid so that the USB camera HAL v3 knows how to open the virtual device.

Daniel / Henry / Shik might have some ideas about how to create the special config. Usually we need to set up a symlink based on the USB vid:pid of the webcam for the camera HAL to locate the camera. Since vivid does not give us a USB device we'll need to find another way.

I think we should be able to create the symlink based on platform device name, like with V4L2 codecs, right?

Do we need to make it run in LIMITED mode? I think after external camera landed, this would be detected as an external camera automatically. By the way, the symlink mechanism would be removed in USB camera HAL v3 as it uses libudev to enumerate video4linux devices directly.

I assume vivid simply emulates a webcam with a fake video capture device. Please correct me if I'm wrong. In this case there's no USB vid:pid info and some of the UVC controls might not work either, and we'll need to treat it as a special device. It should still provide a decent coverage of the CrOS VideoCaptureDevice stack.

Ideally we'd like to have vivid in the Chromium CQ, so that the appropriate
unittests (media_unittests, video_capture_unittests, service_unittests or
whichever), can be run in linux-chromeos-rel. 

bpastene@ is working in crbug.com/732531 in extending the ChromeOS VM and
tests _in the Chromium CQ_, so he could provide more info on the necessary
steps to update said VM.

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/linux-chromeos-rel/9714

#10, for controls, the vivid driver already implements a big number of capture controls, so if there is anything missing, we probably should add it. Do we have a list of controls we're currently using?


No USB VID:PID is obviously something we need to take care of in userspace indeed.

Most controls we are using are tested in camera_V4L2 autotest.

Just take a quick look of camera_V4L2, it will skip devices without USB vid:pid.

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f17f8d583e01d5511a6e9506a0ce7bbad8b02957/client/site_tests/camera_V4L2/camera_V4L2.py#49

The chromeos build team provides the VM images. If we need something tweaked in the image, we should talk to them.

Couple things to note first:

 - The "linux-chromeos-rel" bot that was linked in #11 runs tests on the linux-chromeos build of chrome, not actual chromeos. The VM bot is https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/15036. It's currently running a subset of the tests the linux-chromeos bot runs. (I just tried running media_unittests in it and am seeing test crashes, so it'll need some work.)

 - The VMs we're using on chromium's CQ are for the amd64-generic board. FWIU, some of the VM images are internal only (I think betty falls into that category) and so can't be used on chromium's CQ. Again, if we need to make some of them publicly-available, we'll need to talk to the cros build team.

whoops, didn't mean to un-cc shik@

i'd look at making CONFIG_VIDEO_VIVID=m in our kernel configs so then VM images can just modprobe it as needed by tests.  i think that's the only thing you really need here.

double check the kernel image & module overhead.  i don't think it'll be a big deal since we're already enabling V4L2 support everywhere.

Thanks vapier@. Loading vivid as a kernel module sounds better.
If we do so, should we change each autotest for camera to run modprobe when it is executed on VM? Can autotest scripts know whether the environment is on VM?

if the autotest is written for vivid, then just loading it in there shouldn't be a problem

The autotest is not written for vivid, it's a generic Chrome OS camera autotest and vivid is supposed to behave like a real camera (+/- not being a USB device).

As shik@ said at #11, tests in camera_V4L2 were actually skipped.
To execute tests, we need to prepare the followings:
- IdVendor and IdProduct
- /sys/devices/system/cpu/cpu0/cpufreq/(cpuinfo_max_freq|cpuinfo_min_freq|scaling_governor)  (for https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/utils.py?l=2152)
- /etc/camera/camera_characteristics.conf (for https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/camera_V4L2/src/camera_characteristics.cc?l=20)

After preparing dummy values for them ad-hocly, tests was successfully performed.
(I need to find proper ways to add them later.)

Currently, a test case 'TestResolutions' fails because of unsupported formats/resolutions in vivid.
At least, I should extend vivid to support formats used in https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/camera_V4L2/src/media_v4l2_test.cc?l=286

it shouldn't be a problem to have an autotest detect its environment (such as running under a VM) and then probe vivid on demand

We don't need to bother making camera_V4L2 work with vivid. camera_V4L2 is mostly a test to test the kernel uvc driver and validate the USB camera module, so it brings up little value to run camera_V4L2 with vivid.

We should focus on enabling the camera HAL tests on betty and running capture_unittests on the chromium waterfall VM. These will provide test coverage on the USB camera HAL and the chrome video capture service on CrOS. Please work with shik@ to figure out the proper way to run USB camera HAL on top of vivid.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/944827b95b5e7ab571733ebf5f6985bf418e31ab

commit 944827b95b5e7ab571733ebf5f6985bf418e31ab
Author: Shik Chen <shik@chromium.org>
Date: Fri Jul 13 12:04:12 2018

media: support devices without built-in camera

Some devices like VM or chromeboxes do not have built-in camera at all,
we should not treat it as an error.

BUG=b:77833131,b:64996728,chromium:832816,chromium:852302
TEST=external camera works on teemo (a chromebox)

Change-Id: I2a19afa72ab9e0828f4c65a7761cd2b54d1ecfdd
Reviewed-on: https://chromium-review.googlesource.com/1136451
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Shik Chen <shik@chromium.org>
Cr-Commit-Position: refs/heads/master@{#574884}
[modify] https://crrev.com/944827b95b5e7ab571733ebf5f6985bf418e31ab/media/capture/video/chromeos/camera_hal_delegate.cc

Hi Keiichi, it looks like external camera on devices without built-in camera was fixed as per comment #23. Are we still waiting for some other blockers?

Hi Tomasz.
Thanks to Shik's effort, we can use external cameras on HALv3 now.
For vivid, however, some fixes are still needed.
He is working for it on crrev.com/c/1133099, but I haven't made sure if it also works on betty.
I'm planning to try it in a few days.

Okay, thanks for the update!

I tried vivid on USB camera HALv3.
Though it correctly recognized vivid as an external camera, video capturing didn't work. We encountered error messages: "Failed to create GpuMemoryBuffer" and "Failed to configure streams: Invalid argument".
These messages come from
https://cs.chromium.org/chromium/src/media/capture/video/chromeos/stream_buffer_manager.cc?l=149&rcl=0387a40eee2f56121933a8769da3d08e8636bbf1 and
https://cs.chromium.org/chromium/src/media/capture/video/chromeos/camera_device_delegate.cc?l=471&rcl=eea229f2451fbd8d3c8e4920116f6f591eaab49c

Ricky (or someone else), do you have any idea about this problem?

Currently, we need to cherry-pick the following CLs to try it:
- crrev.com/c/1133099 (arc-camera)
- crrev.com/c/1119741 (kernel config)
- crrev.com/c/1109803 (kernel eclass file)
- crrev.com/i/643548  (betty-overlay)

I suspect that it's because the minigbm virtio-gpu backend doesn't support the formats of buffers we're trying to allocate:
https://cs.corp.google.com/chromeos_public/src/platform/minigbm/virtio_gpu.c

What's the output when you run `modetest` on betty? This is probably because there isn't (enough) minigbm support for betty. The camera buffers are DMA-buf allocated through the minigbm library.

I pasted the output of modetest.
https://paste.googleplex.com/5155661851656192

I wonder why this error didn't occur when I tried vivid without enabling HALv3.
Was the way of allocating buffers changed?

It's like what Tomasz said in c#29. We'll need to add the required pixel formats and GBM_BO usage flags for the virtio_gpu backend in minigbm.

With HAL v3 API the buffers are allocated by the API caller and then passed to the camera stack to fill. Conventional Chrome camera stack opens the USB camera device directly and requests MMAP V4L2 buffers from the UVC kernel driver.

Thanks, Ricky and Tomasz. I got it. Let me try.

By fixing minigbm by crrev.com/c/1160434, I moved to next error in vaapi_wrapper.
'vaInitialize' was failed with the message: 
"libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)"

We need to disable vaapi for betty. How should I do it?

I finally succeeded to use camera app with vivid on betty.
(The error messsage I posted on c#34 was not critical one. Actual causes were in minigbm.)
Currently, there are 6 unmerged CLs related to this work; four in c#28, one in c#34 and crrev.com/c/1163582

I made sure that video recording feature on the camera app works well.
For snapshot feature, need to wait for crrev.com/c/1163558.

---

The next step is to run camera-related autotests on betty.
I think video_WebRtcCamera and video_WebRtcPeerConnectionWithCamera would be the first target test cases.
(Any other suggestion? camera_HAL3 might be good, but I don't know the detail fo it)

At least, we need the followings to make them work:
(1). Update test scripts to probe vivid if it's working on VM (c#21)
(2). Fix  crbug.com/871185 , which says cros_camera_service fails to restart when 'restart ui' is called.
In both testcases, UIs are reset at the begining. So, cros_camera_service is killed at that time.

 crbug.com/871185  might not be a blocker of this work because the problem occurs only when vivid is enable before restarting UI. In the autotests, vivid will be enabled after UI restarting.

Doug, Guenter, we want to enable the vivid module for Chrome OS VM testing inside Chromium infra. Full context is in earlier posts, but we seem to have two main options:

1) just set it to =m in some generic config (
chromiumos-x86_64.flavour.config?)

2) put it under a USE flag (as https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1109803 does)

FYI, the module is 249 KB big. I'm not sure what's taking so much space there, though. Possibly some bitmaps for test patterns.

Do you have any opinion on this?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/minigbm/+/a13dda71b4b9568c4e65fef0bc8e4c362ba3fcf3

commit a13dda71b4b9568c4e65fef0bc8e4c362ba3fcf3
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Tue Aug 14 09:13:26 2018

minigbm: support NV12 in virtio_gpu

To use vivid on VM as an external camera, virtio_gpu has to support
NV12.

BUG=chromium:852302
TEST=Use camera app on betty by using vivid as an external camera
Change-Id: I84ddcbae473a06337918328afe1db8d32003ad84
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1160434
Reviewed-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>

[modify] https://crrev.com/a13dda71b4b9568c4e65fef0bc8e4c362ba3fcf3/virtio_gpu.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/minigbm/+/af94db9f0ad425559d34e654a8a6ed70afa1f3a9

commit af94db9f0ad425559d34e654a8a6ed70afa1f3a9
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Tue Aug 14 09:13:26 2018

minigbm: adjust height of NV12 buffer in helper

When the format is NV12, additional buffer is required for the UV plane.
Though size adjustment for this requirement is currently implemented
only in the msm driver, it is not specific to this device.

This commit moves the adjustment to helper function
'drv_dumb_bo_create', which is called from each driver.

BUG=chromium:852302
TEST=video playback on msm

Change-Id: I690f79ccc6a2c99e680b0d293ffa6002f74ae6bf
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1163582
Reviewed-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>

[modify] https://crrev.com/af94db9f0ad425559d34e654a8a6ed70afa1f3a9/msm.c
[modify] https://crrev.com/af94db9f0ad425559d34e654a8a6ed70afa1f3a9/helpers.c

> FYI, the module is 249 KB big. I'm not sure what's taking so much space 
> there, though. Possibly some bitmaps for test patterns.

Though this isn't huge, this isn't a trivially small amount of space.  By default I'd tend to say do a USE flag to save this space.  That also has the benefit of not exposing an additional attack vector in case there are bugs in this module.

...if there is some benefit for making this a standard kernel module then it's probably fine too, but we'd want to give it a closer look...

Thanks, Doug.
Then, I should put it under USE flag.

---

Updates:
I succeeded to make video_WebRtcCamera.(vp8|h264) and video_WebRtcPeerConnectionWithCamera be passed on betty with vivid in my workstation.
Some CLs are under reviewing.
After all CLs are merged, I'm going to create a new issue to ask Ben to add these tests to Chromium CQ.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/3ee386ec3b75eec5dc32a3fee08a3c58cc7d1a2c

commit 3ee386ec3b75eec5dc32a3fee08a3c58cc7d1a2c
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Fri Aug 17 18:18:12 2018

cros-kernel2: Add a USE flag for vivid

Add a USE flag for vivid, which is Linux's virtual video test driver and
used for camera tests on VM.

BUG=chromium:852302
TEST=modprobe vivid on betty

Change-Id: Iac0e25bd8a21d67ddc7d16f767e1a67f19c49c91
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1109803
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/3ee386ec3b75eec5dc32a3fee08a3c58cc7d1a2c/eclass/cros-kernel2.eclass

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-amd64-generic-cheets-private/+/714b78224dae9fdbe7ce6a1026b5e4cabe49d116

commit 714b78224dae9fdbe7ce6a1026b5e4cabe49d116
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Tue Aug 21 00:12:18 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/eded771ad55731ec3c0d5f27d3a3907ba848a162

commit eded771ad55731ec3c0d5f27d3a3907ba848a162
Author: Shik Chen <shik@chromium.org>
Date: Thu Aug 23 19:09:22 2018

usb: add vivid support

Make it work as an external camera.

BUG=chromium:852302,b:77833131
TEST=modify the kernel, and check vivid works on nautilus

Change-Id: I8f1405f7b7cb851215bbbf169a3fb7cd23facef9
Reviewed-on: https://chromium-review.googlesource.com/1133099
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Shik Chen <shik@chromium.org>
Reviewed-by: Heng-ruey Hsu <henryhsu@chromium.org>
Reviewed-by: Keiichi Watanabe <keiichiw@chromium.org>

[modify] https://crrev.com/eded771ad55731ec3c0d5f27d3a3907ba848a162/hal/usb/common_types.h
[modify] https://crrev.com/eded771ad55731ec3c0d5f27d3a3907ba848a162/hal/usb/camera_hal.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/269e43ad19c72e31f6fe01d47ba5ced25cd7c42e

commit 269e43ad19c72e31f6fe01d47ba5ced25cd7c42e
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Wed Aug 29 18:15:35 2018

Use vivid in WebRTC camera tests if they are on VM

BUG=chromium:852302
TEST=Run video_WebRtcCamera on betty
TEST=Run video_WebRtcPeerConnectionWithCamera.h264 on betty
TEST=Run video_WebRtcPeerConnectionWithCamera.vp8 on betty

Change-Id: I22570f0d6eb05eca2b8765688b7f76e798659a21
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1119737
Reviewed-by: Shik Chen <shik@chromium.org>

[modify] https://crrev.com/269e43ad19c72e31f6fe01d47ba5ced25cd7c42e/client/site_tests/video_WebRtcPeerConnectionWithCamera/loopback.html
[modify] https://crrev.com/269e43ad19c72e31f6fe01d47ba5ced25cd7c42e/client/site_tests/video_WebRtcPeerConnectionWithCamera/video_WebRtcPeerConnectionWithCamera.py
[modify] https://crrev.com/269e43ad19c72e31f6fe01d47ba5ced25cd7c42e/client/site_tests/video_WebRtcCamera/video_WebRtcCamera.py
[modify] https://crrev.com/269e43ad19c72e31f6fe01d47ba5ced25cd7c42e/client/site_tests/video_WebRtcCamera/getusermedia.html

Created another issue for creating public VM image where vivid and HALv3 are available. 
 http://crbug.com/879140 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-betty-private/+/7b402a7499dd18a7e5f77d4687962a8fe7e0df44

commit 7b402a7499dd18a7e5f77d4687962a8fe7e0df44
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Sun Sep 02 12:19:45 2018

My CLs for enabling vivid and HALv3 on amd64-generic, which is a board used in VM testing in Chromium waterfall, were merged in  crbug.com/879140 .
(need to wait for it being LKGM for a while though)

So, the next step is adding our camera tests.
I'm planning to add the following two kinds of tests:
(1) capture_unittests
https://cs.chromium.org/chromium/src/media/capture/video/video_capture_device_unittest.cc
(2) video_WebRtcCamera, video_WebRtcPeerConnectionWithCamera, which are Chrome OS's autotests

(1) is already running on Chrome infra, but many test cases are skipped due to the lack of camera input devices. So, vivid must be loaded before running.
I will add it after  crbug.com/881300  is fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/40387e4196caec4e46c37c0ca824d7f16b29fa3f

commit 40387e4196caec4e46c37c0ca824d7f16b29fa3f
Author: Shik Chen <shik@chromium.org>
Date: Fri Sep 14 19:08:38 2018

hal_adapter: fix vivid detection after service is restarted

The udev property (ID_V4L_PRODUCT) we used to detect vivid device is
stored in /run/udev/data, so we need to bind it inside the sandbox.

BUG= chromium:871185 , chromium:881300 ,chromium:852302
TEST=On amd64-generic VM, do:
       1. $ modprobe vivid n_devs=1 node_types=0x1
       2. Check vivid is detected
       3. $ restart cros-camera
       4. Check vivid is still detected

Change-Id: Ibfe4b974ed73ae88f42eeb70517ed6fdcf9621dc
Reviewed-on: https://chromium-review.googlesource.com/1214903
Commit-Ready: Shik Chen <shik@chromium.org>
Tested-by: Shik Chen <shik@chromium.org>
Tested-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-by: Keiichi Watanabe <keiichiw@chromium.org>

[modify] https://crrev.com/40387e4196caec4e46c37c0ca824d7f16b29fa3f/hal_adapter/init/cros-camera.conf

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48

commit 45f1af41f2405a04e9855e07d30ff4ebdb5f6e48
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Sun Sep 16 11:04:36 2018

video: Port WebRTC camera tests from autotest

Add video.WebRTCCamera and video.WebRTCPeerConnectionWithCamera(H264|VP8).
These are Tast version of video_WebRtcCamera and
video_WebRtcPeerConnectionWithCamera.(h264|vp8) in autotest, respectively.

Since main procedures for each test are originally written in
JavaScript, HTML/JS files added by this commit are almost same with
ones in autotests repository.
Each test is executed by calling JS functions from webrtc.go.

If tests are executed on VM, vivid will be used as a video input device.

BUG= chromium:882433 , chromium:852302
TEST=Run each test on kevin by 'tast run'
TEST=Run each test on amd64-generic VM by 'tast run'
CQ-DEPEND=CL:1220426, CL:1222293, CL:1214903

Change-Id: I666c4f960688686a9a91a79d04ee4db49d7b1e6f
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1218308
Reviewed-by: Dan Erat <derat@chromium.org>

[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/webrtc/webrtc.go
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8.go
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera.go
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/data/third_party/ssim.js
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/data/loopback.html
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/data/getusermedia.html
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/data/third_party/munge_sdp.js
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264.go
[add] https://crrev.com/45f1af41f2405a04e9855e07d30ff4ebdb5f6e48/src/chromiumos/tast/local/bundles/cros/video/data/third_party/blackframe.js

Let me wrap up the current status and remaining TODOs.
The goal is to make camera tests run on VM in Chrome CQ. (i.e. amd64-generic-rel)
The target tests are Chrome's capture_unittests and two Tast tests for WebRTC added by crrev.com/c/1218308.

# General
TODO:
1. Fix  crbug.com/885075 

I found that vivid didn't work well on the current ToT.
Since it worked well on older version of ChromeOS VM that I was using previously, some recent changes should break something.
I created  crbug.com/885075  and will investigate it with shik@. First of all, this problem needs to be fixed.

# capture_unittests
TODO:
2. Make capture_unittests work correctly with vivid on VM ( crbug.com/881300 )
3. Enable vivid on Chrome CQ

This test is already running on Chrome CQ's chromeos-amd64-generic-rel, but vivid is not used there. So, some testcases that need a USB camera are skipped.
We need to have vivid be loaded before running the test. (3)

But, there is a bug that capture_unittests doesn't properly use vivid (2).
This problem is different from one I mentioned above.
There, though capture_unittests says all cases are passed, it doesn't find vivid as an external camera even on the VM where vivid works well. 
I guess we should add some fixes into cros-camera or capture_unittests.

# Tast WebRTC tests
TODO:
4. Create a script for checking device capabilities of boards ( crbug.com/885007 )
5. Make sure if it is working on VM board in Chrome infra ( crbug.com/885016 )
6. Make them run on Chrome CQ as a part of vm_sanity
7. Remove 'informational' attributes
8. (optional) Add perf metrics in these two tests and use it for alert mails

Tast version of WebRTC tests started running in ChromeOS waterfalls.
https://stainless.corp.google.com/search?view=matrix&row=model&col=build&first_date=2018-09-12&last_date=2018-09-18&test=video.WebRTC

But, as you can see there, our tests are also running and failing on Chromebox, which has no camera.
Though they doesn't raise any alerts now because these tests have 'informational' attributes, the attribute should be removed in the future. (7)
So, we need a script to detect devices' capabilities. (4)

Also, we want to run these tests on Chrome CQ. It should be good to add them as a part of vm_sanity like other Tast tests. (6)
Before adding them, however, I'd like to make sure if they work well stably also on VM images. (5)
I think that one possible way is to add a VM image in ChromeOS waterfalls. ( crbug.com/885016 )
Or, can we add Tast tests to Chrome CQ 'experimentally'? In other words, is there any way to ignore failures on specific tests in Chrome CQ?

We could possibly run tests and catch exceptions so they pass. We don't really have an experimental capability at the moment, but it's not hard to add.

If we expect the test to fail and it's failing locally, we probably shouldn't waste resources running it hundreds of times a day though, especially if the test doesn't fail quickly.

 http://crbug.com/876587  covers the work to have a standalone Tast target instead of being a part of vm_sanity. I think it probably makes sense to run this test once we have that.

Thanks, Achuith.
I'm actually planning to add our tests after fixing all bugs and making sure that they work stably in local environments. I was just afraid of possibility that my change might block everyone's CLs.

Adding them as standalone targets sounds helpful to know which part is broken. I appended it to my TODOs:)

#51: Re detecting devices' capabilities, is there a USE flag specifying whether the device has a camera, or is it communicated via autotest-capability's YAML files ( issue 885007 ), or is it only testable at runtime?

#54: I meant checking autotest-capability's YAML files ( issue 885007 ) there. (Since it seems you already had deeper discussion in the issue, this response must be too late:P)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/93eecb21def16dad5f6204df7dea1a1e0b97be87

commit 93eecb21def16dad5f6204df7dea1a1e0b97be87
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Wed Sep 26 17:32:21 2018

tast-tests: Add short/full versions of WebRTC tests

Shorten the performance time of existing video.WebRTC tests and
add three video.WebRTC*Perf tests as the full versions.
While short versions are intended to be run in Chrome CQ, where tests
have to be done in several seconds, the full versions will be used
for gathering the performance data in waterfall.

This commit also moves the starting points for counting time, i.e.
the places where setTimeout are called in getusermedia.html and
loopback.html. This is because we want to exclude the time for
completing getUserMedia calls.

BUG= chromium:882433 , chromium:852302
TEST=Run each test on kevin by 'tast run'
TEST=Run each test on amd64-generic VM by 'tast run'

Change-Id: I1852b86c23e53beb271cb9ff6d4c9f09f0285f62
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1237804
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/data/loopback.html
[add] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera_perf.go
[modify] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera.go
[modify] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/data/getusermedia.html
[modify] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8.go
[modify] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264.go
[add] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264_perf.go
[add] https://crrev.com/93eecb21def16dad5f6204df7dea1a1e0b97be87/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8_perf.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/d2ca2a5de848ff886c5c1809cafcdbc799df382f

commit d2ca2a5de848ff886c5c1809cafcdbc799df382f
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Tue Oct 02 12:19:42 2018

tast-tests: Add autocaps dependencies to WebRTC tests

Add autotest-capability:usb_camera in SoftwareDeps in WebRTC tests,
which require that DUT supports USB cameras.

BUG=chromium:852302,  chromium:888883 
TEST="tast run -checktestdeps always <guado> <testname>" skips the tests
TEST="tast run -checktestdeps always <amd64-genericVM> <testname>" run correctly

Change-Id: I9bc0830c929df17b4e1e4dde914a76f33bcc0239
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1253490
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/d2ca2a5de848ff886c5c1809cafcdbc799df382f/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera_perf.go
[modify] https://crrev.com/d2ca2a5de848ff886c5c1809cafcdbc799df382f/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera.go
[modify] https://crrev.com/d2ca2a5de848ff886c5c1809cafcdbc799df382f/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264.go
[modify] https://crrev.com/d2ca2a5de848ff886c5c1809cafcdbc799df382f/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8.go
[modify] https://crrev.com/d2ca2a5de848ff886c5c1809cafcdbc799df382f/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8_perf.go
[modify] https://crrev.com/d2ca2a5de848ff886c5c1809cafcdbc799df382f/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264_perf.go

Document for work including this issue: go/cros-camera-waterfall-test

FYI: amd64-generic board, which is used in Chrome CQ as a VM, started kernel 4.14 from crrev.com/c/1203638

I don't think we need additional changes for this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d3ee041e6e72508a8d877ec0f8d143c9373b1659

commit d3ee041e6e72508a8d877ec0f8d143c9373b1659
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Thu Oct 11 20:25:50 2018

FROMGIT: media: vivid: Support 480p for webcam capture

Support 640x480 as a frame size for video input devices of vivid.

Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
Reviewed-by: Kieran Bingham <kieran.bingham+renesas@ideasonboard.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
(cherry picked from https://git.linuxtv.org/media_tree.git/commit/?id=8caec72e8cbff65afa38928197bea5a393b67975)

BUG=chromium:852302
TEST=tast run <amd64-generic VM> video.WebRTCCamera

Change-Id: Ie93e74a2a4c5d3c1107832b5e9bd50f8dcb592b3
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1270698
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Reviewed-by: Tomasz Figa <tfiga@chromium.org>

[modify] https://crrev.com/d3ee041e6e72508a8d877ec0f8d143c9373b1659/drivers/media/platform/vivid/vivid-vid-cap.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/62d2bd99255082967be127282cd2ecc0ce5fcc7e

commit 62d2bd99255082967be127282cd2ecc0ce5fcc7e
Author: Shik Chen <shik@chromium.org>
Date: Mon Oct 15 17:22:30 2018

[Video Capture, Chrome OS] Wait until vivid is detected in ChromeOS VM

BUG= chromium:881300 ,chromium:852302
TEST=Pass capture_unittests with/without vivid enabled on amd64-generic
     VM, and verify vivid is used when it's enabled.
TEST=Pass capture_unittests on teemo (a Chromebox), and verify it
     doesn't wait for camera.

Change-Id: I2a11a848639eba426daa4db2f8d22b90c01e08b9
Reviewed-on: https://chromium-review.googlesource.com/c/1260806
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Reviewed-by: Christian Fremerey <chfremer@chromium.org>
Commit-Queue: Shik Chen <shik@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599658}
[modify] https://crrev.com/62d2bd99255082967be127282cd2ecc0ce5fcc7e/media/capture/video/chromeos/camera_hal_delegate.cc
[modify] https://crrev.com/62d2bd99255082967be127282cd2ecc0ce5fcc7e/media/capture/video/chromeos/camera_hal_delegate.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/93ec054211ddbdce6e1221c7bc3be87b57f1ed29

commit 93ec054211ddbdce6e1221c7bc3be87b57f1ed29
Author: Shik Chen <shik@chromium.org>
Date: Thu Oct 18 13:16:22 2018

usb: fix StreamOn for non integer fps

BUG=chromium:852302, chromium:881300 
TEST=Run capture_unittests with vivid on VM, which uses 1280x720 in 12.5
     fps.

Change-Id: I76f09c4a9a5de0b1cb2bf43c3a4184446e678bfc
Reviewed-on: https://chromium-review.googlesource.com/1286024
Commit-Ready: Keiichi Watanabe <keiichiw@chromium.org>
Tested-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-by: Heng-ruey Hsu <henryhsu@chromium.org>

[modify] https://crrev.com/93ec054211ddbdce6e1221c7bc3be87b57f1ed29/hal/usb/v4l2_camera_device.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/790cb4937fd21c65a40a6c67a18db6e93a51b843

commit 790cb4937fd21c65a40a6c67a18db6e93a51b843
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Tue Oct 23 20:01:34 2018

tast-tests: Check and store perf metrics in video.WebRTC tests

Add checking for frame statics and make tests fail if there were too many frames during video capturing.
In addition, video.WebRTC*Perf will upload the statistics as perf metrics.

BUG=chromium:852302,  chromium:882433 
TEST=run on amd64-generic VM and kevin

Change-Id: I46051e1c2807ad97b0b90f364cfd0d76cf941033
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1273435
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>

[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc/webrtc.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera_perf.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/data/loopback.html
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/data/getusermedia.html
[add] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/lib/videotype/videotype.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_h264_perf.go
[modify] https://crrev.com/790cb4937fd21c65a40a6c67a18db6e93a51b843/src/chromiumos/tast/local/bundles/cros/video/webrtc_peer_connection_with_camera_vp8_perf.go

Let me share the current status of this issue.
The remaining tasks to complete this issue are the following:
- Run capture_unittests on Chrome CQ. ( Issue 900849 )
- Run capture_unittests on PFQ. (Issue 893501)
- Run tast.video.WebRTC* tests on Chrome CQ and ChromeOS CQ. ( Issue 900857 )

The reason why I made them our goal is in go/cros-camera-waterfall-test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a81bdb917ecd5c3bd158865d35338a37ca300995

commit a81bdb917ecd5c3bd158865d35338a37ca300995
Author: Keiichi Watanabe <keiichiw@chromium.org>
Date: Fri Nov 16 07:12:14 2018

testing/buildbot: Use vivid for capture_unittests on ChromeOS VM in CQ

Load vivid before running capture_unittests on chromeos-amd64-generic-rel.
This makes it possible to run test cases that require a webcam.

Bug: 852302,  904730 
Change-Id: I5d4f5a1dd7a8ea27ef7eabd438fd766621573096
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/1282424
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Cr-Commit-Position: refs/heads/master@{#608688}
[modify] https://crrev.com/a81bdb917ecd5c3bd158865d35338a37ca300995/build/chromeos/run_vm_test.py
[modify] https://crrev.com/a81bdb917ecd5c3bd158865d35338a37ca300995/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/a81bdb917ecd5c3bd158865d35338a37ca300995/testing/buildbot/test_suites.pyl