Bot recovered after revert https://chromium-review.googlesource.com/c/536913

chfremer@ could you please reconsider the CL?

Thanks for finding this and doing the revert.

I checked the logs of the failed builds. It looks like the set of failing WebRtcImageCaptureSucceedsBrowserTest tests is not always the same. The largest set contains 11 failures, while the smallest one contains only 5. Here are the 11 ones:

    WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/2 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.GetTrackCapabilities/2 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.GetTrackCapabilities/3 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.GetTrackSettings/2 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.GetTrackSettings/3 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/2 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/3 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.ManipulateZoom/2 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.ManipulateZoom/3 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/2 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)
    WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/3 (../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:151)

Observations: 
All these tests run with 4 parameters. /0 and /1 use a fake device, which seems to never fail. /2 and /3 are using a real webcam. /2 is using the legacy in-process video capture stack and /3 is using the new video capture service.

My CL is adding cases /1 and /3. It looks like adding these even causes some /2 that passed without the CL to start failing. My best guess at this point is that the /3 cases can somehow bring the camera into a bad state. Extra evidence for this is that there are capture_unittests, that also start failing with this CL, even though nothing about them was touched.

This may get very tricky to figure out, since it may depend on not just the build parameters, but also the particular webcam and driver on the bot. I do not get any failures on my local machine.

Does anyone have a way to find out what build parameters and what camera is used on this type of bot?

chfremer@ We also see some unrecoverable VideoDevice test failures on chromium.webrtc.fyi
See details in https://bugs.chromium.org/p/chromium/issues/detail?id=733640.

It does look like some tests (or some sequence of tests) dealing with a webcam bring it into a bad state.
Quite a lot of failures started after https://chromium-review.googlesource.com/c/528020/ landed.

Shouldn't this bug be merged into https://crbug.com/733640 now
that you guys have reverted chfremer@s CL? 

The latest run I see at 
https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/30086

has four failing tests:
[  FAILED  ] WebRtcImageCaptureBrowserTest.GetPhotoCapabilities/1, where GetParam() = 1-byte object <00> (200 ms)
[  FAILED  ] WebRtcImageCaptureBrowserTest.GrabFrame/1, where GetParam() = 1-byte object <00> (229 ms)
[  FAILED  ] WebRtcImageCaptureBrowserTest.GetTrackCapabilities/1, where GetParam() = 1-byte object <00> (354 ms)
[  FAILED  ] WebRtcImageCaptureBrowserTest.ManipulateZoom/1, where GetParam() = 1-byte object <00> (136 ms)

where /1 implies that they are using the real physical webcam, so 
https://bugs.chromium.org/p/chromium/issues/detail?id=733640#c10
seems like a relevant comment to me.

The run you refers to has a different failure: WebRtcAudioBrowserTest.CanMakeVideoCallAndThenRenegotiateToAudio (https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/30086/steps/content_browsertests/logs/stdio)

These are two different bots, so I'm not sure it's entirely an infra issue as https://bugs.chromium.org/p/chromium/issues/detail?id=733640#c10 says. To me it looks like tests do something which ends up in webcam failure.

#5: I searched for the pattern "[  FAILED  ]" (w/o the quotes)
and only the four I pointed out came up.  Perhaps other tests have
transient fail-like logs while the test itself works in the end.

I found the build parameters that are used for these tests runs (see below) and tried to repro with those on my linux box. The issue does not repro with that. Maybe it depends on the camera model. I think it would be useful for all tests that use a real camera to output the camera model and usb id to the logs. Unless there are objections to that, I'll do a CL to add that.

ffmpeg_branding = "Chrome"
is_component_build = false
is_debug = false
proprietary_codecs = true
strip_absolute_paths_from_debug_symbols = true
use_goma = true

#5: content_browsertests' ImageCapture seem to be working fine now (see e.g. [1]),
olka@ can we close this bug?

mcasas@: I would prefer to have these tests either working correctly or being skipped by the bot (with a clear message why); i.e. they should be either tolerant to build parameters/camera HW, or should detect mismatches.

The way they are (were) failing is very confusing, WebRTC in Chrome sheriffs wasting time investigating these failures and bisecting.
And we also see problems with CLs like [https://chromium-review.googlesource.com/c/536913] being reverted because of that as well.

Leaving it up to you and chfremer@ weather to work on it here or in a separate bug.

olka@: You are raising an interesting question regarding the purpose of these tests running on these bots. My understanding is that the very reason that these bots have real webcams is in order to allow us to catch issues that happen on real devices but not on our fake devices. The failures tracked in this bug are an example of this happening. I agree that the symptoms caused by issues with real cameras are often confusing and difficult to investigate. And many times they point to issues with the devices/drivers instead of issues in our code. But doing so seems important and useful nevertheless.

Not sure if my understanding of the purpose of the real webcam tests is accurate, though. It seems like if we really want to get test coverage with real devices, it would be both more effective and efficient to have a dedicated test suite for that (and maybe a separate team of sheriffs).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a95e0bf52bfe99622e4bf498adb761cf9eb218a9

commit a95e0bf52bfe99622e4bf498adb761cf9eb218a9
Author: Christian Fremerey <chfremer@chromium.org>
Date: Thu Jun 22 16:41:41 2017

[Video Capture] Log name and id of real camera device used in tests

Some unittests and integration tests are run with real camera devices.
In order to facilitate troubleshooting of failures from these tests, it is
useful to easily see what camera device is being used. This CL adds logs
to these tests outputting the name and usb id of the camera.

     caputer_unittests --gtest_filter="VideoCaptureDevice*"

Bug: 733582,733640
Test: content_browsertests --gtest_filter="*WebRtcImageCapture*"
Change-Id: Ic6e090337a72c044e9f0a49466cf78be93949064
Reviewed-on: https://chromium-review.googlesource.com/538126
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Emircan Uysaler <emircan@chromium.org>
Commit-Queue: Christian Fremerey <chfremer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#481560}
[modify] https://crrev.com/a95e0bf52bfe99622e4bf498adb761cf9eb218a9/content/browser/webrtc/webrtc_image_capture_browsertest.cc
[modify] https://crrev.com/a95e0bf52bfe99622e4bf498adb761cf9eb218a9/content/public/test/content_browser_test_utils.cc
[modify] https://crrev.com/a95e0bf52bfe99622e4bf498adb761cf9eb218a9/content/public/test/content_browser_test_utils.h
[modify] https://crrev.com/a95e0bf52bfe99622e4bf498adb761cf9eb218a9/media/capture/video/video_capture_device_unittest.cc

The CL that started this was relanded [1] and got first picked by the chromium.webrtc Linux Tester bot in build 30196. The reland disabled newly added test cases involving the physical camera in combination with the video capture service, while new test cases involving fake devices remain enabled. 

It looks like this helped, since the tests are not failing like they did in the first attempt to land.

One thing I noticed is that both before and after landing this, there are typically 3-4 test cases in each run that fail at the first attempt and succeed at the second. All of these are test cases involving the real camera. These cases can easily be found by opening the stdio logs from content_browsertest runs, e.g. [2], and searching for the string "Expected". I will open a new bug entry to track this.

Example output from a failure:
[171/214] WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/0 (224 ms)
[ RUN      ] WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/2
Xlib:  extension "RANDR" missing on display ":9".
[5548:5585:0622/073344.655188:618324168:ERROR:devtools_http_handler.cc(804)] 
DevTools listening on 127.0.0.1:40176

../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:168: Failure
Value of: RunImageCaptureTestCase("testCreateAndGetPhotoCapabilitiesSucceeds()")
  Actual: false
Expected: true
[  FAILED  ] WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/2, where GetParam() = (1-byte object <00>, 1-byte object <00>) (164 ms)


[1] https://chromium-review.googlesource.com/c/544180/
[2] https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/30208/steps/content_browsertests/logs/stdio

(reopening with an updated theory on the root cause)

I was able to reproduce the failures on my local Linux machine by adding --test-launcher-bot-mode to the command-line for running the tests. It seems the difference is that with this flag, tests run in parallel (one process per test run), while without the flag, they run one by one.

Unsurprisingly, trying to access the same single physical webcam simultaneously from several processes can cause problems. I verified that when trying to open a C920 camera from two Chromium instances at the same time, the second instance is unable to open it.

It is currently not 100% clear which platforms this affects how. It seems MacOS handles simultaneous requests to use a physical camera a bit better. But I would not count on this being free of problems on any platform.

phoglund@: Would it be possible to change the configuration of the bots such that tests involving a physical camera never run in parallel?

I just found that the bot already has a separate step for content_browsertests that need to be run sequentially, which uses --test-launcher-jobs=1 in the command line. What is left to do is to make sure that all tests involving a real camera run as part of that and not in task that allows parallel test cases.

capture_unittests already run with --test-launcher-jobs=1, so nothing to do for those.

I will try to come up with a way that allows us to add a reasonably simply --gtest_filter pattern to distinguish the test cases in webrtc_image_capture_browsertest.cc that do and do not use the physical webcam.

Simply rename your tests in content_browsertests so they fit the pattern '--gtest_filter=WebRtc*MANUAL*Webcam*',

and you're good to go.

I created the sequential step precisely for this reason, but I didn't know you had written more tests that access the real webcam.

re #15: I believe this may unfortunately not be that simple, because of the following:

The image capture test cases in for WebRtcImageCaptureSucceedsBrowserTest, see [1], are parameterized. They run both with fake and real camera, and in both cases share the same test case name. We want the runs using the fake camera to stay on the test task that has parallel execution, and we want the ones using the real camera to execute on the test task that has sequential execution.

Because the test case name is the same for real and fake devices, I am unable to mark just the runs with real cameras as MANUAL. I tried, but adding a MANUAL_ prefix via INSTANTIATE_TEST_CASE_P does not have any effect. And adding a prefix matching "WebRtc*MANUAL*Webcam*" in the INSTANTIATE_TEST_CASE_P for the real webcam cases is not enough to keep those tests from being picked up by the parallel test task, because that filters for "WebRtc*".

What I am proposing instead in my CL [2], is to use a prefix that does not match "WebRtc*", and then configure the sequential test task to filter for that instead of "WebRtc*MANUAL*Webcam*".

Please let me know if you agree with that proposal.

[1] https://cs.chromium.org/chromium/src/content/browser/webrtc/webrtc_image_capture_browsertest.cc?q=webrtc_image_capture_browsertest.cc&dr&l=226
[2] https://chromium-review.googlesource.com/c/chromium/src/+/872260

Re #16: Ha, nothing is ever simple. UsingRealWebcam is fine, let's go with that.

Note though: it is a bit of a problem to have content browsertests that run by default and access the real webcam, if there is one. This isn't a problem on the bot, because the test will be simply skipped (right?), but it will be confusing to a developer if they run content_browsertests on a machine with a borked or busy webcam. That's why I made my test manual back in the day. I don't know how to solve the MANUAL vs. TEST_P problem though. Perhaps this works well enough nowadays that you can get away with it...

Also: the tests will run in parallel by default, so if you think this might be a problem on the bots it will also be if running by hand.

Well, again, perhaps people don't run by hand that often anymore, and if they do they will usually have a filter, so maybe this isn't a big problem.

CL out: let me know when your rename lands and I'll land my patch.

https://chromium-review.googlesource.com/c/chromium/tools/build/+/876003

re #19: Good observation and a very valid concern. I think for the first step of renaming, we will probably get away with this, because we are not activating any tests that didn't already run before, i.e. there are WebRtcImageCaptureTests already today that run on real cameras as part of content_browsertests. 

A cleaner solution would be if we had a way to annotate/tag/prefix tests in a way that would the test runner (even when run without any command-line args) know which tests are fine being run in parallel and which ones require being run sequentially with respect to which other ones. Real-webcam tests can run in parallel with non-real-webcam tests, just not with other real-webcam tests.

It would be interesting to see if the gtest framework has any mechanism that would allow us to express this cleanly. And if it doesn't maybe we can consider adding something.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0f2a785127eeda2c9bdc2d52458d7f0481c2eb13

commit 0f2a785127eeda2c9bdc2d52458d7f0481c2eb13
Author: Christian Fremerey <chfremer@chromium.org>
Date: Fri Jan 19 16:23:01 2018

Prefix content_browsertests that use real webcam with UsingRealWebcam

Before this change, certain image capture tests using a real webcam
would run on webrtc bots as part of a build step called
content_browsertests, which executes multiple test cases in parallel.

By giving all content_browsertests that involve a real webcam a
common prefix "UsingRealWebcam" we make it simpler for the bot
configuration to run these tests in a different build step where we
can ensure they get run sequentially.

Bug: 733582, 730068
Change-Id: I2691f13a454af899a0249932ed1a90e76df8e6b4
Reviewed-on: https://chromium-review.googlesource.com/872260
Reviewed-by: Patrik Höglund <phoglund@chromium.org>
Commit-Queue: Christian Fremerey <chfremer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#530529}
[modify] https://crrev.com/0f2a785127eeda2c9bdc2d52458d7f0481c2eb13/content/browser/webrtc/webrtc_image_capture_browsertest.cc
[modify] https://crrev.com/0f2a785127eeda2c9bdc2d52458d7f0481c2eb13/content/browser/webrtc/webrtc_webcam_browsertest.cc
[modify] https://crrev.com/0f2a785127eeda2c9bdc2d52458d7f0481c2eb13/content/browser/webrtc/webrtc_webcam_browsertest.h

re #20: The test rename has landed. Please go ahead and land your patch when you're ready.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1b89029724ad624aa43d4c81ca2c4c5f1f2f3c08

commit 1b89029724ad624aa43d4c81ca2c4c5f1f2f3c08
Author: Patrik Höglund <phoglund@chromium.org>
Date: Mon Jan 22 07:26:44 2018

Change webcam test filter to work with TEST_P tests.

The current filter includes MANUAL_, but there are some tests that use
TEST_P, for which MANUAL_ doesn't work. Therefore, we make the pattern
not depend on MANUAL_ and rename tests accordingly.

Bug: chromium:733582
Change-Id: I0a2a9e7a452e041d4764c2137d3ba0168c06e64c
Reviewed-on: https://chromium-review.googlesource.com/876003
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Patrik Höglund <phoglund@chromium.org>

[modify] https://crrev.com/1b89029724ad624aa43d4c81ca2c4c5f1f2f3c08/scripts/slave/recipe_modules/chromium_tests/chromium_webrtc.py

Done, it ran these tests:

1/15] UsingRealWebcam_WebRtcWebcamBrowserTest.MANUAL_CanAcquireVga (781 ms)
[2/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/0 (520 ms)
[3/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/1 (93 ms)
[4/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoSettings/0 (519 ms)
[5/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoSettings/1 (92 ms)
[6/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/0 (650 ms)
[7/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/1 (76 ms)
[8/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/0 (519 ms)
[9/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/1 (109 ms)
[10/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackCapabilities/0 (519 ms)
[11/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackCapabilities/1 (93 ms)
[12/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackSettings/0 (519 ms)
[13/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackSettings/1 (76 ms)
[14/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.ManipulateZoom/0 (650 ms)
[15/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.ManipulateZoom/1 (77 ms)

Does that look correct?

Thanks! Yes this is the set of tests that I was expecting to see.

With that, as a next step, I can now attempt to activate more such tests, that we had previously disabled, due to flakiness. This will tell us whether or not the image capture tests having been non-sequential was the cause for the flakiness.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d1e3241f3215e6a2f861a83c89fa78e41d4de608

commit d1e3241f3215e6a2f861a83c89fa78e41d4de608
Author: Christian Fremerey <chfremer@chromium.org>
Date: Wed Jan 31 18:10:08 2018

Enable image capture tests with real webcams using video capture service

This is an attempt to enable tests that had been disabled due to flakiness on
webrtc bots. There is hope that the flakiness is resolved by the tests now
being run sequentially since [1] and [2] have landed.

The test pass when I run them on my local machine.

[1] https://chromium-review.googlesource.com/872260
[2] https://chromium-review.googlesource.com/876003

Test: content_browsertests --gtest_filter=UsingRealWebcam* --run-manual --test-launcher-jobs=1

Bug: 733582
Change-Id: If536e4c0b87c0a7de50ffc35f8c0a1d9abc52359
Reviewed-on: https://chromium-review.googlesource.com/894883
Reviewed-by: Emircan Uysaler <emircan@chromium.org>
Commit-Queue: Christian Fremerey <chfremer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#533325}
[modify] https://crrev.com/d1e3241f3215e6a2f861a83c89fa78e41d4de608/content/browser/webrtc/webrtc_image_capture_browsertest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e15e1660150c53f0fca00337131817da0303490d

commit e15e1660150c53f0fca00337131817da0303490d
Author: Christian Fremerey <chfremer@chromium.org>
Date: Wed Jan 31 20:09:11 2018

Revert "Enable image capture tests with real webcams using video capture service"

This reverts commit d1e3241f3215e6a2f861a83c89fa78e41d4de608.

Reason for revert: This still causes bot failures on Linux Tester, see https://ci.chromium.org/buildbot/chromium.webrtc/Linux%20Tester/35190

Original change's description:
> Enable image capture tests with real webcams using video capture service
> 
> This is an attempt to enable tests that had been disabled due to flakiness on
> webrtc bots. There is hope that the flakiness is resolved by the tests now
> being run sequentially since [1] and [2] have landed.
> 
> The test pass when I run them on my local machine.
> 
> [1] https://chromium-review.googlesource.com/872260
> [2] https://chromium-review.googlesource.com/876003
> 
> Test: content_browsertests --gtest_filter=UsingRealWebcam* --run-manual --test-launcher-jobs=1
> 
> Bug: 733582
> Change-Id: If536e4c0b87c0a7de50ffc35f8c0a1d9abc52359
> Reviewed-on: https://chromium-review.googlesource.com/894883
> Reviewed-by: Emircan Uysaler <emircan@chromium.org>
> Commit-Queue: Christian Fremerey <chfremer@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#533325}

TBR=emircan@chromium.org,chfremer@chromium.org

Change-Id: I643af751a57c84ccdeafd11547f9ca025ccedbb1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 733582
Reviewed-on: https://chromium-review.googlesource.com/896062
Reviewed-by: Christian Fremerey <chfremer@chromium.org>
Commit-Queue: Christian Fremerey <chfremer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#533375}
[modify] https://crrev.com/e15e1660150c53f0fca00337131817da0303490d/content/browser/webrtc/webrtc_image_capture_browsertest.cc

Ok, so I really want to get to the bottom of these failures.
Here is what I know so far:

A failing test run looks like this:
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.webrtc%2FLinux_Tester%2F35190%2F%2B%2Frecipes%2Fsteps%2Fcontent_browsertests_sequential%2F0%2Fstdout

A successful test run looks like this:
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.webrtc%2FLinux_Tester%2F35192%2F%2B%2Frecipes%2Fsteps%2Fcontent_browsertests_sequential%2F0%2Fstdout

The failures only happen on Linux. The same tests ran fine on Windows 8, Windows 10 and Mac OS.
https://ci.chromium.org/buildbot/chromium.webrtc/Win8%20Tester/40127
https://ci.chromium.org/buildbot/chromium.webrtc/Win10%20Tester/24931
https://ci.chromium.org/buildbot/chromium.webrtc/Mac%20Tester/76529

The failures do not happen when I run the tests locally on a machine with the same camera model 
  HD Pro Webcam C920 (046d:082d)
the same gn args
  ffmpeg_branding = "Chrome"
  is_component_build = false
  is_debug = false
  proprietary_codecs = true
  strip_absolute_paths_from_debug_symbols = true
  use_goma = true
the same command-line
  content_browsertests --test-launcher-bot-mode --gtest_filter=UsingRealWebcam* --run-manual --test-launcher-jobs=1
and the same Linux version
  Ubunti 14.04

It looks like in a failing build, first, some of the newly enabled tests run fine. Then, at a certain point all tests that try to use the physical webcam fail, even the ones that always passed before. And even ones that are part of a much later build step, i.e. are executed with a new command-line at a later time.

Here is the list of tests to look at:
[1/15] UsingRealWebcam_WebRtcWebcamBrowserTest.MANUAL_CanAcquireVga (781 ms)
[2/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/0 (454 ms)
[3/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/1 (77 ms)
[4/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoSettings/0 (453 ms)
[5/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoSettings/1 (76 ms)
[6/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/0 (650 ms)
[7/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/1 (76 ms)
[8/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/0 (453 ms)
[9/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/1 (59 ms)
[10/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackCapabilities/0 (453 ms)
[11/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackCapabilities/1 (76 ms)
[12/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackSettings/0 (454 ms)
[13/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetTrackSettings/1 (77 ms)
[14/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.ManipulateZoom/0 (519 ms)
[15/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.ManipulateZoom/1 (76 ms)

The tests ending with /0 are the ones using the legacy in-process video capture path. The tests ending with /1 use the out-of-process video capture service. In the above results, the tests with /1 were actually skipped, which is why there are no failures and why they take much less time. When enabling them, it looks like this:

[1/15] UsingRealWebcam_WebRtcWebcamBrowserTest.MANUAL_CanAcquireVga (781 ms)
[2/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/0 (453 ms)
[3/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoCapabilities/1 (388 ms)
[4/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoSettings/0 (455 ms)
[5/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GetPhotoSettings/1 (388 ms)
[6/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/0 (650 ms)
[7/15] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.TakePhoto/1 (650 ms)
[ RUN      ] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/0
Xlib:  extension "RANDR" missing on display ":9".
DevTools listening on ws://127.0.0.1:43952/devtools/browser/b9fd5f05-8389-463e-840d-7dd304bfea66
Fontconfig warning: "/etc/fonts/fonts.conf", line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf
[8193:8215:0131/110022.193070:INFO:content_browser_test_utils.cc(133)] Using camera HD Pro Webcam C920 (046d:082d)
../../content/browser/webrtc/webrtc_image_capture_browsertest.cc:198: Failure
Value of: RunImageCaptureTestCase("testCreateAndGrabFrameSucceeds()")
  Actual: false
Expected: true
[  FAILED  ] UsingRealWebcam/WebRtcImageCaptureSucceedsBrowserTest.GrabFrame/0, where GetParam() = (4-byte object <00-00 00-00>, 1-byte object <00>, 4-byte object <00-00 00-00>) (363 ms)

and all subsequent tests fail. Note that the first failing test is GrabFrame/0, which is a test that consistently passes when the /1 tests are disabled. This suggests that the issue arises somewhere during the teardown of the preceding test, TakePhoto/1. It somehow gets the camera or driver into a bad state from which it does not recover, even on subsequent build steps.

I am out of better ideas, so I propose temporarily landing a CL that changes several DVLOGs to LOGs to help us see what is going on. Even that may end up not being enough, because the interesting thing probably happens during teardown of TakePhoto/1 and --test-launcher-bot-mode seems to strip log output unless the test fails.

phoglund@: Do you have any good alternative/additional ideas/suggestions?