from https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/24394

see the logs in "shard #0 (failed)" link under the test:
2019/01/17 08:04:49 Ran 17 test(s) in 7m9.439s
2019/01/17 08:04:49 2 failed:
2019/01/17 08:04:49   video.WebRTCCamera
2019/01/17 08:04:49   video.WebRTCPeerConnCameraVP8

go to the ui logs in its isolated out:
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=84ef067c949fcf7e8d55ce1e4291a74a5c2108bd&as=ui.20190117-155508

The browser is crashing with:
"ERROR:gpu_interface_provider.cc(87)] Not implemented reached in virtual void"

https://chromium-review.googlesource.com/c/1362777 is in the blamelist of the build. Looks like it was failing like that for all but the last chromeos-amd64-generic-rel tryrun in the CQ. Not sure why the last one made it through..

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8ae71818cf945bea9f803e8a539e050da37ba054

commit 8ae71818cf945bea9f803e8a539e050da37ba054
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Jan 17 18:29:12 2019

Revert "ozone: Remove mesa i965 same-fd workaround restriction"

This reverts commit 46eb9e2770083a3789d4bde29645f905a5a9c417.

Reason for revert: causing tast failures on amd64-generic VM bot

See https://bugs.chromium.org/p/chromium/issues/detail?id=923061#c2

Original change's description:
> ozone: Remove mesa i965 same-fd workaround restriction
> 
> NOTE: This is based on crrev.com/c/549576.
> 
> After Mesa same-fd for eglCreateImageKHR restriction was relaxed in
> crrev.com/c/378198/, it is now possible to remove the workaround in chromium.
> 
> The changes are as follows:
> 1. ClientNativePixmapDmaBuf can map multiple plane using multiple FDs.
> 2. GLImageNativePixmap can create EGLImage using multiple FDs.
> 3. GbmBuffer/GbmPixmap manages multiple FDs.
> 
> BUG= 642410 , 911370
> TEST=VDA unittest and VEA unittest
> TEST=Playback with Chrome ARC++ and ARC++ DRM L1 on kevin, hana, eve
> TEST=Playback with Chrome on peach_pit and nyan_big
> 
> Change-Id: I23848b8bcd0e240307633a5201b4b8e58b374282
> Reviewed-on: https://chromium-review.googlesource.com/c/1362777
> Commit-Queue: Hirokazu Honda <hiroh@chromium.org>
> Reviewed-by: Robert Kroeger <rjkroege@chromium.org>
> Reviewed-by: Daniel Cheng <dcheng@chromium.org>
> Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
> Reviewed-by: Pawel Osciak <posciak@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#623686}

TBR=rjkroege@chromium.org,posciak@chromium.org,dcheng@chromium.org,dcastagna@chromium.org,hiroh@chromium.org

Change-Id: Ie9620b2022ec37ca5e55613d4fbe63247d357862
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  642410 , 911370, 923061
Reviewed-on: https://chromium-review.googlesource.com/c/1418124
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#623761}
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/components/arc/video_accelerator/gpu_arc_video_decode_accelerator.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/gpu/ipc/common/gpu_memory_buffer_impl_native_pixmap.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/gpu/ipc/common/gpu_memory_buffer_impl_native_pixmap.h
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/media/gpu/v4l2/generic_v4l2_device.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/media/gpu/v4l2/v4l2_slice_video_decode_accelerator.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/media/gpu/v4l2/v4l2_video_decode_accelerator.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/gfx/linux/client_native_pixmap_dmabuf.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/gfx/linux/client_native_pixmap_dmabuf.h
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/gfx/native_pixmap.h
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/gl/gl_image_native_pixmap.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/gl/test/gl_image_test_template.h
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/ozone/common/linux/gbm_buffer.h
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/ozone/common/linux/gbm_wrapper.cc
[modify] https://crrev.com/8ae71818cf945bea9f803e8a539e050da37ba054/ui/ozone/platform/drm/gpu/gbm_surface_factory.cc

video.WebRTCCamera is still failing on the most recent build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/24404. Appears to be flaky.


2019/01/17 14:13:02 Ran 17 test(s) in 3m48.84s
2019/01/17 14:13:02 1 failed:
2019/01/17 14:13:02   video.WebRTCCamera

2019-01-17T14:12:30.474952-08:00 WARNING cros_camera_service[411]: StreamOn(): Failed to set V4L2_CID_EXPOSURE_AUTO_PRIORITY
2019-01-17T14:12:30.476459-08:00 ERR cros_camera_service[411]: SetPowerLineFrequency(): Invalid setting for power line frequency: 4
2019-01-17T14:12:33.625702-08:00 INFO tast[411]: video.WebRTCCamera: Results: [{Width:640 Height:480 FrameStats:{TotalFrames:0 BlackFrames:0 FrozenFrames:0} Errors:[]} {Width:1280 Height:720 FrameStats:{TotalFrames:48 BlackFrames:0 FrozenFrames:0} Errors:[]}]
2019-01-17T14:12:33.628868-08:00 INFO tast[411]: video.WebRTCCamera: Error at webrtc.go:189: 640x480 was not healthy: no frame was displayed
2019-01-17T14:12:33.700336-08:00 INFO tast[411]: video.WebRTCCamera: ======== end

Seems there is a sig abort in the logs (which might be expected by the test?)

Received signal 6
Received signal Received signal 6
Received signal 6
6
#0 0x57a9abbde47f <unknown>
#1 0x57a9abbddff1 <unknown>
#2 0x7d20773ffab0 <unknown>
#3 0x7fff119d4aa3 ([vdso]+0xaa2)
#4 0x7d207685c3a9 __clock_gettime
#5 0x57a9abbf625a <unknown>
#6 0x57a9abb876d5 <unknown>
#7 0x57a9abb9777c <unknown>
#8 0x57a9abb89fbf <unknown>
#9 0x57a9abb8b532 <unknown>
#10 0x57a9abb96150 <unknown>
#11 0x57a9abb96731 <unknown>
#12 0x57a9abb51b67 <unknown>
#13 0x57a9abb96988 <unknown>
#14 0x57a9abb72d78 <unknown>
#15 0x57a9b021dbd4 <unknown>
#16 0x57a9ab7560ee <unknown>
#17 0x57a9ab75d6aa <unknown>
#18 0x57a9ab754e31 <unknown>
#19 0x57a9a90499af <unknown>
#20 0x7d2076770ad4 __libc_start_main
#21 0x57a9a8e6702a <unknown>
  r8: 0000000000000000  r9: 0000000000012926 r10: 023b88efd538a407 r11: 0000000000000246
 r12: 00007fff1191f418 r13: 00003bc7b22b9bd0 r14: 00007fff1191f638 r15: 00003bc7b22b57c0
  di: 0000000000000001  si: 00007fff1191f418  bp: 00007fff1191f400  bx: 0000000000000001
  dx: 0000000023b88efd  ax: 0000000000000000  cx: 00007fff119d4aa3  sp: 00007fff1191f3e0
  ip: 00007fff119d4aa3 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]

2019-01-17T14:12:04.527011-08:00 INFO crash_reporter[411]: libminijail[8128]: mount '/dev/log' -> '/dev/log' type '' flags 0x1001
2019-01-17T14:12:04.611449-08:00 WARNING crash_reporter[411]: [user] Received crash notification for chrome[7619] sig 6, user 1000 group 1000 (ignoring call by kernel - chrome crash; waiting for chrome to call us directly)
2019-01-17T14:12:04.703893-08:00 INFO cros_camera_service[411]: OnServiceMojoChannelError(): Mojo connection to CameraHalDispatcher is broken
2019-01-17T14:12:04.713666-08:00 INFO cros_camera_service[411]: main(): Child exited: status=104
2019-01-17T14:12:04.719616-08:00 INFO crash_reporter[411]: libminijail[8131]: mount '/dev/log' -> '/dev/log' type '' flags 0x1001
2019-01-17T14:12:04.870714-08:00 WARNING crash_reporter[411]: [user] Received crash notification for chrome[7685] sig 6, user 1000 group 1000 (ignoring call by kernel - chrome crash; waiting for chrome to call us directly)
2019-01-17T14:12:05.750360-08:00 WARNING chapsd[411]: Unload Token event received for unknown path: /home/root/f16ec486a70a54db36a4bfce1283b2b2097bd36d/chaps
2019-01-17T14:12:05.758879-08:00 WARNING cryptohomed[411]: Lazily unmounting stale shadow mount: /run/fuse/.ctnv4a from /home/.shadow/f16ec486a70a54db36a4bfce1283b2b2097bd36d/mount/user/GCache/v2/8da7f253839391246504f0b792f88827
2019-01-17T14:12:05.810554-08:00 INFO tast[411]: video.WebRTCCamera: Clearing device owner info

Hi, I am owner of the CL to be reverted.
Let me check.

We have a failure of video.WebRTCPeerConnCameraVP8 in a CQ run:

https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/CQ/b8924020616259276592

Is it related to this, or is it some flake ("failed to connect to Chrome")?
A quick reply would be appreciated so I can exculpate the CLs.

One more failure of this test.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8923973533220545424

Is anybody looking into this?  This (or these) bugs have killed the CQ and PFQ, so it's fairly urgent.

Here's a snippet from the failure in #7.  It looks different, but it's suspicious that these camera tests fail so much, maybe there's a common cause.
 
019/01/18 11:32:52 [11:32:50.873] Creating new page with URL http://127.0.0.1:37013/getusermedia.html
2019/01/18 11:32:52 [11:32:50.946] Connecting to Chrome at ws://127.0.0.1:41269/devtools/page/61BF577224B8F35C2296820B1A227DF7
2019/01/18 11:34:21 [11:34:20.127] Error at webrtc.go:61: Failed to evaluate enumerateDevicesError: cdp.Runtime: Evaluate: context deadline exceeded
2019/01/18 11:34:21 [11:34:20.127] Stack trace:
Failed to evaluate enumerateDevicesError
	at chromiumos/tast/local/bundles/cros/video/webrtc.runTest (webrtc.go:61)
	at chromiumos/tast/local/bundles/cros/video/webrtc.RunWebRTCCamera (webrtc.go:177)
	at chromiumos/tast/local/bundles/cros/video.WebRTCCamera (webrtc_camera.go:37)

#6: The failure at https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/CQ/b8924020616259276592 ("Failed to connect to Chrome: OOBE not dismissed: context deadline exceeded; last error follows: chrome://oobe target still exists") indicates an issue during Chrome login that's likely unrelated to what video.WebRTCPeerConnCameraVP8 is trying to investigate. Please ask the gardener to take a look; that's their job.

The failure from #7 that you quoted in #8 is indeed different and suggests an issue in the functionality that the test is trying to verify (or in the test itself). If you can't get a reply from someone on the video team and you can't find any evidence that the failures were triggered by a change that's in the CQ or a recent Chrome uprev, please temporarily add an "informational" attribute to the test so it won't run on the CQ and PFQ (see http://go/tast-failures).

- video.WebRTCPeerConnCameraVP8 is indeed failing as derat@ mentioned with:

[23:09:41.313] Error at webrtc.go:41: Failed to connect to Chrome: OOBE not dismissed: context deadline exceeded; last error follows: chrome://oobe target still exists

- Looking at the chrome logs, I can see:

[8106:8106:0117/230948.142237:ERROR:device_event_log_impl.cc(159)] [23:09:48.142] Network: network_portal_detector_impl.cc:483 Portal detection timeout:  name=Ethernet id=e422d235-2128-4159-860f-453fcf1e3628
[8106:8106:0117/230953.143243:WARNING:error_screen.cc(202)] Network error screen message is shown
[8106:8106:0117/231005.505912:ERROR:device_event_log_impl.cc(159)] [23:10:05.505] Network: network_portal_detector_impl.cc:483 Portal detection timeout:  name=Ethernet id=e422d235-2128-4159-860f-453fcf1e3628
[8106:8106:0117/231010.507718:WARNING:error_screen.cc(202)] Network error screen message is shown

- With matching timestamps. Maybe that's why OOBE was not dismissed? +jdufault@ +alemate@


- There are also several empty crash dmp.txt files which suggests minidump_stackwalk failed to symbolize these crashes:

chromium-gpu-process-minidump-37051d45e1eb85bb.dmp.txt

- There's also the infamous kernel panic reported at issue 871915.

[  156.288384] [<c031eb98>] (namespace_unlock+0xbc/0x114) from [<c03210a0>] (put_mnt_ns+0x64/0x78)
[  156.288396] [<c03210a0>] (put_mnt_ns+0x64/0x78) from [<c024df50>] (free_nsproxy+0x28/0xc4)
[  156.288409] [<c024df50>] (free_nsproxy+0x28/0xc4) from [<c024e1ec>] (switch_task_namespaces+0x64/0x68)
[  156.288421] [<c024e1ec>] (switch_task_namespaces+0x64/0x68) from [<c024e20c>] (exit_task_namespaces+0x1c/0x20)
[  156.288433] [<c024e20c>] (exit_task_namespaces+0x1c/0x20) from [<c022a2f8>] (do_exit+0x390/0x89c)
[  156.288444] [<c022a2f8>] (do_exit+0x390/0x89c) from [<c022b764>] (do_group_exit+0x5c/0xc0)
[  156.288454] [<c022b764>] (do_group_exit+0x5c/0xc0) from [<c022b7e8>] (__wake_up_parent+0x0/0x30)
[  156.288466] [<c022b7e8>] (__wake_up_parent+0x0/0x30) from [<c0205f60>] (ret_fast_syscall+0x0/0x30)
[  156.288476] Code: e5943010 e594800c e5844008 e584300c (e5973090) 
[  156.288484] ---[ end trace b57b8a4f5876fe26 ]---
[  156.290063] Kernel panic - not syncing: Fatal exception

#7: It seems that an external camera isn't detected properly. (Here, the external camera is vivid.)

Error at webrtc.go:65: Timed out waiting for video device to be available: context deadline exceeded; last error follows: "!!(checkVideoInput())" is false

https://00e9e64bacf0c4c595772a4265e925c5e71f64c9de21d3dd82-apidata.googleusercontent.com/download/storage/v1/b/chromeos-image-archive/o/amd64-generic-paladin%2FR73-11606.0.0-rc2%2Ftast_vm_test_results_2%2Ftast_vm_paladin%2Ffull.txt?qk=AD5uMEuhM5i9NHabrPLOsltyWqsRo8JWYe43bPUFUc4bYh7-0fKq8agSn8WL0b1GMdnBXWpltPJ4WZ3DphA4ziacULAwYWpZMenGR6hlO6rq2ae2w_630VpwLK8M-ka9u1fvjLuBDK3TjTxUcT6n6HzayrRi9vaAzgJ-Y1bKY0NE0WS9VosLfmCmlSY30tpgLyjtRjYLuKL01NhGMBhGtqzCm5bY59utrhqwfyKf7hzyM79y45QPkpCbN8_FDtzOUuOgFxR1PB2fmN0y3Y5bgMKBnoKDW-CaTTr8yEtLpB6JrCl3L2JEtWL55MJI8oz4jfDI6Nlab5I7dMpSrdEgyMaRpIdQslJcXR1u12VoQW3sFs9oIXSUIj_bwWsdIQWLr5ifbqz7TMHUK2YFnSI4IsJSpnr8MKCC0aEgTnNDfd8QyruSs0552W6ogN1UE3_gLKPgaGl0pKEePh5zQY6WtXQ0UDYHmyKIHiVSOHjBXFp7xpiOvMbvN0eB6UkyCfRNLu45jk2CXYYA9tJctJtPjK7UrwEyErhJkxr-RenBZcqJqkkNglmR-Xt_MmtRqp_8BP4lwoG5wq-jxezzE_j-VJq0E3qHQJedORTON0JBzL3UAM1QQf4UvpOfyITxn97erpU6Ia0JoB-VJCbe0Uq0GcB0M9vvWssqijwzwnqrYFztFbw4EnpxYzWMQdfUf1e2e09Xnr9MjvVwG-sPj9PUFqfcjtB00-HVqvBjdFTFm9dO9xwheO1MV07M4kSvGipshhuvSa2z_fOOKVSV_6lxeHkq2wRUOTED1GH-K7rHXkB4KtNIJQhDQIga3KQiuge55iD6nhOpyhGE_0pS78HfxJ9mw0eSDp3u4mqP40cL2CYCHmfX6ylJ_W4yo4mHGKmKxcVydOWAIp5s

I have no idea if we have changed related parts recently.
+CC'd camera folks.

> This (or these) bugs have killed the CQ and PFQ, so it's fairly urgent.

The (chromium) CQ appears fine. Don't see any flakes on the trybot:
https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel?limit=200

And the waterfall bot is clean:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel

We can easily disable WebRTCPeerConnCameraVP8 on the chromium bots if needed, but I'm not seeing it block anything right now.

Thanks.  CL at https://crrev.com/c/1422881

Note that at least some of these failures are on amd64-generic-paladin.  I don't know what kind of HW that is and whether it has a camera.

#14: amd64-generic builders use VMs (they're the same as betty but without ARC, as I understand it), so they'd probably also be using the vivid module.

The failure may be related to https://chromium-review.googlesource.com/c/chromiumos/platform/arc-camera/+/1405216/.

We recently had this CL to filter out camera output resolutions that < 30fps. We should limit the filtering to built-in cameras.

Let's set the WebRtcCamera test to informational while we prepare the fix. The WebRtcCamera test should probably be added to Chromium OS CQ test as well.

The CL is ready to go, I can chump it any time (or you can too).  Is there much upside to NOT chumping it?

I just checked the CQ and Chrome PFQ and they seem fine at the moment. From stainless the tast.video.WebRTCCamera looks healthy as well.

https://stainless.corp.google.com/search?test=%5Etast%5C.video%5C.WebRTCCamera%24&exclude_non_release=true&exclude_cts=true&col=build&row=board_model&view=matrix&first_date=2019-01-13&last_date=2019-01-19

So I'm not sure if we need to disable the test. If WebRTCCamera failure is a tree blocker then we shall chump the CL to unblock the CQ/PFQ.

OK, let's not chump this the time being, since it hasn't failed again.

However, I recommend fixing this quickly.  Even a rare flake is hurtful.  Last week the CQ succeeded only a handful of times, but only one failure was due to a bad CL.

chrome_all_tast_tests seems to fail again multiple times: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel

#21: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/24509 looks like it was the same video.WebRTCCamera failure as in #4:

2019/01/21 03:57:37 Running video.WebRTCCamera
2019/01/21 03:57:37 Restarting ui job
2019/01/21 03:57:40 Clearing device owner info
2019/01/21 03:57:41 Waiting for org.chromium.SessionManager D-Bus service
2019/01/21 03:57:41 Asking session_manager to enable Chrome testing
2019/01/21 03:57:41 Waiting for Chrome to write its debugging port to /home/chronos/DevToolsActivePort
2019/01/21 03:57:42 Removing cryptohome for testuser@gmail.com
2019/01/21 03:57:42 Finding OOBE DevTools target
2019/01/21 03:57:42 Connecting to Chrome at ws://127.0.0.1:35991/devtools/page/13EF6D448BF0F19296293BB9E3457540
2019/01/21 03:57:42 Waiting for OOBE
2019/01/21 03:57:49 Logging in as user "testuser@gmail.com"
2019/01/21 03:57:50 Waiting for cryptohome for user "testuser@gmail.com"
2019/01/21 03:57:58 Waiting for OOBE to be dismissed
2019/01/21 03:58:01 Creating new page with URL http://127.0.0.1:39781/getusermedia.html
2019/01/21 03:58:01 Connecting to Chrome at ws://127.0.0.1:35991/devtools/page/0BFECD466F42E58134D15C8381CEC778
2019/01/21 03:58:11 Results: [{Width:640 Height:480 FrameStats:{TotalFrames:0 BlackFrames:0 FrozenFrames:0} Errors:[]} {Width:1280 Height:720 FrameStats:{TotalFrames:24 BlackFrames:0 FrozenFrames:0} Errors:[]}]
2019/01/21 03:58:11 Error: [webrtc.go:189] 640x480 was not healthy: no frame was displayed
2019/01/21 03:58:11 Finished video.WebRTCCamera

I don't see any reason for this issue to have Restrict-View-Google.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/e2253754392cfd9b8ff4be25e381876e97a11f03

commit e2253754392cfd9b8ff4be25e381876e97a11f03
Author: Luigi Semenzato <semenzato@chromium.org>
Date: Mon Jan 21 13:33:58 2019

tast-tests: Make video.WebRTCCamera informational temporarily.

Temporary fix until we find out why it fails in various builds.

BUG=chromium:923061
TEST=ran on DUT

Change-Id: Id128e18c41068e8e4e4c087c43330e2a3b2147f2
Reviewed-on: https://chromium-review.googlesource.com/c/1422881
Tested-by: Luigi Semenzato <semenzato@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/e2253754392cfd9b8ff4be25e381876e97a11f03/src/chromiumos/tast/local/bundles/cros/video/webrtc_camera.go

I chumped CL:1422881 to avoid the WebRTCCamera from blocking CQ. Henry has a fix CL:1423891 in the CQ. Let's re-enable the test after we verify the fix.