New issue
Advanced search Search tips

Issue 810133 link

Starred by 1 user
Status: Fixed
Owner: ----
Closed: Aug 7
Cc:
m...@chromium.org
mfo...@chromium.org
powerb@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Chrome , Mac
Pri: 2
Type: Bug

Blocking:
issue 828274
issue 797470



Sign in to add a comment

Move FrameSink video capture cursor rendering into VIZ

Project Member Reported by m...@chromium.org, Feb 7 2018 
To unblock the VIZ launch, all screen video capture cursor rendering is being done in the browser process as a post-processing step (in content::FrameSinkVideoCaptureDevice). However, the original design called for this to happen in VIZ, perhaps even as part of the display compositing pipeline.

Note that the main blocker was due to a "plumbing" concern: We need to resolve how to get mouse move/click information to the viz::FrameSinkVideoCapturerImpl. It was not clear how the APIs would change around that, or how the mouse events would reach VIZ from the Window Service (or the browser process).

Comment 1 by m...@chromium.org, May 16 2018

Blocking: 797470
Labels: -Pri-3 M-70 Pri-2
Seems that this is now blocking Chromium-wide migration to the new shared memory API. Tentatively targeting for M70, but this would depend on lower-level APIs becoming available: the new shmem API and the mojo API changes to go along with it: https://docs.google.com/document/d/1lk2-7W7Cy7FDqG-6lgS8LRkzXABYTOytf6VUsWNf8-0/edit?usp=sharing
Project Member

Comment 2 by bugdroid1@chromium.org, Jul 12 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b7bed8483e07637b8b24e7492d97bbc0499b6c16

commit b7bed8483e07637b8b24e7492d97bbc0499b6c16
Author: Yuri Wiitala <miu@chromium.org>
Date: Thu Jul 12 03:36:13 2018

Add viz::VideoCaptureOverlay to support mouse cursor rendering VIZ-side.

VideoCaptureOverlay provides the post-GPU-readback rendering of small
SkBitmaps onto the VideoFrames captured by viz::FrameSinkVideoCapturer.
This will be used to move mouse cursor rendering out of the browser
process, and into VIZ, as was intended in the original design. More
details in crbug/810133.

A soon-upcoming series of changes will provide mojo IDL definition as
well as integration of VideoCaptureOverlay with FrameSinkVideoCapturer.

Bug:  810133 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Change-Id: I16897dbd361af153d10a1223339d9fb5f7e5f05a
Reviewed-on: https://chromium-review.googlesource.com/1132318
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#574480}
[modify] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/service/BUILD.gn
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/service/frame_sinks/video_capture/video_capture_overlay.cc
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/service/frame_sinks/video_capture/video_capture_overlay.h
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/service/frame_sinks/video_capture/video_capture_overlay_unittest.cc
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_clips_ll.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_clips_lr.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_clips_ul.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_clips_ur.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_full_cover.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_full_cover_scaled.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_moves_0_0.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_moves_1_0.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_moves_2_0.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_moves_2_1.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_moves_2_2.png
[add] https://crrev.com/b7bed8483e07637b8b24e7492d97bbc0499b6c16/components/viz/test/data/video_capture/overlay_moves_lr.png

Comment 3 by m...@chromium.org, Jul 24

Blocking: 828274
Project Member

Comment 4 by bugdroid1@chromium.org, Jul 25 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77

commit 03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77
Author: Yuri Wiitala <miu@chromium.org>
Date: Wed Jul 25 21:45:44 2018

Add MouseCursorOverlayController.

This controller is heavily based on content::CursorRendererXYZ,
providing the logic for processing platform mouse events and using them
to control the new viz::VideoCaptureOverlay.

In a later CL, this will be integrated with the tab/desktop capture
stack, to complete the migration discussed in crbug 810133.

Bug:  810133 ,828274
Change-Id: I6502e1ef3cbc878e1c9b062b0abf6a0e6229f860
Reviewed-on: https://chromium-review.googlesource.com/1149264
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578079}
[modify] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/BUILD.gn
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller.cc
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller.h
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller_aura.cc
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller_browsertest.cc
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller_mac.mm
[modify] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/test/BUILD.gn
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 1 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d

commit 209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d
Author: Yuri Wiitala <miu@chromium.org>
Date: Wed Aug 01 01:15:29 2018

Integrate VideoCaptureOverlays into viz::FrameSinkVideoCapturer.

Introduces the mojo IDL and implementation "glue" to be able to create
overlays to be rendered on top of each video frame captured by a
FrameSinkVideoCapturer. This allows for use cases like mouse cursor
rendering and debug status displays. Also, the new overlay APIs have
been added to ClientFrameSinkVideoCapturer (a client-side wrapper that
handles the auto-reconnecting-to-VIZ after a VIZ process crash).

Bug:  810133 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel
Change-Id: I963b6a1d736e1510618a723694f3b22ac0bf1589
Reviewed-on: https://chromium-review.googlesource.com/1150929
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Reviewed-by: Andrey Kosyakov <caseq@chromium.org>
Reviewed-by: Saman Sami <samans@chromium.org>
Reviewed-by: Mustafa Emre Acer <meacer@chromium.org>
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#579654}
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/host/client_frame_sink_video_capturer.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/host/client_frame_sink_video_capturer.h
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/service/frame_sinks/video_capture/frame_sink_video_capturer_impl.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/service/frame_sinks/video_capture/frame_sink_video_capturer_impl.h
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/service/frame_sinks/video_capture/video_capture_overlay.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/service/frame_sinks/video_capture/video_capture_overlay.h
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/components/viz/service/frame_sinks/video_capture/video_capture_overlay_unittest.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/content/browser/devtools/devtools_video_consumer_unittest.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/content/browser/media/capture/frame_sink_video_capture_device_unittest.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/content/browser/media/capture/lame_window_capturer_chromeos.cc
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/content/browser/media/capture/lame_window_capturer_chromeos.h
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/services/viz/privileged/interfaces/compositing/BUILD.gn
[modify] https://crrev.com/209b438b5b9f22a4a8a5f55a95c7d5ae88a6e54d/services/viz/privileged/interfaces/compositing/frame_sink_video_capture.mojom
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 1 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e

commit c1ff2ed2c0b33ef6d0de60f0c034486a4519261e
Author: Yuri Wiitala <miu@chromium.org>
Date: Wed Aug 01 20:09:23 2018

'Place-holder' overlay rendering for CrOS window capture.

A minimal FrameSinkVideoCaptureOverlay implementation for browser window
video capture on ChromeOS (i.e., not desktop capture, and not
WebContents capture). The intention is for this code to be a temporary
solution until more work is complete on services/ui/ws (the new Window
Service).

Bug:  810133 , 806366 
Change-Id: I4f929248a9620f96f2a538caaac9761a472cbe09
Reviewed-on: https://chromium-review.googlesource.com/1155985
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#579913}
[modify] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/browser/BUILD.gn
[add] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/browser/media/capture/lame_capture_overlay_chromeos.cc
[add] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/browser/media/capture/lame_capture_overlay_chromeos.h
[add] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/browser/media/capture/lame_capture_overlay_chromeos_unittest.cc
[modify] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/browser/media/capture/lame_window_capturer_chromeos.cc
[modify] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/browser/media/capture/lame_window_capturer_chromeos.h
[modify] https://crrev.com/c1ff2ed2c0b33ef6d0de60f0c034486a4519261e/content/test/BUILD.gn
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 4 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c33f27b093f8ca5d3172e9bea6e99b6b6848237c

commit c33f27b093f8ca5d3172e9bea6e99b6b6848237c
Author: Yuri Wiitala <miu@chromium.org>
Date: Sat Aug 04 02:35:35 2018

Switch mouse cursor rendering over to VIZ VideoCaptureOverlays.

Migrates all client-side mouse cursor rendering code to stop using the
old content::CursorRenderer and instead use the new
content::MouseCursorOverlayController+viz::FrameSinkVideoCaptureOverlay
system.

Testing: Tested mouse cursor rendering behavior on Win+Mac+CrOS+Linux
for both tab capture and browser window capture, and also desktop
capture on CrOS.

Bug:  810133 
Change-Id: Ibb0fd28a8e1ea99a9ca27d9940af3f46bd48d6ce
Reviewed-on: https://chromium-review.googlesource.com/1162828
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#580729}
[modify] https://crrev.com/c33f27b093f8ca5d3172e9bea6e99b6b6848237c/content/browser/media/capture/aura_window_video_capture_device.cc
[modify] https://crrev.com/c33f27b093f8ca5d3172e9bea6e99b6b6848237c/content/browser/media/capture/frame_sink_video_capture_device.cc
[modify] https://crrev.com/c33f27b093f8ca5d3172e9bea6e99b6b6848237c/content/browser/media/capture/frame_sink_video_capture_device.h
[modify] https://crrev.com/c33f27b093f8ca5d3172e9bea6e99b6b6848237c/content/browser/media/capture/mouse_cursor_overlay_controller.h
[modify] https://crrev.com/c33f27b093f8ca5d3172e9bea6e99b6b6848237c/content/browser/media/capture/web_contents_video_capture_device.cc
Project Member

Comment 8 by bugdroid1@chromium.org, Aug 7 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e11bd21352639040acc33499da21657a1f0e7283

commit e11bd21352639040acc33499da21657a1f0e7283
Author: Yuri Wiitala <miu@chromium.org>
Date: Tue Aug 07 02:31:39 2018

Remove content::CursorRenderer.

Now that mouse cursor rendering on screen video captures has fully
switched-over to the new VIZ implementation, the old implementation is
dead code and can be deleted.

Bug:  810133 
Change-Id: Ic6a480048019d480b90970ab78443cf68fe089c6
Reviewed-on: https://chromium-review.googlesource.com/1162865
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#581101}
[modify] https://crrev.com/e11bd21352639040acc33499da21657a1f0e7283/content/browser/BUILD.gn
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer.cc
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer.h
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer_aura.cc
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer_aura.h
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer_aura_unittest.cc
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer_mac.h
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer_mac.mm
[delete] https://crrev.com/c34944f3e14b3684c4d4f1791edbcf116e0cc511/content/browser/media/capture/cursor_renderer_mac_unittest.mm
[modify] https://crrev.com/e11bd21352639040acc33499da21657a1f0e7283/content/test/BUILD.gn

Comment 9 by m...@chromium.org, Aug 7

Status: Fixed (was: Available)
►

Sign in to add a comment