BTW--I am totally willing to dedicate some of my time to design/discussions/meet-ups, and even author/review code changes to ensure we have a great migration here. :)

Several aspects to consider:

1. In the past, for performance reasons, we'd prefer the "FrameSubscriber" to get screen-captured frames as the renderer composites frames and notifies the browser process. However, the mechanism we've always had is a "passive" approach where we issue copy requests after a "sampling decision," and *hope* they execute (which only tends to happen when a full browser composite occurs). That means we miss content changes from the renderer in some cases, reducing the buttery-smoothness of tab-captured video (mostly on low-end machines).

2. Latency/Timing information: Because we are moving capture to a separate process, we will now need accurate reference clock (e.g., base::TimeTicks) timing information on the captured frames. This is necessary to ensure A/V synchronization for "video streaming --> tab capture" use cases. There are already performance_browser_tests on the perf bots measuring A/V sync; and huge regressions will result if we do not correctly plumb-in the necessary timing information.

3. Sampling logic: For frame subscription, there is some rather involved logic for deciding which frames should be captured whenever content changes. This would have to be moved to the Mus process (most likely). However, this logic uses feedback signals from the Video Capture Stack, tab contents resize events, and a few other pieces of information as part of its decision-making/control logic. It's not entirely clear to me how much of the object graph we want to move out of the browser process here.

4. Integration with Chrome Video Capture Stack: Currently, tab capture (and desktop or browser window capture) are all implementations of the media::VideoCaptureDevice interface. Whether for screen capture, or for webcam/video capture from the operating system, all impls of VCD currently live in the browser process. They generate video frames into shared memory, and then IPC-notify consumers in render processes to deliver the frames. The Video Capture Stack itself it undergoing mojoification, with later plans to migrate out of the browser process. However, we will need to coordinate with those folks (chfremer@ and mcasas@ are currently main on this work).

5. Throttling: It would be great, again for performance reasons, to have a reliable feedback mechanism as to how hard capture is hitting the GPU (i.e., resources needed for GPU readback during/after compositing). With a utilization feedback signal, we could automatically reduce resolution or sample frame rate to allow GPUs on low-end systems to keep-up. See "Measuring Utilization" subsection "GPU" in: https://www.chromium.org/developers/design-documents/auto-throttled-screen-capture-and-mirroring That text describes a previous approach that did not pan-out; but the rest of the design doc was implemented and launched with great results over the past 2 years.

6. Offscreen capture: There is an "offscreen tab" + "tab capture" implementation that supports 1) the W3C Presentation API, and 2) the chrome.tabCapture.captureOffscreenTab() extension API. While there have never been issues with Mac, on Aura, we've had to implement ugly hacks to ensure compositing and capture of offscreen tabs is executed. As part of this migration, it would be great to have dedicated support for offscreen tab rendering and capture; even when there are no browser windows open on-screen. Reference (the hack): https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/tab_capture/tab_capture_registry.cc?rcl=6761779daf843b5afefd405fc3761e961d04313c&l=40

7. Code de-duping: "CrOS desktop capture" and "all-Aura browser window capture" have duplicated a lot of code found in content::DelegatedFrameHost (in content::AuraWindowCaptureMachine). Our goal here, for these platforms, could be to simply support the capture of any "aura::Window" entity; and be able to "FrameSubscribe" to its content changes. There are multiple crbugs open around this.

> a "passive" approach where we issue copy requests after a "sampling decision," and *hope* they execute (which only tends to happen when a full browser composite occurs)

I think this is referring to the fact copy requests go thru the UI compositor's layer tree right now. We should design this to be built directly around surfaces - either top level surfaces (application UI) or child surfaces (tab contents) or unparented/non-visible surfaces (offscreen). Then they'll be captured on each draw instead of being tied to the commit/submit rate from the browser UI like today.

> It's not entirely clear to me how much of the object graph we want to move out of the browser process here.

I think one key thought is that this will now be used for capturing general applications, it's no longer chrome-specific. It can use feedback signals from the App (from the browser for chrome) but it will also be used for android apps and whatever else chromeos  could potentially run that are shown thru the display compositor.

> 5. Throttling: It would be great, again for performance reasons, to have a reliable feedback mechanism as to how hard capture is hitting the GPU 

Gpu scheduler should give us some good ways to get feedback in the gpu process where display compositor and this stack would live.

> As part of this migration, it would be great to have dedicated support for offscreen tab rendering and capture;

If everything is done in terms of surfaces this should be very reasonable and come naturally I think. I suspect the hacks in aura are because copy requests are going thru the UI compositor's layer tree. If in terms of surfaces instead, in the display compositor, there's no need to ensure things are part of the "scene" but hidden and such.

> > As part of this migration, it would be great to have dedicated support for offscreen tab rendering and capture;

> If everything is done in terms of surfaces this should be very reasonable and come naturally I think. I suspect the hacks in aura are because copy requests are going thru the UI compositor's layer tree. If in terms of surfaces instead, in the display compositor, there's no need to ensure things are part of the "scene" but hidden and such.

We currently have multiple display compositors, so we'd need to be able to pick one to draw the offscreen tab and be able to choose when to draw it independently vs. drawing it as part of drawing an onscreen frame. I agree that doing this based around surfaces should allow for a simpler mechanism, though.

Taking ownership of prototype and initial design.

It seems like we've settled on the general approach discussed in the prior comments on this bug. We had several face-to-face meetings and a long e-mail thread discussion among team members. To summarize:

First, there was an idea to introduce a separate "capture cc::Display," cc::OutputSurface, cc::DisplayScheduler, etc., that would run alongside the normal cc::Display. This may still be the right long-term solution, but is not the one we are going to pursue for now (mostly due to logistics). There are numerous new concepts that would have to be accounted for to support this, such as: 1) The idea of sharing the results of RenderPasses across multiple Displays' draws, regardless of which Display executed the draw of the original RenderPass, to prevent redundant compositing. 2) Sharing a single BeginFrameSource, and resolving different desired framerates.

The goal with capture is that, in the Mus world, we want to basically have the VIZ process capture any Surface with certain framerate/resolution constraints and policy, and emit results to an external process (e.g., a video encoder running in a render process). Therefore, we're going back to an earlier idea: We will simply add a "Frame Subscriber" concept to Surfaces, with the "subscriber" also running in the VIZ process alongside everything. "Subscribers" would be registered on a Surface, and SurfaceAggregator::Aggregate() would:

1. Prewalk the tree to identify the damage rect (as it does now).
2. Query any subscribers it encounters to determine if they want a texture containing a copy of the of the re-drawn region.
3. If so, add CopyOutputRequests to the root RenderPass (to have DirectRenderer execute them during Draw()). The copy result would be passed back to the subscriber as a texture mailbox handle.

Downstream of this, we would execute the scaling and I420/YUV conversion; and handle all the other cross-process integration concerns. Basically, we would take relevant parts of the code from DelegatedFrameHost, content::GLHelper and media/capture/content, and consolidate them into a simpler chunk of code that runs in the VIZ process.

All of the above is focused on changes in the compositor, but there is still design work to do to figure out how to integrate this with new Mus UI concepts and cross-process APIs. I'll begin prototyping on my desktop to figure this out, and then write-up a proposal design doc that encompasses all changes at all layers/boundaries: 1) compositing/surface changes; 2) Mus APIs; 3) integration with the media::VideoCapture stack.

Updates:

1. For interfacing (mojo), I wanted to make sure that both the Mus and non-Mus code could share the same capturer implementation (i.e., either from the window service or from the likes of DelegatedFrameHost). I'm currently working on Mojo interfaces that define the APIs needed to capture video of a CompositorFrameSink and be able to switch to capture different CompositorFrameSinks during operation (e.g., as RenderWidgetHostViews are created and destroyed during tab navigation).

2. I've reviewed the existing implementation of capture, GPU readback, the YUV conversion, etc.; and talked with a few stakeholders to get the "tribal knowledge" behind these. I've also discussed performance concerns w/ Hubbe, as well as color space handling (and possible future support for HDR).

3. I'm going to begin prototyping the expansion of cc::CopyOutputRequests to produce YUV textures. This will involve pulling-out some of the code from viz::GLHelper such that the YUV conversion code paths are woven into cc::GLRenderer's draw of a render pass. This should greatly improve performance and GPU memory usage since we no longer will need to make a full RGB texture copy after a render pass like we do now. Instead, we will be able to read from the original texture directly and do YUV conversion from that. More details as I discover them... :-)

4. Assuming #3 pans out, I'm almost certain the remaining "glue" will fall into place nicely. I'll be in a good position to write a design doc for wider-audience review. Ideas/Approaches have changed somewhat since c#8.

Update: Was OOO last week. On the few days prior to that, I made a ton of progress on prototyping. The only thing left to check is how to move YUV conversion into the direct renderer for the CopyOutputRequests. I've already figured out a simple way to extend the existing CopyOutputRequest API to request YUV texture results.

Closing out this old bug. Since we designed the thing and the implementation at this point is mostly landed, I'd say we achieved the goal stated in the Summary. :)