New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 890928 link

Starred by 3 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Feature

Blocked on:
issue 526883



Sign in to add a comment

Check the availability of MS new Screen Capture API in Chromium on Windows10

Reported by jiso...@oblong.com, Oct 1

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36

Example URL:

Steps to reproduce the problem:
In Windows10

1. Open Opera or Electron or any browser/app that uses Chromium/webrtc
2. Start Hangouts and start a call
3. With the window picker selects the window of Google Chrome and share it

What is the expected behavior?
The remoter can see Google Chrome

What went wrong?
The remoter see black frames.

Did this work before? No 

Is it a problem with Flash or HTML5? HTML5

Does this work in other browsers? N/A

Chrome version: 69.0.3497.100  Channel: stable
OS Version: 
Flash Version: 

Contents of chrome://gpu: 

This is working fine on Windows7 and Windows8. This is issue goes away when hangouts is run in the same app that the one being shared as it uses a different way to share the content, i.e. AuraWindowVideoCaptureDevice.

The issue comes from webrtc::WindowCapturerWin.cc which should maybe be improved to use a newer system/win32 API to capturer a window content. (Like the ScreenCapturerWind{gdi, directx, magnifier})

See https://chromium.googlesource.com/external/webrtc/+/master/modules/desktop_capture/window_capturer_win.cc#312
 
Cc: braveyao@chromium.org zijiehe@chromium.org
Components: -Internals>Media Blink>WebRTC
Chrome window doesn't support GDI capture methods. So cropping it out from a screenshot is the only way (we can't utilize the new IE getDisplayMedia() API yet, which looks like need its own picker).
The Chrome window in the picker should has black frame too.

Is screencapture working on that machine?
Yes the screencapture is working, probably because it defaults to ScreenCapturerDirectx instead of ScreenCapturerGdi. But cropping out its result won't work if the captured app/window is being occluded. Is there a way to use directx for the WindowCapturerWin too ?
It's weird. AFAICT, cropping window out should work with both ScreenCaptureDirectX and ScreenCaptureDgi. So if screencapture works, then we should be able to get the Chrome window.

I don't think MS will offer other way to capture a window for Win32 API, but focusing on their UWP APIs(with its own picker), which we can't utilize so far. 

Is there any log which may show the problem?
I think I miss understood. Do you mean to take an image with ScreenCaptureDirectX then crop it to the chrome window using its position and size ? I am not sure to see how it can work if the chrome window is covered by other application window. (I assumed it won't work so I have not tried) Thx!
Labels: Needs-Triage-M69
No, Chrome can't capture occluded windows of UWP apps. This is a knows issue, which is also a necessary compromise to the current situation on Win10.
Blockedon: 526883
Oki thx. I did more digging and found the following:

From https://codereview.webrtc.org/1371383003 "It was confirmed from Microsoft that there is no support for modern app's window capture." , i.e. UWP (https://en.wikipedia.org/wiki/Universal_Windows_Platform)

So maybe Google Chrome should be filtered out and the current filter https://codereview.webrtc.org/1371383003/patch/80001/90001 should be improved ?

Also it seems this filter "(wcscmp(class_name, L"ApplicationFrameWindow") == 0 ||
wcscmp(class_name, L"Windows.UI.Core.CoreWindow") == 0))" does not work for Microsoft Power point and other MS Office apps.





Update on the reproduction steps from comment #0:

when the hardware acceleration is disabled in Google Chrome settings, then WindowCapturerWin.cc works. So I am confused now, if Google Chrome was a UWP application the capture should not work no matter if hw accel is on or off right ?
No Chrome is not a UWP app. But the root reason is same: HW accelerated content.

MS Office apps, and Chrome, can be captured with the cropping method. So we'd like to list them in the picker. That filter will filter out many system windows which are not visible to user while IsWindowVisible() returns TRUE.

Yes that logic is quite messy. Hope there is better rule and capture method.
oki thx for the details. How can I try to the cropping method with Google Hangouts ? Or is it something not yet available ?
Cropping is the default for window capture on Win10. Try to capture a window and cover it with other window during capturing. Different target window will have different result, depending on how it supports printWindow/BitBlt.
Components: -Blink>WebRTC Blink>GetUserMedia>Desktop
I had a deeper look and I see what you were saying now. 

I also had a look to the implementation: CroppingWindowCapturerWin::ShouldUseScreenCapturer()

https://chromium.googlesource.com/external/webrtc/+/master/modules/desktop_capture/cropping_window_capturer_win.cc#170

(And where it gets instantiated https://chromium.googlesource.com/chromium/src.git/+/71.0.3568.2/content/browser/media/capture/desktop_capture_device.cc#530)

On the win10 laptop I use for this, it is working fine when the Chrome window is on the foreground, just like you mentioned.

But whenever the Chrome window goes beyond the visible screen are, even by 1 pixel then I can see black frames on the remoter. Same when the Chrome window is partially covered even by 1 pixel.
Should the remoter see the last captured frame instead ?

For Microsoft Powerpoint the remoter sees a very old frame instead of the very last shared frame (can be a very old frame) so this confuses the user.

1 - Do you think this can be improved to make sure to see the last shared frame ? (instead of black or very old frame)  Maybe it is supposed to do that and there is a bug ?

2 - When being on foreground but partially out of the screen, do you think it can be improved to still share the visible part ? (for the invisible part, it can copy it from the last frame but I can imagine how it can end up with a weird image on the remoter)

3 - About " ... focusing on their UWP APIs(with its own picker), which we can't utilize so far " : is there any chromium or webrtc bug entry to track that effort ? If no, what is preventing us to utilize UWP apis ?

Thx!
1. It's the strategy to show black frame if capturing fails. The problem of Office window is another one, which is from the OS GDI method as we fallback to it. Some other window, i.e. File Explorer, work fine with GDI methods. So fallback is still the best choice to attempt with. 

2. It's hard to evaluate all kinds of window themes. For example, with or without board, content margin, etc. No good idea so far.

3. As to the new UWP apis, first we haven't tried yet as it's not available on our Win workstations yet, the second concern is it has to bring up its own picker to proceed the selecting and capturing target window. We have no idea if we can work around it yet.
Components: Internals>WebRTC
Labels: -OS-Linux -Type-Bug OS-Windows Type-Feature
Thx for the clarification on 1 and 2. So I understand that the current behavior described in this bug description is a worksform.

It seems there is no bug open to investigate using UWP apis in chromimu-webrtc-window-capturer-win so let's keep this bug open. This will avoid someone else opening a new bug and asking the same questions (usually people search for existing bugs in the open list) and we can keep track of the UWP investigation here.

(Related to UWP there is an open bug about making Chrome an UWP app, see https://bugs.chromium.org/p/chromium/issues/detail?id=759949, but this is a different subject)

So this bug is now about 3) above.


Cc: -zijiehe@chromium.org
Summary: Check the availability of MS new Screen Capture API in Chromium on Windows10 (was: Black frames when sharing Google Chrome with Hangouts on Windows10)
Change the summary accordingly.
And here is the official doc: https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture
Cc: vamshi.kommuri@chromium.org
Labels: Triaged-ET Target-71 M-71 FoundIn-71 FoundIn-70 FoundIn-69
Status: Untriaged (was: Unconfirmed)
From comment#17, this seems to be a Feature request hence marking it as Untriaged. Adding the appropriate labels.

Thanks!
Status: Available (was: Untriaged)

Sign in to add a comment