New issue
Advanced search Search tips

Issue 625237 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 2
Type: Bug



Sign in to add a comment

Mouse position inaccurate for tabCapture MediaStream

Reported by vi...@opentest.co, Jul 1 2016

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36

Steps to reproduce the problem:
1. Get a reference to a tabCapture MediaStream via the chrome.tabCapture API.
2. Get a reference to a microphone MediaStream via the window.navigator.webkitGetUserMedia API.
3. Add the audio track from the microphone MediaStream to the tabCapture MediaStream.
4. Record the resultant stream via the MediaRecorder API.

What is the expected behavior?
The mouse position should be accurate.

What went wrong?
The mouse position is off. It seems that this is not reproducible for everyone, but that it is consistently reproducible for those that it affects. 

Did this work before? N/A 

Chrome version: 51.0.2704.106  Channel: stable
OS Version: OS X 10.11.4
Flash Version: Shockwave Flash 22.0 r0

We have a running thread of this issue for our own Chrome extension here:

https://productpains.com/post/opentest/mouse-pointer-is-recorded-wrong

Everyone posting here has video evidence of the bug as well as their OS and Chrome Version posted, so I'm sure that would be super useful (also free range for Chrome peeps to ask people in that thread directly for more insight into their devices).
 
Components: -UI Blink>Media

Comment 2 by vi...@opentest.co, Jul 12 2016

Hey guys just wondering what the process was for checking in on this for progress. Has anyone looked into it yet? We're getting more videos and OS/browser versions showcasing this issue if that helps at all. Please let me know what I could personally do to debug what may be going on as well (even firing up Chrome from terminal with some sort of flag).

Comment 3 by vi...@opentest.co, Jul 12 2016

Here are several videos showcasing the bug and with the accompanying OS and browser versions:

https://www.opentest.co/share/7b0f57f03d2b11e69dccf53440b4b495 (Mac Chrome 51.0.2704.103 (64-bit))

https://www.opentest.co/share/8d0954103ecf11e6809e67830a797369 (OS X Yosemite 10.10.5 Chrome Version 51.0.2704)

https://www.opentest.co/share/b1f2ec20479711e6a319771b94a5ca8d (Chrome 51.0.2704.103 (64-bit) Mac v 10.11.3 (OS X El Capitan))

https://www.opentest.co/share/7de65cc0479f11e6abafd17c7608c0b8 (Google Chrome Version 51.0.2704.103 (64-bit) AND Version 53.0.2783.2 canary (64-bit) on OS X El Capitan v10.11.5)

Comment 4 by vi...@opentest.co, Jul 14 2016

https://www.opentest.co/share/cd30118049d511e6a319771b94a5ca8d (Chrome 51.0.2704.103 (64-bit) OS X 10.11.5)
Cc: mcasas@chromium.org
Components: Internals>Media>Capture

Comment 6 by mcasas@chromium.org, Jul 15 2016

Cc: sergeyu@chromium.org qiangchen@chromium.org m...@chromium.org

Comment 7 by mcasas@chromium.org, Jul 15 2016

Components: -Blink>Media Blink>GetUserMedia>Desktop
Cc: -qiangchen@chromium.org
Owner: qiangchen@chromium.org
Status: Assigned (was: Unconfirmed)
As the raw captured frame does not naturally contain the mouse pointer.
It is chrome code to compose the mouse pointer onto the frame by coordinate detecting.

The bug shows we did not resolve the coordinate offset correctly under some circumstances.

I will take a look.

Comment 9 by vi...@opentest.co, Jul 19 2016

Thank you @qiangchen!
I think I've found out the method to reproduce the issue:
1. Open any website and Download something, so that you have the download status bar at the bottom of Chrome. (Do NOT close that status bar!)
2. Do a tab sharing, then you are going to see that the mouse point is off.

The underlying reason is that 
In [1], we take the mouse coordinate, but this coordinate is with respect to the Chrome window, but later we simply use this coordinate as if it is with respect to the tab content, then there is a vertical shift with amount equal to the height of the status bar. 

P.S. 
1. In Mac the left lower corner has coordinate (0,0).
2. I believe that if you can make chrome have any non-web-content view  attaching on the left side of Chrome, there will be a horizontal shift of mouse point during tab capture. Though I did not think of any way to do that, maybe some extension made their UI be a view attached to the left side of chrome?

[1] https://cs.chromium.org/chromium/src/content/browser/media/capture/cursor_renderer_mac.mm?dr=C&q=CursorRendererMac&sq=package:chromium&l=63

Comment 11 by vi...@opentest.co, Jul 20 2016

Hmm that's very interesting. Yeah coordinates should definitely be calculated with an offset to draw the mouse relative to the tab if it's a tabCapture. Great catch!

As for the non-web-content view, how would someone do that via an extension? I figured you can't since extensions can only interact within the DOM from the content script.

I doubt that's the case for a lot of these people in the videos - unless there are side bars you can open/close in Chrome (none come to mind). My naive theory for the horizontal offset is that there may be rounding/offset errors that become exacerbated if operated on multiple times. Or perhaps it's the same issue with the scroll bar being present messing up the calculation.

Either way, your theory about the vertical offset seems to be true, although there were no issues horizontally (which makes sense) when I had the download footer bar open:

https://www.opentest.co/share/700a8c604eb711e68356998afd9fab62
Status: Started (was: Assigned)
https://codereview.chromium.org/2166953002/
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f5a740f351bda75b14e6631baf00a0764f86ea15

commit f5a740f351bda75b14e6631baf00a0764f86ea15
Author: qiangchen <qiangchen@chromium.org>
Date: Mon Jul 25 23:56:23 2016

Bug Fix: Tab Capture Mouse Pointer Off On Mac

The cursor renderer on Mac, takes the chrome window based coordinate
as web contents based coordinate.

This CL fixes that bug.

BUG= 625237 

Review-Url: https://codereview.chromium.org/2166953002
Cr-Commit-Position: refs/heads/master@{#407637}

[modify] https://crrev.com/f5a740f351bda75b14e6631baf00a0764f86ea15/content/browser/media/capture/cursor_renderer_mac.mm

Status: Fixed (was: Started)

Comment 15 by vi...@opentest.co, Jul 26 2016

@qiangchen awesome that this is fixed! Do you have any idea what version of Chrome this would make it into? Or is that still pretty much TBD for a while? Would just like to update the bug thread for our little Chrome extension community. :-)
It will be rolled out with M54.


Comment 17 by vi...@opentest.co, Jul 26 2016

Cool thanks for the update!

Comment 18 by vi...@opentest.co, Sep 27 2016

Hey guys - seems like the cursor is not showing up at all for one of our users on Mac OS Sierra Chrome 53.

https://www.opentest.co/share/aabd147084c911e68035af33b8ecc370

Not sure if you guys would like me to open another bug, but this seems like a regression.

Sign in to add a comment