New issue
Advanced search Search tips

Issue 828274 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug

Blocked on:
issue 810133



Sign in to add a comment

[GetUserMedia API]wrong cursor position during Mouse Cursor Dragging

Reported by k.nobuha...@gmail.com, Apr 3 2018

Issue description

UserAgent: Mozilla/5.0 (X11; CrOS armv7l 10452.30.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.67 Safari/537.36
Platform: 10452.30.0 (Official Build) beta-channel veyron_minnie

Steps to reproduce the problem:
1. share desktop and start capture desktop
2. drag mouse cursor(move window or drag on desktop)
3. stop share and stop capture

What is the expected behavior?
the mouse cursor moves according to dragged tracks

What went wrong?
the mouse cursor potion wrong(not move while it is dragging)

Refer to he attached video file
1.CursorCaptureIssue_UseDesktopCapture.webm
 used app: Desktop Capture Sample
2.CursorCaptureIssue_UseScreencastify.webm
 use app: Screencastify-Screen Video Recoder

Did this work before? N/A 

Chrome version: 66.0.3359.67  Channel: beta
OS Version: 10452.30.0
Flash Version: 29.0.0.134 /opt/google/chrome/pepper/libpepflashplayer.so

Use navigator.webkitGetUserMedia API

happened only at the mouse dragging
not happened while mouse cursor move
not happened on MacOS
not happened on WindowsOS
 
CursorCaptureIssue_UseDesktopCapture.webm
5.1 MB View Download
CursorCaptureIssue_UseScreencastify.webm
2.7 MB View Download

Comment 1 by junov@chromium.org, Apr 3 2018

Components: -Blink Blink>GetUserMedia>Desktop
Owner: braveyao@chromium.org
Status: Assigned (was: Unconfirmed)
Yes, I can reproduce this problem locally. Will take a look soon.
Cc: braveyao@chromium.org
Owner: m...@chromium.org
The problem is during dragging the window around, there is no MouseEvent [1] received.
[1]: https://cs.chromium.org/chromium/src/content/browser/media/capture/cursor_renderer_aura.cc?dr&g=0&l=119

Some logs as enclosed, quoted some here:
[30723:30723:0518/101805.566807:ERROR:cursor_renderer_aura.cc(120)] OnMouseEvent: event type: 4
[30723:30723:0518/101805.726686:ERROR:cursor_renderer_aura.cc(120)] OnMouseEvent: event type: 1
[30723:30723:0518/101816.146355:ERROR:cursor_renderer_aura.cc(120)] OnMouseEvent: event type: 10
[30723:30723:0518/101816.146536:ERROR:cursor_renderer_aura.cc(120)] OnMouseEvent: event type: 3
[30723:30723:0518/101817.528490:ERROR:cursor_renderer_aura.cc(120)] OnMouseEvent: event type: 4

In my test, I dragged the window around for 10s. As the logs above shows that within the 10s between event type 1(ET_MOUSE_PRESSED) and event type 3(ET_MOUSE_RELEASED), there is no other mouse event received(only a event type 10, "ET_MOUSE_CAPTURE_CHANGED", just before mouse release. What does this mean?).

miu@, is this WAI or a known issue? Anything we can do?
MouseEvent.txt
5.0 KB View Download

Comment 5 by m...@chromium.org, May 19 2018

Cc: m...@chromium.org
Components: Internals>Media>ScreenCapture
Labels: Hotlist-GoodFirstBug
Owner: ----
Status: Available (was: Assigned)
Sounds like this is something that none of the "cursor renderer" authors (myself included) have ever tested. :)

From the description and #c4 that on ChromeOS Ash desktop, it seems that mouse events are not being received by CursorRendererAura while the window is being dragged around. To solve this problem, we'd probably want to add a listener for window position changes (or maybe making the mouse event listener a "pre-listener" would be simpler?).

Is the event ET_MOUSE_DRAGGED (https://cs.chromium.org/chromium/src/ui/events/event_constants.h?dr&g=0&l=14) for such a scenario?
Who may know more about this? Ash team?
Cc: xiy...@chromium.org sadrul@chromium.org
+owners from ui/events and ash/shelf.

xiyuan@ and sadrul@, any insight?

Comment 8 by xiy...@chromium.org, May 30 2018

Sadrul is the expert here. The suspicion in #6 is probably correct. From code in [1], ET_MOUSE_DRAGGED is used instead of ET_MOUSE_MOVED when there is a mouse button down.

[1]: https://cs.chromium.org/chromium/src/ui/events/event.cc?rcl=cad86e84f28c289d62399cf7588d7fd46ad98343&l=575

Comment 9 by sadrul@chromium.org, May 30 2018

It should be DRAGGED, yes.

Although, when you are dragging a window, the CursorRendererAura (as a pre-target handler for the window) may still not see the DRAGGED events.
What does the pre-target handler (and pre-listener in #5) mean here? Why does it suppose not being able to get the DRAGGED events?

Comment 11 by m...@chromium.org, Jun 18 2018

Owner: m...@chromium.org
Status: Started (was: Available)
I'm looking into this, as I'm currently working on moving the mouse cursor rendering out of the browser process (into VIZ).
Blockedon: 810133
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77

commit 03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77
Author: Yuri Wiitala <miu@chromium.org>
Date: Wed Jul 25 21:45:44 2018

Add MouseCursorOverlayController.

This controller is heavily based on content::CursorRendererXYZ,
providing the logic for processing platform mouse events and using them
to control the new viz::VideoCaptureOverlay.

In a later CL, this will be integrated with the tab/desktop capture
stack, to complete the migration discussed in crbug 810133.

Bug:  810133 ,828274
Change-Id: I6502e1ef3cbc878e1c9b062b0abf6a0e6229f860
Reviewed-on: https://chromium-review.googlesource.com/1149264
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578079}
[modify] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/BUILD.gn
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller.cc
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller.h
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller_aura.cc
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller_browsertest.cc
[add] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/browser/media/capture/mouse_cursor_overlay_controller_mac.mm
[modify] https://crrev.com/03ff689101b3bcca2b7c2dd2fccf6c6ee8b8bc77/content/test/BUILD.gn

Labels: -Pri-2 Pri-3
Owner: ----
Status: Available (was: Started)
Just revisited using current CrOS stable: The new cursor rendering code will: 1) Hide the cursor (after dragging for a few seconds); and 2) When the window dragging stops, the cursor re-appears at the correct location.

Internally, the UI toolkit doesn't provide a convenient way to receive mouse events while dragging is taking place. There might be a way to install some kind of pre-target handler to get mouse events while dragging, but that would require some research.

Setting status to P3 and Available so someone else could pick this up. Unfortunately, I won't be able to continue working on it.

Sign in to add a comment