This would compliment or overlap work being done to get input batching to match beginframes in the renderer.

danakj chongz tdresser Can you explain more about this work?

Here's the bug: https://bugs.chromium.org/p/chromium/issues/detail?id=625689
And the parent bug: https://bugs.chromium.org/p/chromium/issues/detail?id=394562
Part of the things we can fix in the compositor from it: https://bugs.chromium.org/p/chromium/issues/detail?id=551134
A WIP doc about understanding why we need it for compositor: https://docs.google.com/document/d/1icPlOwcfuudo9jGVERjuihEg_H0R7OW5fu9r-GDED60/edit

Note that we're currently only aligning scroll and pinch to the BeginFrame signal, since smoothness matters a lot for these gestures. We're also only doing this in certain cases (when these events are being handled on the compositor).

We don't want to align gestures such as tap, since it would introduce latency without buying us anything (as far as I can tell).

Sadrul offered to picked this up next week. I think this will likely be an overall performance win. The naive idea is as follows:

Common case:

1. Input events arrive in the window server. No work happens immediately. Instead, the window server asks the display's FrameGenerator for a BeginFrame and queues up the event.
2. Until the BeginFrame arrives, pointer moves coalesce.
3. When the BeginFrame arrives, we send out all queued up events.

Synchronization case:

1. When a client submits a pending CompositorFrame (the CompositorFrame has blockers), the display compositor signals the window server.
2. The window server queues up events for the blocked client and coalesces events.
3. Once the client is unblocked, the window server either immediately sends out queued events, or requests another BeginFrame. Maybe we still do the common case (receive BeginFrames if there are any queued up events) but we don't send the events out UNTIL the client is unblocked.

This ensures that during a resize, we don't keep sending out events that create more resizing until a deadline hits. This reduces the amount of work the clients need to do, and increases the likelihood of synchronization within a fixed deadline.

We called touchmove, mousemoves "continuous" events and anything else "discrete" events when we did something similar in the render thread.

The problem I think with delaying the messages is that you don't know when the rAF signal is going to run. The main thread could be busy and then you could be one frame out of events that occurred that could have gotten delivered to the main thread.

Is the concern that rAF might run before the events are delivered (which rAF may want to depend on?)and so you end up  with an extra frame of latency for an input-driven animation?

+gklassen@ because we briefly chatted about this yesterday.

1) Main thread busy
2) BeginFrame starts
3) rAF signal enqueued
4) Mouse/Touch Events recevied
5) rAF signal runs
 a) Input event is delivered (stale info) <--- This is where we are going to do prediction.
 b) Javascript rAF signal runs

I think we should start with holding only the pointer-move events (i.e. do the same as the renderer). If some other event happens (e.g. pointer-down), then we release all the events immediately.

What's the goal here? Throttling only should have an impact right now on scrolling in the compositor thread, which is being done in the renderer. Otherwise events are all essentially queued until the compositor commits.

#14 I believe UI immediately processes input so this is an attempt to generalize throttling across Blink and UI. Is that right, Sadrul?

Depends what you mean by process. It can set values all it wants but they are all collected and acted on in the commit. The compositor lock prevents commit from occuring while we're waiting on a renderer frame during resize.

What are you specifically expecting to not run that would otherwise from this? Do you have a stack trace or something? I think that'd help understand what the goal is and if we've reached it or not.

#16: Good question Dana! Here's the stack trace:

4800 #1 0x7fe97ec199bf aura::WindowTreeClient::ScheduleInFlightBoundsChange()
4801 #2 0x7fe97ec1a493 aura::WindowTreeClient::OnWindowMusBoundsChanged()
4802 #3 0x7fe97ec162e3 aura::WindowPortMus::OnDidChangeBounds()
4803 #4 0x7fe97ec2859e aura::Window::OnLayerBoundsChanged()
4804 #5 0x7fe97ebb1e67 ui::Layer::SetBoundsFromAnimation()
4805 #6 0x7fe97ec259e3 aura::Window::SetBoundsInternal()
4806 #7 0x7fe97dcf7f57 content::BrowserMainRunnerImpl::Run()
4807 #59 0x7fc00884a908 ash::WmWindow::SetBoundsDirect()
4808 #8 0x7fe97dced028 ash::wm::WindowState::SetBoundsDirect()
4809 #9 0x7fe97dcf30a2 ash::WorkspaceLayoutManager::SetChildBounds()
4810 #10 0x7fe97dcf4230 ash::WorkspaceWindowResizer::Drag()
4811 #11 0x5613efe31994 content::BrowserMain()
4812 #60 0x7fc008e31bf8 ash::mus::DragWindowResizer::Drag()
4813 #12 0x7fe97dcee0e2 content::ContentMainRunnerImpl::Run()
4814 #61 0x7fc00113bc3f service_manager::Main()
4815 ash::wm::WmToplevelWindowEventHandler::OnMouseEvent()
4816 #13 0x7fe97ed2405c ui::EventDispatcher::DispatchEventToEventHandlers()
4817 #14 0x7fe97ed23c9b ui::EventDispatcher::ProcessEvent()
4818 #15 0x7fe97ed23aca ui::EventDispatcherDelegate::DispatchEvent()
4819 #16 0x7fe97ed247ce ui::EventProcessor::OnEventFromSource()
4820 #17 0x7fe97ed24c56 ui::EventSource::SendEventToSink()
4821 #18 0x7fe97ec1cd66 aura::WindowTreeClient::OnWindowInputEvent()

As you can see, we are allocating new LocalSurfaceIds to pass to the child every input event during a resize...

Actually, I guess you can't see that we're allocating new LocalSurfaceIds but we do that inside ScheduleInFlightBoundsChange:

https://cs.chromium.org/chromium/src/ui/aura/mus/window_tree_client.cc?q=scheduleinflightbounds+package:%5Echromium$&l=662

The ids wouldn't be sent to the renderer yet would they? Cuz resize of the renderer is throttled by the resize lock (DelegatedFrameHost::GetRequestedRendererSize() is called from https://cs.chromium.org/chromium/src/content/browser/renderer_host/render_widget_host_impl.cc?rcl=fdad51f24bced15b7cab495106b3e2152ced8dbd&l=701 so it won't change while the lock is held)

The IDs are sent to the renderer now.https://cs.chromium.org/chromium/src/content/renderer/mus/renderer_window_tree_client.cc?q=RendererWindowTreeClient+package:%5Echromium$&l=178

Note that they're not sent from the browser yet until we implement surface hit testing in viz :(

There is a race there, but I think that can be addressed by deferring commits on resize UNTIL a new LocalSurfaceId arrives.

Are we talking about having multiple vsync buffers for a given input here (one in MUSE and again later in the renderer)?

For the best latency/smoothness tradeoff I believe the buffering/alignment must be done exactly once and as late as possible.  This is why, for example, when using our own renderer-side vsync alignment we will disable the one in the Android framework.  The same model should apply for ChromeOS.

The intent would be to do it in one place. So I've been able to get away with a prototype that throttles input in UI giving us smooth resize so this approach isn't necessary, for now.

However, from the perspective of Mus being a foundational system service, and for "not making ui a special snowflake" it seems like the right long term approach is to coalesce input in a central place in the system that is not specific to Blink or UI.

Rick, I'm curious why we feel input throttling/coalescing/v-sync align should happen as late as possible? Is it loss of samples for prediction or something? Can we bundle the stream of events along with a single IPC instead, once per BeginFrame instead perhaps and leave it up to the app (Blink in this case) to interpret it?

Thanks!

The problem is you don't know when the rAF signal is going to hit the main thread. When you pick the data up on the main thread then you can do prediction because you know the current time and the history of events.

We should probably chat about this in person sometime.

As Rick said, buffering/alignment must be done exactly once and as late as possible. This is because different consumers need to sync to different points in time. Input on the main thread needs to be synchronized to rAF, input on the compositor needs to be synchronized to vsync, etc.

There isn't a one size fits all solution.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

We don't have anything planned here, so I'm closing this out.