Adding the images (they seamed to stop me adding issue)

Deleted: input_event_requests_left_browse_real_app.png 311 KB

	input_event_requests_left_browse_real_app.png 311 KB View Download

Deleted: input_event_requests_right_browse_real_app.png 461 KB

	input_event_requests_right_browse_real_app.png 461 KB View Download

Also attaching full event dump from: chrome://net-internals for a test run.

Deleted: net-internals-log.zip 578 KB

net-internals-log.zip 578 KB Download

skyostil: Looks like this might be related to 548606. Should this be merged in?

My understanding is that defect 548606 was ported to the 2526 branch. 
https://codereview.chromium.org/1411133003

I was just looking at:
https://chromium.googlesource.com/chromium/src/+/6575ed2c8c23a1db4360cba48b548699fc186f5f/components/scheduler/renderer/renderer_scheduler_impl.cc

I find the  following comment interesting (line 344):

  // We regard MouseMove events with the left mouse button down as a signal
  // that the user is doing something requiring a smooth frame rate.

Special treatment of the left mouse button seams to slowing down down http requests.

Thanks Kris

Should all button downs be treated the same? I know that won't fix the original issue  the author is indicating but at least it would be consistent.

For my application: 
* all mouse buttons should be treated the same.
* The ajax requests should have consistent (fast) response times

The primary defect is that for some reason when the left mouse button is down the ajax requests take longer to complete. 

I searched for places where the left mouse button is checked and the render scheduler looks like the only place that stands out. Could the issue be thread contention "prioritized" rendering thread results in unresponsive http threads? Which kills my http heavy application?

I think my theory of rendering thread boost in priority is definitly causing the issue. I modified my checkout of renderer_scheduler_impl.cc to remove the left mouse button priority boost: so that:

----------
bool RendererSchedulerImpl::ShouldPrioritizeInputEvent(
    const blink::WebInputEvent& web_input_event) {
  // We regard MouseMove events with the left mouse button down as a signal
  // that the user is doing something requiring a smooth frame rate.
  /*if (web_input_event.type == blink::WebInputEvent::MouseMove &&
      (web_input_event.modifiers & blink::WebInputEvent::LeftButtonDown)) {
    return true;
  }*/
  // Ignore all other mouse events because they probably don't signal user
  // interaction needing a smooth framerate. NOTE isMouseEventType returns false
  // for mouse wheel events, hence we regard them as user input.
  // Ignore keyboard events because it doesn't really make sense to enter
  // compositor priority for them.
  if (blink::WebInputEvent::isMouseEventType(web_input_event.type) ||
      blink::WebInputEvent::isKeyboardEventType(web_input_event.type)) {
    return false;
  }
  return true;
}
----------
And this definitely fixes the issue for me.

I assume this code was added to improved other applications, I could imagine large JavaScript rendering applications would suffer from this change. Could it be possible to return true if there are not many http requests or boost the priority of process or something instead? This would make things more complex but might cover my case and others. 

Any thoughts?

Another question / worry is: When I tested using chrome I found it was considerably slower than CEF. See image attached (screen shot from xlsx file in other zip file).

These graphs show all time take for all control messages sorted largest to smallest. Different colors are for the different buttons.

They show that Chrome's best request times are about 4ms and CEF best times are about 1ms. Should I raise a separate bug for this? It doesn't impact my app because I use CEF to Chrome, but not sure if this is intentional.

Deleted: input_event_request_distrubution_per_button.png 25.7 KB

	input_event_request_distrubution_per_button.png 25.7 KB View Download

Thanks for the report. To help us diagnose, would you mind recording a trace of the sequence? To do that:

1. Open a tab with the attached "index.html".
2. Open another tab at "chrome://tracing".
3. Click "Record".
4. Choose "Manually selected settings".
5. Click "All" above the left side pane and enable "renderer.scheduler" and "renderer.scheduler.debug" in the right side list.
6. Click record, switch back to the other tab, perform some mouse gestures, then switch back and click "Stop".
7. After the trace file loads, click "Save" and attach the result here.

(Note: the trace will contain information about all open tabs in your browser.)

My guess is that mouse dragging is taking the browser into a main thread driven gesture policy, which means rendering tasks will be given priority to increase smoothness. Generally this should not be an issue if the rendering tasks are cheap, but let's see what the trace tells us.

Here are those traces, I did it for left drag and right drag

Deleted: trace_left_drag.json.gz 6.0 MB

trace_left_drag.json.gz 6.0 MB Download

Deleted: trace_right_drag.json.gz 7.8 MB

trace_right_drag.json.gz 7.8 MB Download

A new version of index.html that analyses the AJAX requests as you interact and displays the results on the page, without needing to review the network data directly.

Deleted: index.html 7.6 KB

index.html 7.6 KB View Download

Was the trace I uploaded useful? Is there anything else I can provide to help with this defect?

Thanks

Kris

Yes that trace was useful and it does explain what you're seeing.  Your AJAX requests are getting throttled by the ResourceDispatchThrottler.  I don't think it's working well in this case and needs tuning.

What seems to be going wrong is the page is generating XHRs at ~120Hz but the  ResourceDispatchThrottler is getting sporadically invoked and it buffers up requests and flushes to flush them them at 60Hz.  

In practice this means some XHRs are getting delayed by 16ms causing the problems noted in bug report.

This is interesting, originally I suspected something like that was going on and I experimented with making my application queue input event requests and send them in window.requestAnimationFrame, but this didn't fix the problem. Possibly because in my real app their are additional requests being sent to the server (requests for cursors, images etc). Manually throttling my input events also made the application slower in general :(

I'm less worried about a page that does XHR at 120hz, than I am that we possibly don't throttle correctly. If a page is doing XHR a 1hz, will it ever get delayed/throttled? What I mean to say is, if we just make a single request, does the throttler throttle that or fire it off immediately when the mouse is down? If it's the latter, I'm not sure there's a bug we need to fix here.

Hi Ojan,

There was an issue with 1Hz requests being throttled: https://bugs.chromium.org/p/chromium/issues/detail?id=548606#  

My colleague raised that one, it has been fixed - Thank you :)

What is the justification that throttling XHR above 60Hz is acceptable? I am writing a high performance web application, in this case running in loopback.

If my application is handling input events at 60Hz and requesting frames from the server at 60hz and other resources also at 60hz then the total frequency of XHR's could be ~180Hz to achieve a frame rate of 60Hz. 

At the very least this throttling should be highlighted in the debugger.

Also my observations are that the throttling is much more complex than this. In my real app when the server responds faster (cheaper rendering task) then even with the left mouse button the frame rate increases which makes me think it is a function of the request time. Is it?

I find it strange that it only throttles on left drag.

Thoughts?

Thanks

Kris

> If a page is doing XHR a 1hz, will it ever get delayed/throttled?

The ResourceDispatchThrottler will not throttle 1hz requests. It will only throttle if it sees a certain number of requests within a given time window. It was actually designed to accommodate requests up to 180hz, but that may only work if the requests themselves are bursty. I think we can tweak the logic to make it handle ~120hz requests that arrive at continuous, distinct intervals.

Tentative fix up for review at crrev.com/1834853003.

I look forward to trying it. thank you :)

Kris, HTTP requests are very heavyweight. Doing HTTP requests at 60hz is a design that will chew through the user's battery no matter how you slice it. Therefore it's always a bad thing to do and I don't think we should try to support it. The better design for such a task is to use WebSockets. Is there a reason your app can't use WebSockets? If so, we'd love to know so we can fix it. :)

For what it's worth, the network requests are getting delayed anyways because the network stack will only start a certain number (6?, 12?) of pending requests to a given domain at a time.

That said, I'm not really sure we should have a ResourceDispatchThrottler at all. That seems like the wrong layer to do the throttling. The IO thread should  be smarter about prioritizing input vs. network requests. Flooding it with network requests shouldn't cause input to be blocked.

I'm not commenting on jdduke's patch itself. I don't know the code in question well and leave it to the rest of you to decide whether the new code is architecturally better than the old code. 

Ojan, Thanks for the feedback. Our system is actually very complicated. We started with long poll before websockets were well supported and the architecture has been working well for us. I did some performance analysis on windows 7 (loopback) a couple of years ago comparing our XHR, websockets and Tcp and found that websockets wasn't much better XHR. TCP was a lot faster and more consistent. We use WinHttp server API's in our C++ server and Chrome have very good performance (better on windows 8 & 10). And it wasn't worth the investment in moving to Websockets. Our architecture shouldn't make more than 5 web requests in parallel to work with the limitation in Chrome and other browsers. 

Now that websockets are better supported we should re-test with latest tools like: https://code.msdn.microsoft.com/windowsdesktop/WinHTTP-WebSocket-sample-50a140b5.  

Deleted: 20140714_tcp_xhr_websocket_results.png 398 KB

	20140714_tcp_xhr_websocket_results.png 398 KB View Download

Thanks for the patch Jared (and hi! :). I checked that fixing the sliding window issue makes the test case from this bug behave correctly.

Alex and I discussed about the architectural side of things and it seems like having a scheduler on the IO thread would make things a little easier to reason about and might also let use forego the need to throttle in the first place. The only sticking point is that I'm not entirely sure what happens to the IO thread in mojo land -- it might be better to focus our efforts into making mojo schedulable instead.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/46c01a93646af14072ce25fc59a9b59b29f097d7

commit 46c01a93646af14072ce25fc59a9b59b29f097d7
Author: jdduke <jdduke@chromium.org>
Date: Tue Mar 29 16:33:37 2016

Allow continuous request dispatch below the throttle rate

ResourceDispatchThrottler will throttle requests when a give threshold
is reached within a given time window. However, it is overly restrictive
when the requests are continuous, as the sliding time window was keyed
off the most recently sent request.

Fix this by tracking the flush time, rather than the last sent request
time, allowing continuous request dispatch at or below the specified
throttling rate.

BUG= 592037 

Review URL: https://codereview.chromium.org/1834853003

Cr-Commit-Position: refs/heads/master@{#383740}

[modify] https://crrev.com/46c01a93646af14072ce25fc59a9b59b29f097d7/content/renderer/scheduler/resource_dispatch_throttler.cc
[modify] https://crrev.com/46c01a93646af14072ce25fc59a9b59b29f097d7/content/renderer/scheduler/resource_dispatch_throttler.h
[modify] https://crrev.com/46c01a93646af14072ce25fc59a9b59b29f097d7/content/renderer/scheduler/resource_dispatch_throttler_unittest.cc

> Alex and I discussed about the architectural side of things and it seems like having a scheduler on the IO thread would make things a little easier to reason about and might also let use forego the need to throttle in the first place. The only sticking point is that I'm not entirely sure what happens to the IO thread in mojo land -- it might be better to focus our efforts into making mojo schedulable instead.

Absolutely. The throttler was never intended as a long term solution. Rather, as a simple drop-in fix that could be easily removed and would address the most immediate symptoms of IO thread flooding from resource requests.