How to create a smoothness/jank measurement/metric that actually works
Reported by
jer...@duckware.com,
Feb 19 2017
|
||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Steps to reproduce the problem: There is the need for a measurement (for automated testing) that programmatically does the same thing that the "VSYNC" indicator on vsynctester.com does visually – detect jank. Measuring how accurately frames are produced -- does not work. Measuring inter-frames times -- do not work. All measurements that are based upon the ‘production’ of frames -- do not work. The one key takeaway from coding vsynctester.com is that how accurately a web browser produces frames -- does not matter -- AT ALL. All that matters is: did the frame that was produced end up in the correct/proper/expected display frame? So, we must look at and measure how (accurately) frames are consumed (displayed). If we are trying to synchronize frames to the display Hz, there is jank if... (1) two produced frames end up in the same display frame, or (2) a display frame goes by without receiving a produced frame (if we are animating and expect every display frame to be filled). Consider r272724 "Disable Vsync when Aero Glass is enabled", which would have worked except for one major detail. The internal timebase/interval that Chrome was using at that time (timebase=0; interval=16.666ms) was wrong. The end result was an interference pattern (jank) between the actual display Hz and the frequency Chrome internally used. And no one noticed at the time, and no automated testing at the time caught this -- until Chrome 37 was released to the public, and the public noticed. There should have been an automated test to catch this. The work stanisc is doing in issue 467617 provides a key opportunity to actually implement a working smoothness/jank measurement/metric on Windows -- every time the 'wait for vsync' function call returns, increment a "nFrame" counter. Then, after a produced frame is 'Present()'ed... 1) nFrame = (obtain from stanisc code) 2) if (nFrame==nLastFrame), then jank 3) if (nFrame>nLastFrame+1), then jank (if we are expecting frames at the display Hz) 4) nLastFrame = nFrame A possible improvement: try to obtain the display frame number (from the OS) that a presented frame landed in -- if that exists and is reliable. What is the expected behavior? What went wrong? - Did this work before? N/A Chrome version: 56.0.2924.87 Channel: stable OS Version: 6.1 (Windows 7, Windows Server 2008 R2) Flash Version:
,
Feb 20 2017
Considering this as feature request , marking it as untraiged for further inputs. Thanks!
,
Feb 20 2017
,
Feb 20 2017
,
Feb 20 2017
Shubhie, I think you mentioned something about an API that would let sites measure the full end-to-end latency of content updates. I wonder if that would be useful for this too?
,
Feb 20 2017
,
Feb 20 2017
Is there a way we could very quickly expose various jank metrics to a rAF callback (and in a way that allows for future additions)?
An obvious way is for all (jank) metrics to be properties of an object. And a transparent way to quickly achieve this would be send that object as a second argument to the rAF callback. Only a rAF callback that expects to get the new object will, and if run under a browser that does not support the object, the second argument will be 'undefined' (detectable).
At some point, there would need to be a decision as to what properties to include (or not) in the object, but a great first start would be to distill everything down into just *ONE* critical metric ('nNumPresentsInLastDisplayFrame'): the number of Chrome rendered frames that landed/presented in the last display/vsync frame.
For any rAF animation, 'nNumPresentsInLastDisplayFrame==1' means no visual jank (otherwise, there WAS visual jank).
My initial goal is to provide a metric to JavaScript that replicates programmatically what the vsync indicator on vsynctester.com achieves visually.
Given the number of times that true vsync was unintentionally broken in Chrome over the years -- I would think that the need for an automated test to detect vsync regressions is really needed.
,
Feb 20 2017
skyos/5: "full end-to-end latency of content updates" -- Measuring the 'production' side of things does not work at all as a metric of 'jank'. Latency can be nothing and there can still be jank. Or, the latency of some frames can be *larger* than an entire frame, and there may be no jank. I can show examples of both. What has been missing for years is a metric that shows how many rendered frames landed in a single display frame. For virtually all rAF animations, this should be one-to-one. And if not, there is jank.
,
Feb 21 2017
For most platforms, we don't know whether the GPU managed to display a frame at a given vsync or not. Can you provide some additional information on what this signal would be used for? "There is the need for a measurement (for automated testing) that programmatically does the same thing that the "VSYNC" indicator on vsynctester.com does visually – detect jank." If you're doing automated testing, there's no need for a web exposed API: you could record traces and scrape the performance data out of the traces. Would that address your use case?
,
Feb 21 2017
Having such metric in a web exposed API would help sites implement graceful degradation of animation quality and so on.
,
Feb 21 2017
We can get away with a much more general API for the graceful degradation use case though. For example, the browser could expose to rAF its best guess of how much time you have to render before you'll miss a frame. This API would be easier to use, and still make sense on platforms where we can't tell if we hit vsync or not.
,
Feb 21 2017
tdres/9: So Google breaks vsync BADLY in Chrome 37, and no one notices until it is released to the public and the public immediately notices (issue 422000). Google breaks ANGLE vsync for nearly two years, and no one notices ( issue 465356 and issue 632785 ). Android vsync just regresses and no one notices ( issue 694267 ). What broke Chrome 37 was a *purposeful* turning off of ANGLE vsync, which would have worked, except that the timebase/interval that Chrome then used was also busted (wrong) -- but no one internally noticed. The recorded traces (as looked at from the production side of things) of course looked great (so even if developers back then were looking at traces, they would have seen nothing obvious). Go back and and run r279186 from the snapshots builds. Run vsynctester.com, click on the gear icon, check "Even frame pre-delay" and set to 2ms. tracing looks fine. From the production side of things, everything looks fine and yet there is horrible jank. When Chrome breaks vsync, even a rAF animation that uses *NO* time (<1ms), will jank. How can you detect that jank? What is proposed above will detect that jank.
,
Feb 21 2017
This is definitely important, and we should fix it, but I'm not seeing why a web exposed API is needed for this. The issues you mention, though legitimate, don't require a web performance API. If we correctly expose whether or not we hit vsync in traces, and have internal performance benchmarks looking at this data, that would be sufficient, correct?
,
Feb 21 2017
tdres/13: Because knowing is a great diagnostic tool -- that I would use on vsynctester.com, and game developers would use. And because there are other very useful metrics to add into the object.
,
Feb 21 2017
Seems like we should split this bug into two issues: web-exposed and chrome-side. Then the chrome-side bug goes to scheduler team to contemplate, and the web-exposed side goes to panicker@ to backlog or take action on. Tim, can you file two fresh bugs that capture the spirit of this thread, and route them accordingly? You can then dupe or close this.
,
Feb 21 2017
This might be related to 614147. The plan I had in mind is to use DXGI frame statistics to know exactly when each frame was displayed. See: https://bugs.chromium.org/p/chromium/issues/detail?id=614147#c6 Earlier I implemented ANGLE EGL_CHROMIUM_sync_control extension to return DXGI frame statistics from ANGLE. https://chromium.googlesource.com/chromium/src/gpu/+/master/GLES2/extensions/CHROMIUM/EGL_CHROMIUM_get_sync_values.txt Then I attempted to leverage EGL_CHROMIUM_sync_control on Windows to use it instead of VSyncProviderWin to produce vsync parameters similar to how it is done on Linux, but that resulted in a video playback smoothness regression and I had to revert it. The current plan is bring EGL_CHROMIUM_sync_control extension back but use it only for diagnostic purposes by tracing frame # / display time pairs.
,
Feb 5 2018
,
Aug 16
,
Aug 16
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by ranjitkan@chromium.org
, Feb 20 2017