New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 694075 link

Starred by 4 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Feature



Sign in to add a comment

How to create a smoothness/jank measurement/metric that actually works

Reported by jer...@duckware.com, Feb 19 2017

Issue description

UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36

Steps to reproduce the problem:
There is the need for a measurement (for automated testing) that programmatically does the same thing that the "VSYNC" indicator on vsynctester.com does visually – detect jank.

Measuring how accurately frames are produced -- does not work.  Measuring inter-frames times -- do not work.  All measurements that are based upon the ‘production’ of frames -- do not work.  The one key takeaway from coding vsynctester.com is that how accurately a web browser produces frames -- does not matter -- AT ALL.  All that matters is: did the frame that was produced end up in the correct/proper/expected display frame?

So, we must look at and measure how (accurately) frames are consumed (displayed).

If we are trying to synchronize frames to the display Hz, there is jank if... (1) two produced frames end up in the same display frame, or (2) a display frame goes by without receiving a produced frame (if we are animating and expect every display frame to be filled).

Consider r272724 "Disable Vsync when Aero Glass is enabled", which would have worked except for one major detail.  The internal timebase/interval that Chrome was using at that time (timebase=0; interval=16.666ms) was wrong.  The end result was an interference pattern (jank) between the actual display Hz and the frequency Chrome internally used.  And no one noticed at the time, and no automated testing at the time caught this -- until Chrome 37 was released to the public, and the public noticed.

There should have been an automated test to catch this.

The work stanisc is doing in issue 467617 provides a key opportunity to actually implement a working smoothness/jank measurement/metric on Windows -- every time the 'wait for vsync' function call returns, increment a "nFrame" counter.  Then, after a produced frame is 'Present()'ed...

1) nFrame = (obtain from stanisc code)
2) if (nFrame==nLastFrame), then jank
3) if (nFrame>nLastFrame+1), then jank (if we are expecting frames at the display Hz)
4) nLastFrame = nFrame

A possible improvement: try to obtain the display frame number (from the OS) that a presented frame landed in -- if that exists and is reliable.

What is the expected behavior?

What went wrong?
-

Did this work before? N/A 

Chrome version: 56.0.2924.87  Channel: stable
OS Version: 6.1 (Windows 7, Windows Server 2008 R2)
Flash Version:

 
Labels: Needs-Triage-M56

Comment 2 by hdodda@chromium.org, Feb 20 2017

Cc: hdodda@chromium.org
Components: Blink>HTML>Frame
Status: Untriaged (was: Unconfirmed)
Considering this as feature request , marking it as untraiged for further inputs.

Thanks!

Comment 3 by hdodda@chromium.org, Feb 20 2017

Components: -Blink>HTML>Frame Blink>Scheduling

Comment 4 by hdodda@chromium.org, Feb 20 2017

Labels: -Needs-Triage-M56 M-58
Cc: tdres...@chromium.org panicker@chromium.org
Shubhie, I think you mentioned something about an API that would let sites measure the full end-to-end latency of content updates. I wonder if that would be useful for this too?
Status: Available (was: Untriaged)

Comment 7 by jer...@duckware.com, Feb 20 2017

Is there a way we could very quickly expose various jank metrics to a rAF callback (and in a way that allows for future additions)?

An obvious way is for all (jank) metrics to be properties of an object.  And a transparent way to quickly achieve this would be send that object as a second argument to the rAF callback.  Only a rAF callback that expects to get the new object will, and if run under a browser that does not support the object, the second argument will be 'undefined' (detectable).

At some point, there would need to be a decision as to what properties to include (or not) in the object, but a great first start would be to distill everything down into just *ONE* critical metric ('nNumPresentsInLastDisplayFrame'): the number of Chrome rendered frames that landed/presented in the last display/vsync frame.

For any rAF animation, 'nNumPresentsInLastDisplayFrame==1' means no visual jank (otherwise, there WAS visual jank).

My initial goal is to provide a metric to JavaScript that replicates programmatically what the vsync indicator on vsynctester.com achieves visually.

Given the number of times that true vsync was unintentionally broken in Chrome over the years -- I would think that the need for an automated test to detect vsync regressions is really needed.

Comment 8 by jer...@duckware.com, Feb 20 2017

skyos/5: "full end-to-end latency of content updates" -- Measuring the 'production' side of things does not work at all as a metric of 'jank'.  Latency can be nothing and there can still be jank.  Or, the latency of some frames can be *larger* than an entire frame, and there may be no jank.  I can show examples of both.

What has been missing for years is a metric that shows how many rendered frames landed in a single display frame.  For virtually all rAF animations, this should be one-to-one.  And if not, there is jank.


Cc: briander...@chromium.org
For most platforms, we don't know whether the GPU managed to display a frame at a given vsync or not.

Can you provide some additional information on what this signal would be used for?

"There is the need for a measurement (for automated testing) that programmatically does the same thing that the "VSYNC" indicator on vsynctester.com does visually – detect jank."

If you're doing automated testing, there's no need for a web exposed API: you could record traces and scrape the performance data out of the traces. Would that address your use case?

Comment 10 by woxxom@gmail.com, Feb 21 2017

Having such metric in a web exposed API would help sites implement graceful degradation of animation quality and so on.
Cc: skyos...@chromium.org
We can get away with a much more general API for the graceful degradation use case though. For example, the browser could expose to rAF its best guess of how much time you have to render before you'll miss a frame.

This API would be easier to use, and still make sense on platforms where we can't tell if we hit vsync or not.
tdres/9: So Google breaks vsync BADLY in Chrome 37, and no one notices until it is released to the public and the public immediately notices (issue 422000).  Google breaks ANGLE vsync for nearly two years, and no one notices ( issue 465356  and  issue 632785 ).  Android vsync just regresses and no one notices ( issue 694267 ).

What broke Chrome 37 was a *purposeful* turning off of ANGLE vsync, which would have worked, except that the timebase/interval that Chrome then used was also busted (wrong) -- but no one internally noticed.  The recorded traces (as looked at from the production side of things) of course looked great (so even if developers back then were looking at traces, they would have seen nothing obvious).

Go back and and run r279186 from the snapshots builds.  Run vsynctester.com, click on the gear icon, check "Even frame pre-delay" and set to 2ms.  tracing looks fine.  From the production side of things, everything looks fine and yet there is horrible jank.

When Chrome breaks vsync, even a rAF animation that uses *NO* time (<1ms), will jank.  How can you detect that jank?  What is proposed above will detect that jank.
This is definitely important, and we should fix it, but I'm not seeing why a web exposed API is needed for this.

The issues you mention, though legitimate, don't require a web performance API. If we correctly expose whether or not we hit vsync in traces, and have internal performance benchmarks looking at this data, that would be sufficient, correct?
tdres/13: Because knowing is a great diagnostic tool -- that I would use on vsynctester.com, and game developers would use.  And because there are other very useful metrics to add into the object.

Comment 15 by nduca@chromium.org, Feb 21 2017

Seems like we should split this bug into two issues: web-exposed and chrome-side. Then the chrome-side bug goes to scheduler team to contemplate, and the web-exposed side goes to panicker@ to backlog or take action on. Tim, can you file two fresh bugs that capture the spirit of this thread, and route them accordingly? You can then dupe or close this.
Cc: stanisc@chromium.org
This might be related to 614147. The plan I had in mind is to use DXGI frame statistics to know exactly when each frame was displayed. See:
https://bugs.chromium.org/p/chromium/issues/detail?id=614147#c6

Earlier I implemented ANGLE EGL_CHROMIUM_sync_control extension to return DXGI frame statistics from ANGLE.
https://chromium.googlesource.com/chromium/src/gpu/+/master/GLES2/extensions/CHROMIUM/EGL_CHROMIUM_get_sync_values.txt

Then I attempted to leverage EGL_CHROMIUM_sync_control on Windows to use it instead of VSyncProviderWin to produce vsync parameters similar to how it is done on Linux, but that resulted in a video playback smoothness regression and I had to revert it.

The current plan is bring EGL_CHROMIUM_sync_control extension back but use it only for diagnostic purposes by tracing frame # / display time pairs.
Components: Internals>GPU>Metrics
Components: Internals>Compositing
Labels: -Pri-2 Pri-3
Labels: -M-58

Sign in to add a comment