+cc some relevant folks, skyostil for scheduling, boliu for android/compositor.
Was going to send out an email, but hijacking this bug instead.

I think the behaviour we want for rIC is to never fire it, unless the user has specified a timeout, in which case we fire it when the timeout is reached.
I think the behaviour we want for rAF is to continue firing it as usual, but ignore any commited frames, etc.

We've talked a bit about two possible approaches to solving this, so I wanted to summarize here and ask y'all for some thoughts.

Approach 1: Revert the change that pauses VSync while presenting, introduce a new mode into the compositor that basically says: Don't do any compositing work, don't update surfaces, compute visibility, etc., just fire rAFs as normal, and rICs only when they time out.

Approach 2: Keep the change that pauses VSync while presenting, add new interfaces (or use existing ones) so that VR can trigger rAF/rIC when it chooses to while VR is presenting.

I'm in favor of Approach 2, because it avoids complicating the compositor more and making it aware of things like VR presentation.

Do others agree? Are there other approaches I haven't listed here?

Ok I totally don't understand what VR is trying to do here.

Does the context in  issue 704341  help?

Basically, when WebVR presents, it essentially fullscreens a view over the rest of the browser so the page isn't visible. Rendering the 3D scene inside this fullscreen view bypasses the compositor, so for performance we disable the compositor while presenting ( issue 698923 ).

The problem with disabling the compositor is that requestAnimationFrame and requestIdleCallback callbacks stop firing, and various javascript libraries rely on using these callbacks for non-rendering tasks, so the break when these callbacks stop firing.

So we need some way to get these callback to fire, without re-enabling all of the compositor. Ideas are very much welcome.

So is VR like lifting the webgl context into its own surface? I guess that's similar to video fullscreen? I don't really understand why these callbacks are throttled at all?

Also is the page aware of that action? I assume yes, in which case browser should not be throttling these callbacks, but that page should behave "well" when it goes into VR mode.

If the issue is during VR mode, the page is still reciving rAF from chrome's display compositor, rather than whatever the VR thing is. Then that seem to be the bug. VR probably should behave like its own display compositor, taking over all responsibilities from chrome's one.

We're not intentionally throttling these callbacks, we've turned off VSync while presenting, so the callbacks simply never fire. We agree that the callbacks shouldn't be throttled.

what does turning vsync off mean here exactly? does that imply chrome is not producing new frames?

I mean the code in this CL you reviewed: https://codereview.chromium.org/2754493002/

Yes, it does imply Chrome is not producing new frames.

I thought the implication for that CL is the "VR window" will be hooked up.

> Yes, it does imply Chrome is not producing new frames.

But I assume the webvr/webgl canvas is producing new frames?

Then I think this is a web api issue to be resolved first. If only the canvas is producing frames, then maybe stopping the rAF to the whole page is the right thing. But that canvas needs to be ticked in some other way independently.

Or the contract is rAF should still happen on the main page, in which case I guess that CL should be reverted.

> But I assume the webvr/webgl canvas is producing new frames?

Yes. We tick the webVR canvas independently, at the VR headset's display rate, which may not be the same as the primary display rate, or the non-VR rAF rate.

>Then I think this is a web api issue to be resolved first. If only the canvas is producing frames, then maybe stopping the rAF to the whole page is the right thing. But that canvas needs to be ticked in some other way independently.

skyostil@ mentioned the same thing. We'd like to stop rAF to the whole page, but apparently we've been telling web developers to use rAF for non-display-related scheduling, and many popular libraries rely on it even when doing tasks not related to rendering. rIC is also inexplicably tied to VSync currently, and one of my proposed solutions was to allow VR code to trigger it.

> Or the contract is rAF should still happen on the main page, in which case I guess that CL should be reverted.

Hence the discussion here. We can't simply revert my CL because that had huge performance wins for WebVR. So we either need VR code to trigger rAF/rIC explicitly, or we need to modify the compositor to get the same performance wins as disabling it, but still have it fire rAF/rIC.

> We can't simply revert my CL because that had huge performance wins for WebVR.

Sure you can. You just have to tell regular chrome rendering pipeline to stop producing frames when in VR mode. I'm not sure what form that will take exactly, but if you still want rAFs, then stopping vsync is probably not the right thing.

But I'm still not clear if you still want rAFs, ie I don't think the web platform discussion is resolved. Maybe VR is a new thing and those webpages should just fix themselves?

+cc rbyers for thoughts on web platform issues here.

VR is a new thing, so I would be super happy if we could just tell pages to fix themselves, rather than hacking in support for rAF/rIC with a neutered chrome rendering pipeline.

Also happy to set up a meeting if the discussion here is getting too hard to follow.

In a way, we want to express a state that a page is active but not currently visible, so rAF/rIC should continue running but drawing and related steps should be skipped.

One specific issue for WebVR is that the page contains the WebGL canvas used for VR content, and since this changes every frame it triggers a comparatively expensive update through the normal compositing path. We want to stop it from doing that since it's unnecessary while in VR, the canvas content gets displayed on the headset directly.

See also  http://crbug.com/704341#c13  , I had originally tried to use view()->hide() and view->show() to change visibility which mostly worked but violated an assert elsewhere in Chrome. Also, in brief testing I had issues with rIC not running consistently in that mode.

Is there any precedent for an active-but-not-visible state in Chrome? A background tab that's playing audio seems similar, but that wouldn't necessarily require special handling since it would typically not be doing expensive drawing operations. Also, I'm unsure if rAF/rIC are active in this scenario or if only media-related events continue running.

> One specific issue for WebVR is that the page contains the WebGL canvas used for VR content, and since this changes every frame it triggers a comparatively expensive update through the normal compositing path.

That sounds wrong, or at least odd. I assume that canvas should just no longer be drawn by the compositor in vr mode, in which case why is it causing work for the compositor in the first place?

Fwiw, I was thinking just allow renderer compositor to request begin frames, but just don't submit new frames (and also stop other related work like record and raster). Then this would stop the whole compositor pipeline and still have vsync working. That's just my thinking though. You definitely want to run this by actual compositor folks.

I don't know any precedence.

+bajones, is there any hope of solving these web platform issues with the 2.0 webVR spec?

I think a meeting would be a good idea.

At a high level, it seems like there's no clear right answer here because the semantics of rAF and rIC have been abused and these abuses would need to be supported if we wanted to ensure web compat.

rAF
===

One issue: rAF assumes there is a vsync. This is already kinda wonky with multi monitor, but throw multiple HMD's, all with their own framerates, into the mix. The meaning of rAF becomes very muddy at this point, basically a signal that tries to fire at ~60Hz and stops firing when the page is hidden ...that, sadly, devs have glommed a lot of work non-visual work onto.

Another issue: the page really isn't visible anymore. Given the intended semantics of rAF, it makes sense to turn off rAF (because it doesn't fire when the page is hidden), but you can't, both because of the glomming I mentioned above and because devs sometimes do their WebVR visual work in rAF (though, I believe, this isn't encouraged).

rIC
===

This one is really sad. It's supposed to fire the callback when there's "idle" time. I.e., a spot where the main thread should be uninterrupted for ~50ms. But, as you can probably guess, this never happens in a WebVR app (especially when the display wants updates at 90Hz). So I'd love to ignore rIC, but sadly rIC has a form where you supply a timeout which guarantees the callback will eventually run. There is no good time to execute this work. We could do it right when rIC is requested, or at the timeout. All options are pretty crappy.

Some points in the solution space
=================================
1. Continue to keep the normal compositing path running, but gut it. I.e., lie about it being hidden, but trick it into doing no work.
2. Shut down the "page" vsync (since the page is hidden), and let VR handle things (since the VR content and display are active). This would mean that we could fire rAF at the appropriate times for the HMD, but there are options for what to do with rIC:
  a. Fire rIC callback immediately.
  b. Fire rIC callback at the timeout.
  c. Fire the rIC callback at a random time before the timeout. This has the benefit of preventing devs from assuming and coding for the implementation details represented by a or b (and might discourage the use of rIC in WebVR/WebGL, which would be great).
3. Don't fire rAF or rIC in our case.

My preference is 2c. I'd like to talk with Rick and people who understand the specs better about 3, though.

There's also a good chance I'm missing issues or have details wrong here. I think we should chat though the problems and potential solutions over VC.

I agree that a meeting would be helpful. 

> ... and because devs sometimes do their WebVR visual work in rAF (though, I believe, this isn't encouraged).

Not just discouraged, we actively try to prevent it. In WebVR, calls to vrDisplay.submitFrame must made from inside the VR-specific vrDisplay.rAF callback's context, they are rejected if made from window.rAF or other contexts. That's specifically to keep applications from using the wrong rAF for animations, especially since that might work ok on a mobile screen where both are at 60Hz but then break horribly on a 90Hz desktop HMD.

The core reason to support window.rAF and rIC is to make life easier for developers using third-party libraries or other pre-existing code that assumes that these work when they are present. One example from the previous bug was a resource loading library that split work into chunks to avoid blocking rendering. It would need some rather nasty hacks by developers to work around this if they can't or don't want to modify the code using it, especially since the replacement vrDisplay.rAF needs a live reference to the VRDisplay object being used.

#15: rIC doesn't require 50ms idle blocks in order to work. It can fit into whatever available time there is, i.e., the delay to the next vsync. We routinely give out <10ms idle periods when there is active main thread rendering going on and that has no impact on the frame rate.

We could probably make rIC work if we fed it timing data about the VR display rather than the real display. I'm less sure about rAF because no actual animation is happening, and other APIs such as CSS animations will be similarly broken unless we add a similar hack for them.

 Issue 716647  tracks adding a regression test for this.

The desired behavior for rIC seems to be unclear. Until that is resolved, WebVR applications should not depend on it. This shouldn't be an issue since not everyone implements it (https://github.com/w3c/webvr/issues/225#issuecomment-298778492).

Removing Blink>WebVR component and assigning to Blink>WebXR