We should let headless users specify when and how often animation and rendering happens.
Strawman:
Emulation.enableBeginFrameControl()
Emulation.onSetNeedsBeginFrame(bool beginFrameNeeded)
Emulation.sendBeginFrame(frameTime, frameInterval, deadline)
To those following this feature request: We've now landed a first _experimental_ version of the rendering control (aka BeginFrameControl) - for headless on Linux/Windows only -, under the HeadlessExperimental domain [1]. If you'd like to give it a try, here some first starting points. Feel free to send feedback.
BeginFrames are chromium's internal vsync signal and drive the rendering pipeline. In essence, they control when rendering happens. The new BeginFrameControl (BFC) allows you to replace chromium's default vsync signal and issue BeginFrames manually via DevTools instead.
You can enable BFC when creating a new HeadlessWebContents or DevTools target page (Target.createTarget now accepts an enableBeginFrameControl parameter). Afterwards, you can enable the HeadlessExperimental domain on the new target (HeadlessExperimental.enable) and wait for the HeadlessExperimental.needsBeginFramesChanged event.
This needsBeginFrames signal tells you whether chromium's compositor currently has pending updates that it would like to render. While needsBeginFrames is true, you can send BeginFrames via HeadlessExperimental.beginFrame.
After the target's main frame has committed its first display frame, you will receive the HeadlessExperimental.mainFrameReadyForScreenshots event, which signals you that you can now use HeadlessExperimental.beginFrame to capture a screenshot. For that purpose, you need to set the command's screenshot parameter. You can send such screenshotting BeginFrames even if needsBeginFrames is false.
Note that a BeginFrame may or may not be answered with a display update. For example, sometimes no display update is necessary afterall, or one of chromium's compositor stages may not be ready to send its updates yet. As a consequence, you might need to send multiple BeginFrames while the target's renderer process is initializing until you receive the mainFrameReadyForScreenshots event.
Furthermore, it is likely that you want to use the --run-all-compositor-stages-before-draw command line flag with BFC. This flag ensures that the compositor does not prematurely commit display frames that are missing some compositing updates. At the moment, this mode has some caveats: It doesn't work well during renderer initialization or window resizes yet. We are working on fixing these issues.
We currently only have a browser test to show all this in action [2]. It may be a useful starting point nevertheless.
Please remember that this functionality is still considered experimental and will likely change in the near future. For example, we intend to make the --run-all-compositor-stages-before-draw mode more dynamic - so that you can specify whether to run all stages for individual BeginFrames. We're also working on supporting BFC in desktop (non-headless) mode, which will result in changes to the commands.
[1] https://chromedevtools.github.io/devtools-protocol/tot/HeadlessExperimental/
[2] https://cs.chromium.org/chromium/src/headless/lib/headless_web_contents_browsertest.cc?type=cs&q=HeadlessWebContentsBeginFrameControlTest&sq=package:chromium&l=1038
Hi!
I'm trying to use HeadlessExperimental API to achieve rendering control. My goal is to capture screenshots in a fast way using experimental API.
But, after creating Target with "enableBeginFrameControl: true" page is not rendering at all.
I found this issue by debugging Chrome. Content is present and valid, however, the page is just blank. Any suggestions?
My code:
//other stuff...
client = await CDP('ws://localhost:9222/devtools/browser');
const {Target} = client;
const {browserContextId} = await Target.createBrowserContext();
const {targetId} = await Target.createTarget({
url: url,
width: viewportWidth,
height: viewportHeight,
enableBeginFrameControl: true
});
client = await CDP({target: targetId});
const {DOM, Emulation, Network, Page, Runtime, Security, HeadlessExperimental} = client;
await Page.enable();
await DOM.enable();
await Network.enable();
await HeadlessExperimental.enable();
await Page.navigate({url});
await Page.loadEventFired();
// this event is not fired (blank screen issue I guess)
await HeadlessExperimental.needsBeginFramesChanged();
//other stuff
Thanks in advance.
I've tried both from the example command line flags "--run-all-compositor-stages-before-draw --disable-new-content-rendering-timeout --disable-checker-imaging --disable-threaded-animation --disable-threaded-scrolling --enable-surface-synchronization" and with flags from the documentation
" --enable-surface-synchronization
--run-all-compositor-stages-before-draw
--disable-threaded-animation
--disable-threaded-scrolling
--disable-checker-imaging"
It produces the same result, rendering is not started, blank screen. Event "HeadlessExperimental.needsBeginFramesChanged()" is not fired.
#53: Maybe you're somehow missing the initial HeadlessExperimental.needsBeginFramesChanged event. It may be sent before the Headless.enable command completes. Try setting an async event handler for it before sending Headless.enable?
What I am surprised about, it the fact, that with opening gif with a target
Target.createTarget({
url: 'about:blank',
width: 600,
height: 350,
enableBeginFrameControl: true
});
I could have 70 fps with capturing PNG screenshots. Opening Youtube (as an animated example) with the same resolution produces 7 fps. Why does it fall so dramatically?
let screenshot = await HeadlessExperimental.beginFrame({
screenshot: {
format: 'png'
}
});
Of course, fps could be increased by reducing screenshot quality, but in general, I don't see fps increase comparing to default Page.captureScreenshot. Is that expected?
It's not expected that taking a single screenshot of an animation is more efficient with BeginFrameControl as compared to Page.captureScreenshot.
You can however combine sending BeginFrames with virtual time (see bit.ly/headless-rendering and the CompositorController tests here [1]) to capture frames at specific timestamps of an animation. This wouldn't be a real-time capture anymore though.
[1] https://cs.chromium.org/chromium/src/headless/public/util/compositor_controller_browsertest.cc
Hello,
First of all thanks for the wonderful project guys, I've been a follower of this issue for a long time now. I've managed to get everything working though, the base64 from all of the images are the same, from the tests and the documentation, I was expecting that using the frameTime parameters, I'd be able to render frames at times I specified:
let framesChanged;
HeadlessExperimental.needsBeginFramesChanged().then(function(result) {
framesChanged = result.needsBeginFrames;
}).catch(error => console.log(error));
await HeadlessExperimental.enable();
await Network.setCacheDisabled({cacheDisabled: true});
await Page.navigate({url: 'https://media.giphy.com/media/3o85xt08p2Y0hanhwQ/giphy.gif'});
await Page.loadEventFired();
let currentTime = new Date().getTime();
if (framesChanged) {
let initalResult = await HeadlessExperimental.beginFrame();
console.log(initalResult);
} else {
HeadlessExperimental.needsBeginFramesChanged().then(function(result) {
framesChanged = result.needsBeginFrames;
}).catch(error => console.log(error));
}
let screenshot = await HeadlessExperimental.beginFrame({
frameTime: currentTime + 2000,
screenshot: {
format: 'png'
}
});
console.log(screenshot);
let screenshot2 = await HeadlessExperimental.beginFrame({
frameTime: currentTime + 3000,
screenshot: {
format: 'png'
}
});
console.log(screenshot2);
The reason I'm asking is because the base64 outputs I receive after calling the beginframes is exactly the same although I specify different frameTime parameters.
Providing the frame timestamp only works reliably when using virtual time. Otherwise certain aspects of the page may use the frame timestamp you've provided, but others might not and use real time instead.
Hello Eric,
Thanks very much now I'm able to render screenshots at given times but not always. I've tried implementing it like this to no avail, would you be kind enough to help me on this? It sometimes works on some animations and sometimes it doesn't and skips a lot of frames (non deterministic). It's for a 25 fps / 10 seconds animation.
Here is my code:
totalFrameCount = 250;
delay = 500;
let epoch = await Emulation.setVirtualTimePolicy({policy: 'pauseIfNetworkFetchesPending', budget: totalFrameCount * 40});
epoch = epoch.virtualTimeBase;
console.log(epoch);
var frameCount = 0;
async function reallocateBudget() {
if (frameCount < totalFrameCount) {
epoch = await Emulation.setVirtualTimePolicy({policy: 'pauseIfNetworkFetchesPending', budget: totalFrameCount * 40)});
epoch = epoch.virtualTimeBase;
}
}
client.on('Emulation.virtualTimeBudgetExpired', reallocateBudget);
if (framesChanged) {
let initalResult = await HeadlessExperimental.beginFrame({
frameTime: epoch,
interval: 40
});
console.log(initalResult);
} else {
HeadlessExperimental.needsBeginFramesChanged().then(function(result) {
framesChanged = result.needsBeginFrames;
}).catch(error => console.log(error));
}
let screenshotBuffer;
var folderName = Math.random().toString(36).substring(7);
for (var i = 0; i < parseInt(totalFrameCount); i++) {
screenshotBuffer = await HeadlessExperimental.beginFrame({
frameTime: epoch + parseInt(delay) + (i * 40),
screenshot: {
format: 'jpeg',
quality: 88
}
});
frameCount++;
}
In the current situation it generates the first 50 frames correctly but the 51th frame is the screenshot that is supposed to be at the end of the animation. All the frames between are lost.
Try to make sure that you issue the beginFrame commands only while virtual time is paused, i.e. after the budget has elapsed and before you grant the next budget. Otherwise there's no guarantee at which timestamp the frame will be generated.
Thanks Eric for the quick reply, I'll try that. How much budget should I give? Also after pausing the virtual time what should the frameTime be? Is the current method ok? Epoch plus 40ms for a 25 fps capture (1000/25)?
I've updated my code accordingly Eric thanks, but after frame 51, the 52nd frame is too far ahead into the animation (end of the animation exactly), but weirdly if I set the frameTime explicitly for 52nd frame instead of running in a for loop, the correct screenshot is produced and no frames are skipped:
Loop (The screenshots are skipped to the end of the animation after frame 52):
let frameCount = 200;
async function reallocateBudget() {
console.log('Budget spent.')
}
client.on('Emulation.virtualTimeBudgetExpired', reallocateBudget);
let epoch = await Emulation.setVirtualTimePolicy({policy: 'pauseIfNetworkFetchesPending', budget: 10000});
let virtualTimeBase = epoch.virtualTimeBase;
let initialResult = await HeadlessExperimental.beginFrame({
frameTime: virtualTimeBase,
interval: 40
});
console.log(initialResult);
for (var i = 1; i < frameCount + 1; i++) {
await Emulation.setVirtualTimePolicy({policy: 'pauseIfNetworkFetchesPending', budget: 10000});
screenshotBuffer = await HeadlessExperimental.beginFrame({
frameTime: virtualTimeBase + (i * 40),
screenshot: {
format: 'jpeg',
quality: 88
}
});
fsPath.writeFile("frame-" + i + ".jpeg", screenshotBuffer.screenshotData, 'base64', function(error) {
if (error) {
sendNotification(9, "Could not save frame " + i + " to the directory.", error);
} else {
console.log("Frame " + i + " created.");
}
});
}
Explicit Frame (Correct screenshot in animation):
let frameCount = 200;
async function reallocateBudget() {
console.log('Budget spent.')
}
client.on('Emulation.virtualTimeBudgetExpired', reallocateBudget);
let epoch = await Emulation.setVirtualTimePolicy({policy: 'pauseIfNetworkFetchesPending', budget: 10000});
let virtualTimeBase = epoch.virtualTimeBase;
let initialResult = await HeadlessExperimental.beginFrame({
frameTime: virtualTimeBase,
interval: 40
});
console.log(initialResult);
let explicitFrame = await HeadlessExperimental.beginFrame({
frameTime: virtualTimeBase + 2200,
screenshot: {
format: 'jpeg',
quality: 88
}
});
fsPath.writeFile("frame-" + 52 + ".jpeg", explicitFrame.screenshotData, 'base64', function(error) {
if (error) {
sendNotification(9, "Could not save frame " + 2 + " to the directory.", error);
} else {
console.log("Frame " + 52 + " created.");
}
});
I'm not sure what you're trying to accomplish exactly, but here's what I'd do if I wanted to capture an animation at a specific FPS:
0) init virtual time in paused state, no budget. save |ts = virtual_time_base|.
1) capture a screenshot at |frame_time = ts|.
2) grant |FPS/1000 ms| of virtual time, e.g. 100ms if capturing at 10 FPS.
3) on virtual time expiration, update |ts += FPS/1000ms| and capture another screenshot at |ts|.
4) go to 2 and repeat until you've reached your desired end timestamp.
What you're doing isn't quite that. Try to ensure that the virtual time matches your frame timestamps and use the virtualTimeBudgetExpired event to initiate the next screenshot, i.e. after you grant VT, you should always wait for the expiration event.
;){kind=link}
Comment 1 by skyos...@chromium.org
, Sep 14 2016