canvas.captureStream() should produce a frame after creation
Reported by
vida...@gmail.com,
Nov 9
|
||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0 Steps to reproduce the problem: 1. Create a canvas and draw something to it 2. Wait a while to ensure all drawing is flushed, e.g. with a setTimeout(). 3. Call captureStream() (either with undefined fps or fps=0) on the canvas, and feed it to e.g. a video element. 4. Call requestFrame() on the captured stream. 5. Wait a while, then draw to the canvas. What is the expected behavior? I expected a full frame to be captured from the canvas and sent over the stream immediately after the requestFrame() call was made (step 4). What went wrong? Instead of immediately capturing a frame, it flags that a frame is requested, and only captures it on the next draw (step 5). Did this work before? N/A Does this work in other browsers? N/A Chrome version: 70.0.3538.77 (Official Build) (64-bit) (cohort: Stable) Channel: stable OS Version: 10.0 Flash Version: Shockwave Flash 31.0 r0 The spec is slightly vague on what exactly requestFrame() does, but I would expect it to actually go an get a frame.
,
Nov 10
,
Nov 12
Tried testing the issue on chrome reported version# 70.0.3538.77 using Windows-10 with steps mentioned below: 1) Launched chrome reported version, downloaded and placed both the files into single folder provided in comment# 0 2) Dragged and dropped the file in to new tab page, observed after dropping the file observed single red frame on the webpage, after that immediately seen two red and blue frames on the web page @Reporter: Please find the attached screencast for your reference and provide your feedback on it, if possible could you please provide screencast of the issue which help in better understanding in further triaging it in better way. Thanks!
,
Nov 12
Thanks for the feedback. The captured movie that you supplied actually shows the reported behavior, so I will try to break it down more clearly. To highlight/reproduce the problem itself, it is maybe best to edit the JS such that step 5 is removed entirely. This will highlight that the video element never receives a frame, no matter how much requestFrame() is called. See the attachment "no-step-5" for a screencast. A second step, can then be to re-instate step 5, but maybe with a longer delay (change the current timeout 1001 ms to something longer). This highlights that the actual frame is only captured on the next draw call to the canvas. See the attachment "5s-delay-step-5" for a screencast with a 5 s delay before the blue rect is drawn. Does this help in confirming the issue?
,
Nov 12
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Nov 14
Able to reproduce the issue on Mac 10.13.6, Win-10 and Ubuntu 17.10 using chrome reported version #70.0.3538.77 and latest canary #72.0.3608.4. This is a non-regression issue as it is observed from M60 old builds. Hence, marking it as untriaged to get more inputs from dev team. Thanks...!!
,
Jan 7
vidartf@ added your code to a codepen: https://codepen.io/miguelao/pen/XoYrme?editors=1010 from the code point of view, we request a frame upon captureStream() [1] and I can see how it ripples to the canvas_capture_media_stream_track.cc and from it to canvas_draw_listener.cc, but it's never honored, because HTMLCanvasElement::NotifyListenersCanvasChanged() is never activated (since there's no modifications in the source canvas). [1] https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/mediacapturefromelement/html_canvas_element_capture.cc?sq=package:chromium&dr&g=0&l=85
,
Jan 7
After more debugging, it seems that canvas.captureStream() generates that first frame just fine (also w/ canvas.captureStream(0)), but only if we add step 3 _before_ any wait/sleep code, i.e. I made another codepen to illustrate: https://codepen.io/miguelao/pen/XoYrme?editors=1010 Assuming: function timeout(ms) { return new Promise(resolve => setTimeout(resolve, ms)); } If we create (A) the captureStream() _before_ the await (B) (see doStuff1()), I see a first VideoFrame correctly generated in the captureStream(), whereas if we first await (B) and then generate the |stream| (A) ( see doStuff2()), then that first frame is not generated and we have to wait until more stuff is painted, to see the <video> element being updated. async function doStuff1() { stream = canvas.captureStream();// A await timeout(10); // B video.srcObject = stream; await timeout(2000);' // paint more stuff } async function doStuff2() { await timeout(10); // B stream = canvas.captureStream(); // A video.srcObject = stream; await timeout(2000);' // paint more stuff } This is very bizarre or I'm missing something very obvious :-?
,
Jan 8
> Instead of immediately capturing a frame, it flags that a frame is requested, and only captures it on the next draw (step 5). This is how it is supposed to work as specified in the spec. It is written in the captureStream() section though, instead of requestFrame() which might have confused you. Each time requestFrame() is called, an internal bool property of frameCaptureRequested is set to true. That allows us to capture a frame in the next paint event. https://www.w3.org/TR/mediacapture-fromelement/#html-canvas-element-media-capture-extensions
,
Jan 8
Yeah, after an offline conversation w/ emircan@, the behaviour here seems WAI: (1) <canvas> --<captured_stream>--> data sink (<video>, MediaRecorder, etc) We would only see a VideoFrame through |captured_stream| when: - |captured_stream| has been created and connected to <canvas> (this registers |captured_stream| as a Listener of <canvas>). - The <canvas> context decides to paint, pinging its Listeners. Which explains the behaviour that surprised me in #8: introducing any amount of timeout between drawing and creation+connection of the canvas.captureStream(), will make the latter miss the <canvas> drawing, and we would consequently see no video frames in it. Teaching |captured_stream| to force <canvas> to draw (via FinalizeFrame()) is out of the question: Listeners may not change the <canvas> status, and FinalizeFrame() can change the drawing state (e.g. its dirty rectangle). IOW, the media graph detailed in (1) is a push-based flow and can only be manipulated by the source, in this case <canvas>, creating new data. The intermediate conduit (|captured_stream|) and the sink cannot, and may not, affect the data flow. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by dtapu...@chromium.org
, Nov 9