New issue
Advanced search Search tips

Issue 903832 link

Starred by 3 users

Issue metadata

Status: WontFix
Owner: ----
Closed: Jan 8
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Mac
Pri: 2
Type: Bug



Sign in to add a comment

canvas.captureStream() should produce a frame after creation

Reported by vida...@gmail.com, Nov 9

Issue description

UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0

Steps to reproduce the problem:
1. Create a canvas and draw something to it
2. Wait a while to ensure all drawing is flushed, e.g. with a setTimeout().
3. Call captureStream() (either with undefined fps or fps=0) on the canvas, and feed it to e.g. a video element.
4. Call requestFrame() on the captured stream.
5. Wait a while, then draw to the canvas.

What is the expected behavior?
I expected a full frame to be captured from the canvas and sent over the stream immediately after the requestFrame() call was made (step 4).

What went wrong?
Instead of immediately capturing a frame, it flags that a frame is requested, and only captures it on the next draw (step 5).

Did this work before? N/A 

Does this work in other browsers? N/A

Chrome version: 70.0.3538.77 (Official Build) (64-bit) (cohort: Stable)  Channel: stable
OS Version: 10.0
Flash Version: Shockwave Flash 31.0 r0

The spec is slightly vague on what exactly requestFrame() does, but I would expect it to actually go an get a frame.
 
requestFrame.html
140 bytes View Download
requestFrame.js
794 bytes View Download
Components: Blink>MediaStream>CaptureFromElement
Labels: Needs-Triage-M70
Cc: viswa.karala@chromium.org
Labels: Triaged-ET Needs-Feedback
Tried testing the issue on chrome reported version# 70.0.3538.77 using Windows-10 with steps mentioned below:
1) Launched chrome reported version, downloaded and placed both the files into single folder provided in comment# 0
2) Dragged and dropped the file in to new tab page, observed after dropping the file observed single red frame on the webpage, after that immediately seen two red and blue frames on the web page

@Reporter: Please find the attached screencast for your reference and provide your feedback on it, if possible could you please provide screencast of the issue which help in better understanding in further triaging it in better way.

Thanks!
903832.mp4
1020 KB View Download
Thanks for the feedback. The captured movie that you supplied actually shows the reported behavior, so I will try to break it down more clearly.

To highlight/reproduce the problem itself, it is maybe best to edit the JS such that step 5 is removed entirely. This will highlight that the video element never receives a frame, no matter how much requestFrame() is called. See the attachment "no-step-5" for a screencast.

A second step, can then be to re-instate step 5, but maybe with a longer delay (change the current timeout 1001 ms to something longer). This highlights that the actual frame is only captured on the next draw call to the canvas. See the attachment "5s-delay-step-5" for a screencast with a 5 s delay before the blue rect is drawn.

Does this help in confirming the issue?
no-step-5.gif
63.1 KB View Download
5s-delay-step-5.gif
21.0 KB View Download
Project Member

Comment 5 by sheriffbot@chromium.org, Nov 12

Labels: -Needs-Feedback
Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: Target-72 FoundIn-72 M-72 FoundIn-71 FoundIn-70 OS-Linux OS-Mac
Status: Untriaged (was: Unconfirmed)
Able to reproduce the issue on Mac 10.13.6, Win-10 and Ubuntu 17.10 using chrome reported version #70.0.3538.77 and latest canary #72.0.3608.4.
This is a non-regression issue as it is observed from M60 old builds. 
Hence, marking it as untriaged to get more inputs from dev team.

Thanks...!!
Cc: emir...@chromium.org
Components: Blink>Canvas
Summary: canvas.captureStream() should produce a frame after creation (was: The canvas mediatrack's requestFrame() method should immediately fetch a frame)
vidartf@ added your code to a codepen:
 https://codepen.io/miguelao/pen/XoYrme?editors=1010

from the code point of view, we request a frame upon captureStream() [1]
and I can see how it ripples to the canvas_capture_media_stream_track.cc
and from it to canvas_draw_listener.cc, but it's never honored, because
HTMLCanvasElement::NotifyListenersCanvasChanged() is never activated 
(since there's no modifications in the source canvas).

[1] https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/mediacapturefromelement/html_canvas_element_capture.cc?sq=package:chromium&dr&g=0&l=85

Cc: mcasas@chromium.org
Status: Available (was: Untriaged)
After more debugging, it seems that canvas.captureStream() generates
that first frame just fine (also w/ canvas.captureStream(0)), but
only if we add step 3 _before_ any wait/sleep code, i.e. I made another
codepen to illustrate:
  https://codepen.io/miguelao/pen/XoYrme?editors=1010

Assuming:
  function timeout(ms) { return new Promise(resolve => setTimeout(resolve, ms)); }

If we create (A) the captureStream() _before_ the await (B) (see doStuff1()), 
I see a first VideoFrame correctly generated in the captureStream(), 
whereas if we first await (B) and then generate the |stream| (A) ( see 
doStuff2()), then that first frame is not generated and we have to wait
until more stuff is painted, to see the <video> element being updated.


async function doStuff1() {
  stream = canvas.captureStream();// A
  await timeout(10);              // B
  video.srcObject = stream;
  
  await timeout(2000);'
  // paint more stuff
}

async function doStuff2() {
  await timeout(10);               // B
  stream = canvas.captureStream(); // A
  video.srcObject = stream;
  
  await timeout(2000);'
  // paint more stuff
}

This is very bizarre or I'm missing something very obvious :-?
> Instead of immediately capturing a frame, it flags that a frame is requested, and only captures it on the next draw (step 5).

This is how it is supposed to work as specified in the spec. It is written in the captureStream() section though, instead of requestFrame() which might have confused you. Each time requestFrame() is called, an internal bool property of frameCaptureRequested is set to true. That allows us to capture a frame in the next paint event.
https://www.w3.org/TR/mediacapture-fromelement/#html-canvas-element-media-capture-extensions
Status: WontFix (was: Available)
Yeah, after an offline conversation w/ emircan@, the behaviour here
seems WAI:

(1)  <canvas> --<captured_stream>--> data sink (<video>, MediaRecorder, etc)

We would only see a VideoFrame through |captured_stream| when:
- |captured_stream| has been created and connected to <canvas>
  (this registers |captured_stream| as a Listener of <canvas>).
- The <canvas> context decides to paint, pinging its Listeners.

Which explains the behaviour that surprised me in #8: introducing any 
amount of timeout between drawing and creation+connection of the 
canvas.captureStream(), will make the latter miss the <canvas> drawing, 
and we would consequently see no video frames in it.

Teaching |captured_stream| to force <canvas> to draw (via FinalizeFrame()) 
is out of the question: Listeners may not change the <canvas> status, 
and FinalizeFrame() can change the drawing state (e.g. its dirty 
rectangle).

IOW, the media graph detailed in (1) is a push-based flow and can
only be manipulated by the source, in this case <canvas>, creating
new data. The intermediate conduit (|captured_stream|) and the sink
cannot, and may not, affect the data flow.

Sign in to add a comment