Canvas / Audio stream MediaRecorderAPI timestamp issues
Reported by
shant.ke...@gmail.com,
Apr 16 2018
|
|||||||||||||||||||||
Issue description
Chrome Version : 65.0.3325.181
URLs (if applicable) :
OS version : Mac OS X 10.13.4 (Not restrained to OS happens on Ubuntu 16.04 and Windows 10 too)
Network (such as Cable/DSL/Dial up etc): Cable
Audio/Video format (if applicable): vp8, h264
Special chrome flags (if applicable): N/A
Behavior in Safari (if known): Not known
Behavior in Firefox (if known): Not known
Video issue, Audio issue, both, neither?
Audio issue but could be related to video as well (MediaRecorderApi)
Flash or HTML5? HTML5
What steps will reproduce the problem?
(1) Use getUserMedia(...).then(function(stream) { ... });
(2) Create a video element var video = document.createElement('video'); and set video.srcObject = stream;
(2) create a canvas and capture the stream ex: var canvasStream = canvas.captureStream(30);
(3) draw the video onto the canvas ex: setInterval(function() { context.drawImage(video, 0, 0, 1280, 720); }, 0); (recursive requestAnimationFrame also works)
(4) create a new mediaStream with the audio input stream and canvas capture stream. ex: var mediaStream = new MediaStream([canvasStream.getVideoTracks()[0], stream.getAudioTracks()[0]]),
(5) create a new MediaRecorder with vp8 or h264 ex:
new MediaRecorder(mediaStream, {
mimeType: 'video/webm;codecs=vp8',
//videoBitsPerSecond: 4000000,
audioBitsPerSecond: 128000
});
(6) send the data via socketio or webrtc to an ffmpeg/gstreamer/kurento that streams to an RTMP (ex: nginx-rtmp or facebook rtmp)
ex ffmpeg commands:
vp8:
ffmpeg -i vp8.webm -t 5400 -ar 44100 -b:a 128k -c:v libx264 -c:a aac -pix_fmt yuv420p -r 30 -g 60 -vb 2048k -preset ultrafast -crf 0 -bufsize 4096k -minrate 2000k -maxrate 4000k -f flv rtmp://rtmpserver:1935/some/path
h264 (webrtc):
ffmpeg -i h264.webm -ar 44100 -b:a 128k -c:v copy -c:a aac -f flv rtmp://rtmpserver:1935/some/path
What is the expected result?
To generate video content that is capable of being transcoded / streamed to an RTMP for HLS generation without date timestamp issues, audio ticks or audio/video desync.
What is the actual result?
There are multiple actual results:
- When vp8 ffmpeg complains about the following errors:
ex: stderr: [flv @ 0x7ff3050b2c00] Non-monotonous DTS in output stream 0:1; previous: 230, current: 194; changing to 230. This may result in incorrect timestamps in the output file.
ex: stderr: Past duration 0.999992 too large
ex: stderr: [aac @ 0x7f8f0d057c00] Queue input is backward in time
with audible tick
- When h264:
+ When using the copy command mentioned above (single stream) RTMP generates HLS *.ts are not visible but are audible by most players (VLC, ffplay, jwplayer)
+ When using WebRTC when audio and video are separate streams you get audio desync (audio going ahead of video)
Any additional information (anything else which may help us debug the issue)?
- Less powerful PCs experience it more
- Higher resolution cameras / higher resolution canvas experience it more
Please attach the HTML5/JavaScript code or audio/video files as well as screenshot and/or videos (if applicable)
- Attached is a vp8.webm generated through this process, then with the ffmpeg command:
ffmpeg -i vp8.webm -t 5400 -ar 44100 -b:a 128k -c:v libx264 -c:a aac -pix_fmt yuv420p -r 30 -g 60 -vb 2048k -preset ultrafast -crf 0 -bufsize 4096k -minrate 2000k -maxrate 4000k -f flv rtmp://rtmpserver:1935/some/path
- Attached is h264.webm and resulting HLS m3u8 and ts with copy command
ffmpeg -i h264.webm -ar 44100 -b:a 128k -c:v copy -c:a aac -f flv rtmp://rtmpserver:1935/rtmp/some/path
,
Apr 16 2018
,
Apr 19 2018
Thanks for filing the issue! Tried checking the issue on reported chrome version 65.0.3325.181 using Mac 10.13.1 with the below mentioned steps. 1. Launched chrome 2. Downloaded and unzipped file provided in comment#0 3. Tried opening the files in a new tab. @Reporter: As we are unable to open those files and it's out of scope for us to create a video element and test the issue, requesting you to provide us a test file which helps us to triage the issue in a better way.
,
Apr 20 2018
Attached is an electron fiddle (it was the easiest way without configuring servers, but the same issues happen in the browser, node server, sockets and then going through ffmpeg) Steps to reproduce: 1) Unzip the fiddle 2) Install the project via terminal, go in directory and run `npm install`. 3) In terminal execute from directory type: `pkill "ffmpeg" & npx electron .` (the pkill is incase it didn't stop the ffmpeg process properly and to quickly relaunch) 4) Go to facebook.com create a live video / post (in a group for example) 5) In the modal that appears you will see a "connect" tab at the top 6) Copy and paste the ServerURL and StreamKey into the input in the electron fiddle 7) click "Emit", in terminal you should now see ffmpeg output and error warnings as they happen (more frequent with VP8 and/or slower computers / higher canvas and video resolutions) 8) Once emitting in facebook you should see your self in the preview screen (no need to go live) You can change the vp8 option or to h264 which will change the encoding in the MediaRecorder in index.html Ffmpeg arguments can be found / adjusted in config.js
,
Apr 20 2018
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 24 2018
Mac triage: marking for TE confirmation per #4.
,
Apr 27 2018
Tried checking the issue on reported chrome version 65.0.3325.181 using Mac 10.13.1 with the below mentioned steps. 1. Downloaded/Unzipped the file provided in comment#4 2. In Terminal Changed directory to the folder containing unzipped file then we have run the command "npm install electron-fiddle" to install the project. 3. Given commands `pkill "ffmpeg" & npx electron .` one after other. 4. Observed Electron being launched 5. Navigated to facebook.com and clicked on live video -> Connect 6. Copied and pasted "Server URL" and "Stream Key" in the tab where index.html is opened then clicked "Emit" 7. Tried changing codec from VP8 to H264 and performed step 6. We didn't observe any ffmpeg output and error warnings in Terminal. @Reporter: Could you please let us know if we have missed anything in the process. Any further inputs from your end may be helpful. Thanks!
,
Apr 27 2018
The steps are correct. When switching between codecs it may be better to restart the application with the `pkill "ffmpeg" & npx electron .` command Is ffmpeg already installed on the computer? Weere there any outputs from ffmpeg? Example missing codec? Invalid input/output? etc Thanks
,
Apr 27 2018
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
May 3 2018
Tried checking the issue as per comment#8 i.e., exited electron and restarted the application with commands `pkill "ffmpeg" & npx electron .` while switching between codecs, it shows similar behaviour as said in C#7. When we try to install ffmpeg with command "sudo brew install ffmpeg" we are getting the following... Error: Running Homebrew as root is extremely dangerous and no longer supported. As Homebrew does not drop privileges on installation you would be giving all build scripts full access to your system. Didn't observe any outputs from ffmpeg like missing codec, Invalid input/output ... @Reporter: Could you please let us know, exiting electron and running the above mentioned command(s) in order to restart the application is the right way of doing, Please mention if this isn't. Any further inputs from your end may be helpful. Thanks!
,
May 4 2018
Is there a way for us to schedule a skype, hangouts or other that way we can observe what's happening on your computer? I can also replicate the issue on mine
,
May 4 2018
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
May 4 2018
IIUC the processing pipeline would look a bit like this:
<-----MediaRecorder---------->
<webcam> --> <video> --> <canvas> --> captureStream() --> VideoTrackEncoder \
\----------------------------------------------> AudioTrackEncoder --> WebmMuxer
Some comments:
- MediaRecorder uses an AudioTrackEncoder and a VideoTrackEncoder to encode
independently the audio and video data, respectively. Either is "reactive"
in the sense that AudioBuffers/VideoFrames are encoded as they are provided
from the respective MediaStreamTracks.
- A partial diagram of the classes/objects and the data flow is in [0].
- Once encoded, WebmMuxer purposely maintains a monotonically increasing
timestamp [1] regardless of the "presentation timestamp" or similar associated
with the unencoded data. This is because players don't like (choke on) non
monotonically increasing timestamps. But this also means that encoding a
e.g. heavily processed video and a pass through audio will most likely incur
in a lack of lip sync. Perhaps WebAudio could be used to delay the audio for
this use case, but I'm not sure.
With all that, I'd suggest to try and investigate with a Chromium build
and see what's happening, if it's WAI or there's a bug ..?
https://www.chromium.org/developers/how-tos/get-the-code
[0] https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/blink/renderer/modules/mediarecorder/README.md
[1] https://cs.chromium.org/chromium/src/media/muxers/webm_muxer.cc?sq=package:chromium&dr=CSs&l=355
,
May 7 2018
emircan@/rtoy@ - Gentle Ping...!! Could someone please have a look into the issue and help us in further triaging of the issue as from comment #7 and #10 it doesn't seems to reproducible from TE-end. Thanks...!!
,
May 9 2018
Unable to reproduce the issue on mac 10.13.3 using chrome version #66.0.3359.139 from comment #4. Hence, removing the Needs-TestConfirmation label. Thanks...!!
,
May 10 2018
Here is an attached video, hope it helps. I noticed in C#7 the command run was "npm install electron-fiddle" it should be just "npm install" within the electron-fiddle folder. I've tested on multiple OS, Mac, Ubuntu, Windows, it's reproducible 100%.
,
Jun 14 2018
As this issue is not reproducible at TE end, adding 'TE-NeedsTriageHelp' and requesting someone from Blink>MediaRecording team to look into this issue and help in further triaging. Thanks...
,
Jan 4
I wrote a codepen in https://codepen.io/miguelao/pen/BvYZNV?editors=1010 implementing the following scheme: <video> --> <canvas> --> captureStream() -[canvasMediaStream]--> VideoTrackEncoder \ \---> WebAudio delay -------------[webAudioMediaStream]--> AudioTrackEncoder --> WebmMuxer 2 SECONDS the produced recorded WebM is validated (mkvalidator) and played back correctly using VLC on my Linux Debian. Of course the recorded audio is delayed 2s with respect to the video, but that is not a MediaRecorder issue, it's squarely into the application realm. Another topic is that it might be hard (currently impossible?) to estimate/measure the delay introduced in the --> <canvas> --> captureStream() -- segment, so as to configure the WebAudio delay element on the fly... Since this issue has seen little activity I'll close it as Archived; shant.kevonian@, I would recommend trying without the RTMP/WebRTC section of your demo, please reopen it or comment if you use or modify the codepen so as to show your issue in a clearer way. Thanks! |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by hta@chromium.org
, Apr 16 2018