Bitrate loss when simultaneously recording H264 video using MediaRecorder and encoding it to send to a WebRTC peer
Reported by
donald@donaldharvey.co.uk,
May 23 2017
|
||||||
Issue descriptionUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Steps to reproduce the problem: 1. Use getUserMedia to request the highest-resolution video stream that is supported by the client. 2. Attach it to MediaRecorder set up with videoBitsPerSecond explicitly set to a suitably high bitrate and with codecs=video/webm;codecs=avc1. 3. Set up a webrtc connection with the same media stream, where the SDP is edited such that the video stream is encoded using H264. 4. Attempt to initiate a recording. What is the expected behavior? The resulting video recorded by MediaRecorder should have a similar bitrate to videoBitsPerSecond, and it should continue until MediaRecorder.stop() is called. (As long as a stream isn't attached to webrtc, this seems to work fine.) What went wrong? The resulting video recorded by MediaRecorder has a very low bitrate (for me 40x lower than it should be), and the MediaRecorder stops on its own after a few seconds, without emitting an error. Did this work before? N/A Does this work in other browsers? N/A Chrome version: 60.0.3108.0 Channel: stable OS Version: OS X 10.11.6 Flash Version: This is on OS X on a MacBook Pro Retina 2015 using the built-in 720p camera. Hardware video encoding is definitely enabled.
,
May 23 2017
Can you please provide test url/testcase for the ease of reproducing the bug.
,
May 23 2017
In what order are you making these calls? Can you provide some minimized test case? On Macbook, what I observed is that only one session would be VTCompressionSession is HW accelerated. In this case, it might be that peer connection is HW encoded but MediaRecorder falls back to SW, or vice versa.
,
May 23 2017
Interesting. We're initialising the MediaRecorder instance first, then later adding the stream to the RTCPeerConnection, after which MediaRecorder.start() is called. I'll extract the relevant code from my project tomorrow and try to produce a minimal test case.
,
May 25 2017
Thanks for the test case. As I explained before, hardware accelerated VTCompressionSession is picked up by the WebRTC connection. MediaRecorder cannot create one, so falls back to OpenH264 encoder implementation. OpenH264 implementation uses the calculated bitrate, 3870720 bits per second in your test. The problem regarding it stopping early is recently fixed on issue 716451 . Can you try the canary tomorrow to see if it helps? If bitrate is still lower than expected, it might just be a case of OpenH264 implementation not keeping up with the real time 720p video. You can try to initialize MediaRecorder with VP8 or VP9 to see if it improves.
,
May 25 2017
,
Jun 1 2017
This is better, and doesn't stop early, but still isn't using the full bitrate (1545 kbit/s this time). Ideally, it'd be possible to hardware-encode both sessions. If that's not possible, then for us--and probably in most use-cases--it would make more sense to prioritise the MediaRecorder encoding quality, and use hardware for the MediaRecorder session and have the WebRTC session fall back to OpenH264. For now, as a workaround we're using VP8 for the WebRTC session to allow us to use H264 with MediaRecorder.
,
Jun 1 2017
Thanks for the getting back. Unfortunately, creating two sessions with HW encode is limited by VideoToolbox which we don't have control on. When kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder flag is set, only one session gets created successfully[0] as far as I checked on Macbooks I have. Your solution sounds good to me. We have also observed that software VP8/VP9 has better performance than OpenH264. [0] https://cs.chromium.org/chromium/src/media/gpu/vt_video_encode_accelerator_mac.cc?rcl=a7dc4e822a4cddc14b9929a6f7aa2a85e69252d8&l=472
,
Jun 2 2017
Got it. Thanks for looking further into it! |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by donald@donaldharvey.co.uk
, May 23 2017