[1] described a situation in which the video and audio chunks being
transported in the respective MediaStreams do not have the same clock
origin, which caused a freeze upon playback.
https://crbug.com/597034 , where this issue was uncovered, patched
MediaRecorder's WebmMuxer code by keeping a rolling "last seen timestamp"
but that was a temporary solution: MR produces reproducible files,
but the video and the audio could be out of synch.
This was the interchange:
>On 2016/04/06 00:49:01, miu wrote:
>> lgtm, HOWEVER: The reason the timestamps are far apart is because there is
>> more/less playout buffering going on for audio versus video. The timestamps
>> should be reflecting the playout time at which each frame should be displayed
>> on screen (or, for audio, when each frame should come out the speaker).
>>
>> With this change, I'm willing to bet A/V sync is now a problem. At least, it
>> will be on some platforms. Solving this may involve having your WebM muxer
>> queuing-up either audio or video (or both), and then only output both to the
>> WebM container once they've been aligned w.r.t. their playout timestamps.
>>
>> So, I'm lgtm'ing this, and leave it up to you to decide whether this change
>> should be committed as-is, and whether the A/V sync performance aspects of
>> your feature should be addressed now or later.
>
>I don't think this is a case of audio/video sync adjustments. I
>printed out the base::TimeTicks timestamps of the incoming video
>and audio recorded frames and I got the following numbers.
>
>First a bunch of Video Frames arrive with correlative timestamps
>
>media::WebmMuxer::OnEncodedVideo - 59928B @ 2149924 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo: delaying until audio track ready.
>media::WebmMuxer::OnEncodedVideo - 108B @ 2183258 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo: delaying until audio track ready.
>media::WebmMuxer::OnEncodedVideo - 143B @ 2216591 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo: delaying until audio track ready.
>
>and now the first Audio arrives with completely different timestamps:
>
>media::WebmMuxer::OnEncodedAudio - 386B @ 1269455645930 bogo-microseconds
>media::WebmMuxer::AddAudioTrack format: 1 channel_layout: 2 channels: 1
>sample_rate: 48000 bits_per_sample: 16 frames_per_buffer: 2880 effects: 0
>mic_positions:
>
>and now they start mingling, each with their own reference.
>
>media::WebmMuxer::OnEncodedVideo - 214B @ 2249924 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 217B @ 2283258 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 308B @ 2316591 bogo-microseconds
>media::WebmMuxer::OnEncodedAudio - 383B @ 1269455771097 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 348B @ 2349924 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 309B @ 2383257 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 640B @ 2416591 bogo-microseconds
>media::WebmMuxer::OnEncodedAudio - 383B @ 1269455878194 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 809B @ 2449924 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 977B @ 2483257 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 1118B @ 2516591 bogo-microseconds
>media::WebmMuxer::OnEncodedAudio - 383B @ 1269455988642 bogo-microseconds
>media::WebmMuxer::OnEncodedVideo - 1296B @ 2549924 bogo-microseconds
>
>(I also checked that these timestamps correlate with
>those coming from the Tracks, the recorders did not
>interfere with them significantly).
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=597034#c12
Comment 1 by mcasas@chromium.org
, May 30 2017