AV Sync issues with fMp4 using MSE
Reported by
benjamin...@bonnierbroadcasting.com,
Aug 22 2017
|
|||||
Issue descriptionUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36 Steps to reproduce the problem: 1. Open https://video-dev.github.io/hls.js/demo/?src=http%3A%2F%2Fcsm-e.cds1.yospace.com%2Fcsm%2Flive%2F129564706.m3u8&enableStreaming=true&autoRecoverError=true&enableWorker=true&dumpfMP4=false&levelCapping=-1&defaultAudioCodec=undefined 2. Press the shield in the address bar and press 'load unsafe scripts' 3. Ads are played every 5 minutes using server-side ad-stitching, advertisements always start with a short jingle and the word "Reklampaus" and ends with "Nu fortsätter vi"/"Snart fortsätter programmet" - letting the stream play for about 10 minutes should be enough. 4. Observe the lipsync issue ( AV Sync ) Alternatively using the attached mp4: 1. Open the mp4 in chrome 2. Observe the distorted audio 3. Check the error logs in chrome://media-internals/ What is the expected behavior? There should be no AV Sync issue, lip sync issue. What went wrong? From chrome://media-internals "Large timestamp gap detected; may cause AV sync to drift. time:713814604us expected:716737270us delta:-2922666us" The following is a quote from the maintainer of HLS.js: "Chrome is slowly drifting audio over time (frame by frame). this happens also with video disabled. the workflow is as follow: hls.js loads and parses each TS fragment, then remux them into fMP4, this fMP4 is then injected into the sourceBuffer. each fmp4 is made of a start decode time Offset, followed by all audio samples (each having the same duration) after injection into the sourcebuffer, the web browser processes this fMP4 and fill the audio buffer by parsing all audio frames. what is happening is that on some specific audio sample, Chrome accounts for 2 audio samples instead of 1. => an additional audio sample is inserted, and it is shifting the following ones... which in the end generates an audible drift." A theory is that chrome is misinterpreting an AAC FIL element. The attached mp4 file is a an audio only fmp4 extracted from the above hls.js demo site, I'm not sure if it will help debugging the issue but the it causes a lot of spam of error logs in media-internals about a timestamp gap ( and the playback audio is distorted, which it isn't in eg. Firefox ). Did this work before? N/A Does this work in other browsers? N/A Chrome version: 60.0.3112.101 Channel: stable OS Version: OS X 10.12.6 Flash Version: Hls.js github issue: https://github.com/video-dev/hls.js/issues/828
,
Aug 22 2017
,
Aug 23 2017
Seems like something went wrong when creating the bug: "Does this work in other browsers" Should be, Yes.
,
Aug 24 2017
Asking Dale for initial feedback or alternative owner.
,
Aug 24 2017
Is the issue unique for Mac OS X or can it be reproduced on other platforms as well?
,
Aug 24 2017
Likely your media is bad. Audio timestamps are generated based on the decoded samples so they are 100% accurate. If you time the video frames without regard for the audio samples you'll end up with the error messages like those in the log.
,
Aug 24 2017
@henrika: It can be reproduced on Windows 10 as well. @dalecur: We're investigating this as well, the HLS we're transmuxing is a live stream with ads stitched into the stream and the fragments comes from different sources. The weird thing is that it works fine on other browsers like firefox & IE11.
,
Aug 25 2017
Agree with Dale - it seems likely that the timestamps are incorrect in the fragmented mp4. This is easy to get wrong when you're splicing ads into content.
The audio buffer that comes just before the ad gets spliced in will have some timestamp T and duration D. The timestamp of the next audio buffer, T_next (the first for the ad) should equal T + D.
When I run the hls.js demo I see T_next = T + D + ~60 msec.
Re, this point:
> what is happening is that on some specific audio sample, Chrome accounts for 2 audio samples instead of 1.
The durations I'm seeing, including those around the "Large timestamp gap..." look very consistent.
decoder_stream.cc(565)] OnBufferReady<audio>: 0, timestamp: 136142645 duration: 21333 size: 16 side_data_size: 0 is_key_frame: 1 encrypted: 0 discard_padding (ms): (0, 0)
decoder_stream.cc(565)] OnBufferReady<audio>: 0, timestamp: 136163979 duration: 21333 size: 33 side_data_size: 0 is_key_frame: 1 encrypted: 0 discard_padding (ms): (0, 0)
decoder_stream.cc(565)] OnBufferReady<audio>: 0, timestamp: 136185312 duration: 21333 size: 16 side_data_size: 0 is_key_frame: 1 encrypted: 0 discard_padding (ms): (0, 0)
render_media_log.cc(30)] MediaEvent: MEDIA_ERROR_LOG_ENTRY {"error":" Large timestamp gap detected; may cause AV sync to drift. time:136185312us expected:136121332us delta:63980us"}
decoder_stream.cc(565)] OnBufferReady<audio>: 0, timestamp: 136206645 duration: 21333 size: 16 side_data_size: 0 is_key_frame: 1 encrypted: 0 discard_padding (ms): (0, 0)
decoder_stream.cc(565)] OnBufferReady<audio>: 0, timestamp: 136227979 duration: 21333 size: 16 side_data_size: 0 is_key_frame: 1 encrypted: 0 discard_padding (ms): (0, 0)
decoder_stream.cc(565)] OnBufferReady<audio>: 0, timestamp: 136249312 duration: 21333 size: 16 side_data_size: 0 is_key_frame: 1 encrypted: 0 discard_padding (ms): (0, 0)
This all looks good. 21333 us at 48khz is 1024 samples per buffer - basically no crazy numbers here. No signs of any 2x or .5x size buffers.
Re, the attached mp4: the distortion is present in both firefox (linux, version 55) and chrome for me. I also tried using ffplay and found it there (along with lots of warning output from the AAC decoder). Looks like that file may have other issues, so I've focused on the hls.js demo steps for now.
,
Aug 25 2017
Re: Firefox - I think their media clock implementation is different in a way that makes it more forgiving - perhaps they trust the mp4 timestamps more. IIRC chrome used to trust the muxed timestamps for pacing the media clock, but found that the numbers are often imprecise. The imprecision doesn't necessarily accumulate, but it does awful things for video smoothness.
,
Sep 4 2017
Thanks for sharing your finds! At the moment we're using a workaround to resolve this on chrome ( When ads aren't playing anymore we move back 0.5s in time ). I will continue investigating this when I can. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 Deleted