Issue 534301: HE-AAC v2 does not play back using MediaSource API, but works in Safari
Reported by
r...@mixlr.com,
Sep 21 2015
|
|||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0 Example URL: http://chrome-he-aac-v2-test.s3-eu-west-1.amazonaws.com/test.html Steps to reproduce the problem: 1. Visit the given URL 2. Watch while the proof-of-concept page downloads chunked HE-AAC v2 (generated using Wowza Streaming Engine's Apple HLS packetizer) What is the expected behavior? The stream is HE-AAC v2 which should work in Chrome, and the stream should play successfully. What went wrong? The MediaSource API fails with: pipeline: decode error kInitDemuxer Append: stream parsing failed. Data size=15634 append_window_start=0 append_window_end=inf Did this work before? N/A Is it a problem with Flash or HTML5? HTML5 Does this work in other browsers? Yes Chrome version: 45.0.2454.93 Channel: stable OS Version: OS X 10.10 Flash Version: 18,0,0,232 Note that this stream is audio-only, NOT video. The exact same page plays perfectly in Safari. I believe that Chrome should be able to decode and play HE-AAC v2 and that this is a bug in the MediaSource API. , Sep 22 2015So is that a special codec in addSourceBuffer() call? Any pointers would be appreciated if so. I've tried various including: 'audio/mp4; codecs="mp4a.40.29"': (and similar) give the behaviour above 'audio/aac': gives a different error "Implicit HE-AAC signalling is being used. Please use mp4a.40.5 instead of mp4a.40.2 in the mimetype." , Sep 22 2015The issue is that the decoder needs to be initialized with this info so that it can double the sample rate before it receives the first coded frame. You need to use: - 'audio/mp4; codecs="mp4a.40.5"' for HE-AAC - 'audio/mp4; codecs="mp4a.40.2"' for LC-AAC I agree this is a huge issue from the broadcaster's perspective. I've encountered a good proportion of Smooth Streaming streams that use implicit HE-AAC signaling. This issue more or less forces you to ignore the mimetype from the manifest and hardcode this info somewhere, which is a pain if you have both LC-AAC and implicit HE-AAC. Since Silverlight support in chrome has been removed, I've seen quite a lot of broadcasters moving to MSE playback, but not wanting to move to DASH quite yet, and thus keeping their Smooth Streaming streams. For these reasons my opinion is that the decoder should handle this use case and be able to adjust the sample rate if Implicit HE-AAC is encoutered. Sadly my guess is this is a non-trivial change. , Sep 22 2015Unfortunately initializing the SourceBuffer with the 'audio/mp4; codecs="mp4a.40.5"' codec results in the behaviour described above. Is there any other workaround? This would be incredibly useful for us. , Sep 22 2015
Whoops, actually +wolenetz this time. On Chrome canary, do you see any errors in the chrome://media-internals log for the created player? , Sep 29 2015I've got same problem with HE-AAC. IMO it's insane to require us to tell source buffer it's AAC or HE-AAC. How do you expect us to know that stuff in case of implicit signaling? Seriously, do it the way AAC spec tells you to handle it (up-sampling). Any idea when HE-AAC in MSE is expected to be fixed? , Sep 29 2015Or at least be compliant to aac spec and if SourceBuffer was configured as LC-AAC then drop all unknown bitstream extensions (wasn't that supposed to be done by the decoder already?!). At least it would work and not fail to decode. , Sep 30 2015
Looking at this now. Re:#5, I only see the already-mentioned error: "Append: stream parsing failed. Data size=15634 append_window_start=0 append_window_end=inf" Repro's on linux, expanding to OS-All , Sep 30 2015
, Sep 30 2015
Thanks for taking this one, Chris. At least an improved MEDIA_LOG about the decode error would be helpful ;) , Oct 1 2015
+acolwell
I've tracked this down, need some guidance on what action to take.
'audio/aac' is the right mime-type. Using 'audio/mp4' is wrong because this will cause Chrome to attempt parsing the aac file as an mp4 container. *But*, explicit signalling does not work with 'audio/aac'.
MediaSource.isTypeSupported('audio/mp4; codecs="mp4a.40.5"')
true
MediaSource.isTypeSupported('audio/aac; codecs="mp4a.40.5"')
false
IIUC, this is part of the spec. The AAC bytes stream registry[1] states 'The "codecs" MIME-type parameter must not be used with these MIME-types'. So the "Please use mp4a.40.5 ..." warning is not helpful for aac files.
For this to work in Chrome, we really have to permit implicit signalling. a...@streamroot.io guessed correctly in #3, this appears to not be a trivial change. See code-review discussion here [2]
So how tough is this change? Is it limited to ffmpeg, or do we also need to signal the audio device? Less familiar with this part. Minimally, I can put out a review to improve the advice for explicit signalling (something like: mp4-> use codec=, aac-> sorry, not supported yet). WDYT?
[1] https://w3c.github.io/media-source/mpeg-audio-byte-stream-format.html#mime-types
[2] https://chromiumcodereview.appspot.com/276503002/diff/1/media/filters/ffmpeg_audio_decoder.cc
, Oct 1 2015This is a non-trivial change because it requires changing the audio code to lazy initialize the audio device. In implicit signalling you don't know SBR is there until you have decoded the first AAC frame. All the container headers have values that look like non-SBR content so decoders that don't understand SBR just work. The SBR section is at the end of a normal AAC frame and there aren't any length fields to help you jump to the SBR part w/o completely understanding how to decode an AAC frame. You might want to double check the AAC spec, but I'm pretty sure it is non-trivial. To make this work, you basically have to defer audio device initialization until the place in the code where the "implicit signalling" message is logged OR figure out how to turn off SBR support in FFmpeg. I'm pretty sure broadcasters would like you to do the former instead of the later. :) I suppose you could also allow the codecs mimetype parameter, but that doesn't conform to the mimetype definition for audio/aac and I'm not a fan of allowing/encouraging people to use non-standard mimetypes for something like this. , Oct 1 2015I'm going to take a stab at enabling the implicit signalling (the non-trivial change). Stand by for prototype... , Nov 9 2015This seems related to one of the items in http://www.google.com/url?q=http%3A%2F%2Fengineering.dailymotion.com%2Fintroducing-hls-js%2F&sa=D&sntz=1&usg=AFQjCNE_-GmMfhWWJSgxNhzwn_T1trfyJw : "[HE-AAC vs AAC] audio switching is not straightforward to handle with MediaSource. Some browsers are incapable of switching properly: this raises a video error and playback freezes. In this case the only way to recover is to tear down and re-instantiate MediaSource... and this can result in a small audio/video glitch." , Nov 24 2015
, Dec 4 2015
Issue 563197 has been merged into this issue. , Feb 5 2016Finally found the reason for my troubles! Is there intention to fix this, and if so, is there a timeline? Would we be able to help speed this up by providing additional or different test pages? , Feb 5 2016Sorry for the delay, I've got some higher priority p2's keeping me busy lately. I intend to at least prototype a fix sometime this quarter... so between now and the end of March. Providing extra test pages wont speed up the timeline, but it will ensure that your page is tested when the fix is being developed. More tests are welcome :) , Feb 17 2016Issue 586908 has been merged into this issue. , Mar 4 2016We have had several instances of complaints that playback is not working on Chrome with HE-AAC. It seems like this issue only happens in Chrome - Edge, IE, Firefox are work. Is there an update on this issue? , Mar 4 2016Status is unchanged from comment 18. Will start on this soon. , Mar 7 2016Thanks Chris that's great to hear! We are preparing to move around 1.6bn/month video streams from Flash to HTML5 and this is a requirement before we can go live. If you'd like test streams or example use-cases we'd love to help. I can even set you up with a test harness showing our usage of MSE with HE-AAC and HE-AACv2 if that can help at all. , Mar 10 2016I would love some sample streams! At the moment I just have the one test at the top of the bug. , Mar 11 2016Glad to help, //playertest.longtailvideo.com/adaptive/id3/playlist.m3u8 //playertest.longtailvideo.com/adaptive/440k/playlist.m3u8 //playertest.longtailvideo.com/adaptive/pbs/playlist.m3u8 //playlists.ihrhls.com/c5/1469/playlist.m3u8?listenerId=ABC1234567890 We can also set you up with a beta of our implementation/usage of MSE if you'd like me to email you with access. , Mar 14 2016Definitely interested. Please email me access :) , Mar 18 2016donato, I don't see an email from you yet? Also, some notes on these links. For me to use these in chrome (no HLS) I've dropped your URLs into the HLS.js demo here: http://dailymotion.github.io/hls.js/demo/ I'm finding that the first link doesn't play at all in the HLS.js player. It works in Safari... not seeing any errors in Chrome's media logs. No clue what their player is unhappy about. The next two links do play, but they each work in today's chrome (don't seem to have any implicit AAC profile changes). The final link doesn't work at all. Let me know if I'm not using these in the manner you expected. , Mar 20 2016In regards to the HLS.JS library, as I understand it, they are working around this bug by using a single MediaSource for both audio and video. This solution is not ideal because it requires an extra read/write - equal in size to the entire media file; as well as extra complexity to avoid ridiculous garbage collection. If one use's two MediaSource's, one for audio and one for video, then Chromium will fail and throw a decode error. I believe this is the intended use of MediaSource, and will have a measurable impact on end-users experience. , Mar 21 2016So is this issue only with he-AAC or is it with all AAC codecs? , Mar 21 2016Hmm, do you create an audio and video tag when using two media source objects? If so you will lose audio-video sync capabilities. , Mar 21 2016@jared - this is only HE-AAC and not AAC @dale - Sorry I misspoke, our strategy is to use a single MediaSource with multiple sourcebuffers. This is analagous to the Google Shaka project which does the same thing here : https://github.com/google/shaka-player/blob/master/lib/media/media_source_engine.js#L235 , Mar 21 2016Re: 29, I haven't dug into it too deeply yet, but it seems HLS.js is inspecting the AAC profiles before appending to a source buffer and it makes decisions based on known support limitations [0]. This should not be necessary once I fix this bug :). Re: 30, The issue is not so much with a specific profile - the root problem is streams that switch between profiles (LC -> HE or vice versa) without an explicit init segment describing the new configuration. If your stream is entirely one profile (and its init segment describes it correctly), you should be fine as-is. [0] https://github.com/videojs/videojs-contrib-hls/blob/master/src/videojs-contrib-hls.js#L548 , Mar 21 2016Re: 33, This is true, but only when the AAC is boxed into an mp4 and used in an audio/mp4 sourcebuffer. This means an extra level of computation putting it into a box, and another for the browser to take it out. From the view of a developer, my pain point isn't sending additional init segments when the audio profile changes. That's pretty workable. The real pain point is that we can't use an audio/aac source buffer for HE-AAC audio. , Mar 23 2016
Issue 596683 has been merged into this issue. , Mar 28 2016Project MemberThe following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8a400694209ce5d56e013673f8c9e8a2b0a1017c commit 8a400694209ce5d56e013673f8c9e8a2b0a1017c Author: chcunningham <chcunningham@chromium.org> Date: Mon Mar 28 04:13:21 2016 Enable implicit signalling for HE AAC v1 & v2 in ADTS. This allows MSE to playback HE AAC v1 (Spectral Band Replication) and v2 (Parametric Stereo) in the ADTS (.acc) container. ADTS is not capable of explicitly signaling these profiles, so the frame's metadata will claim the stream is an earlier profile (e.g AAC LC) and the decoder will later tell us its actually HE AAC. See For more background, see these sections of 2009 ISO-14496-3 1.6.5.2 Implicit and explicit signaling of SBR 1.6.6.2 Implicit and Explicit Signaling of Parametric Stereo TEST=new media unit tests and all manual tests in bug report BUG= 534301 , 177872 Review URL: https://codereview.chromium.org/1805013003 Cr-Commit-Position: refs/heads/master@{#383478} [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/content/browser/renderer_host/render_process_host_impl.cc [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/base/audio_buffer.cc [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/base/audio_buffer.h [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/filters/ffmpeg_audio_decoder.cc [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/renderers/audio_renderer_impl.cc [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/renderers/audio_renderer_impl.h [add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-implicit-he-aac-v1.aac [add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-implicit-he-aac-v2.aac [add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-lc-aac.aac [add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-main-aac.aac [modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/pipeline_integration_test.cc , Mar 29 2016
Retested the above issue on All-OS with chrome version '51.0.2693.2'(383600) flash version - 21.0.0.204 and still the playback fails on "http://chrome-he-aac-v2-test.s3-eu-west-1.amazonaws.com/test.html". @chcunningham: Hey, would you mind checking the same ? Appreciate your help. Attach is the print screen for the same. , Mar 29 2016
Tested across 13 streams on Canary, and works flawlessly for each.
I will continue testing more streams today.
ashejole@ - that AAC file works through our players MSE implementation. The error in yours is most likely that the source buffer should be
mediaSource.addSourceBuffer('audio/aac');
instead of audio/mp4.
, Mar 29 2016
donato@ is right. See my comment 11. I've remade that test page with the mime type audio/aac here: http://storage.googleapis.com/chcunningham-chrome-shared/534301/aac_test.html The above test page works in canary, 51.0.2693.0 Requesting merge back to M50. , Mar 29 2016
Your change meets the bar and is auto-approved for M50 (branch: 2661) , Mar 29 2016I can also confirm that our previous failing playback with MSE and HE-AAC is now working on Canary. Thanks! , Mar 29 2016Project Member
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6 commit 8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6 Author: Chris Cunningham <chcunningham@chromium.org> Date: Tue Mar 29 21:26:22 2016 [TO M50] Enable implicit signalling for HE AAC v1 & v2 in ADTS. This allows MSE to playback HE AAC v1 (Spectral Band Replication) and v2 (Parametric Stereo) in the ADTS (.acc) container. ADTS is not capable of explicitly signaling these profiles, so the frame's metadata will claim the stream is an earlier profile (e.g AAC LC) and the decoder will later tell us its actually HE AAC. See For more background, see these sections of 2009 ISO-14496-3 1.6.5.2 Implicit and explicit signaling of SBR 1.6.6.2 Implicit and Explicit Signaling of Parametric Stereo TEST=new media unit tests and all manual tests in bug report BUG= 534301 , 177872 Review URL: https://codereview.chromium.org/1805013003 Cr-Commit-Position: refs/heads/master@{#383478} (cherry picked from commit 8a400694209ce5d56e013673f8c9e8a2b0a1017c) TBR=dalecurtis@chromium.org, ccameron@chromium.org Review URL: https://codereview.chromium.org/1838203003 . Cr-Commit-Position: refs/branch-heads/2661@{#424} Cr-Branched-From: ef6f6ae5e4c96622286b563658d5cd62a6cf1197-refs/heads/master@{#378081} [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/content/browser/renderer_host/render_process_host_impl.cc [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/base/audio_buffer.cc [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/base/audio_buffer.h [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/filters/ffmpeg_audio_decoder.cc [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/renderers/audio_renderer_impl.cc [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/renderers/audio_renderer_impl.h [add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-implicit-he-aac-v1.aac [add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-implicit-he-aac-v2.aac [add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-lc-aac.aac [add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-main-aac.aac [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/pipeline_integration_test.cc [modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/pipeline_integration_test_base.h , Mar 30 2016
, Apr 12 2016Issue 602731 has been merged into this issue. , Apr 26 2016
Verified the issue on Win 7,Mac 10.11.4 and Ubuntu 14.04 using 52.0.2717.0 and 50.2661.87 and its working fine. URL used : http://storage.googleapis.com/chcunningham-chrome-shared/534301/aac_test.html Hence added the respective labels for the same.
|
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by dalecur...@chromium.org, Sep 21 2015