New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 16 users

Issue metadata

Status: Verified
Owner:
Closed: Mar 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug



Sign in to add a comment
link

Issue 534301: HE-AAC v2 does not play back using MediaSource API, but works in Safari

Reported by r...@mixlr.com, Sep 21 2015

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0

Example URL:
http://chrome-he-aac-v2-test.s3-eu-west-1.amazonaws.com/test.html

Steps to reproduce the problem:
1. Visit the given URL
2. Watch while the proof-of-concept page downloads chunked HE-AAC v2 (generated using Wowza Streaming Engine's Apple HLS packetizer)

What is the expected behavior?
The stream is HE-AAC v2 which should work in Chrome, and the stream should play successfully.

What went wrong?
The MediaSource API fails with:

pipeline: decode error
kInitDemuxer
Append: stream parsing failed. Data size=15634 append_window_start=0 append_window_end=inf

Did this work before? N/A 

Is it a problem with Flash or HTML5? HTML5

Does this work in other browsers? Yes 

Chrome version: 45.0.2454.93  Channel: stable
OS Version: OS X 10.10
Flash Version: 18,0,0,232

Note that this stream is audio-only, NOT video.

The exact same page plays perfectly in Safari.

I believe that Chrome should be able to decode and play HE-AAC v2 and that this is a bug in the MediaSource API.
 

Comment 1 by dalecur...@chromium.org, Sep 21 2015

Labels: -Cr-Internals-Media Cr-Internals-Media-Source
Should work, but needs a special sourcebuffer id IIRC. +wolenetz

Comment 2 by r...@mixlr.com, Sep 22 2015

So is that a special codec in addSourceBuffer() call?

Any pointers would be appreciated if so. I've tried various including:

'audio/mp4; codecs="mp4a.40.29"': (and similar) give the behaviour above

'audio/aac': gives a different error "Implicit HE-AAC signalling is being used. Please use mp4a.40.5 instead of mp4a.40.2 in the mimetype."

Comment 3 by a...@streamroot.io, Sep 22 2015

The issue is that the decoder needs to be initialized with this info so that it can double the sample rate before it receives the first coded frame.

You need to use:
- 'audio/mp4; codecs="mp4a.40.5"' for HE-AAC
- 'audio/mp4; codecs="mp4a.40.2"' for LC-AAC

I agree this is a huge issue from the broadcaster's perspective. I've encountered a good proportion of Smooth Streaming streams that use implicit HE-AAC signaling. This issue more or less forces you to ignore the mimetype from the manifest and hardcode this info somewhere, which is a pain if you have both LC-AAC and implicit HE-AAC.

Since Silverlight support in chrome has been removed, I've seen quite a lot of broadcasters moving to MSE playback, but not wanting to move to DASH quite yet, and thus keeping their Smooth Streaming streams.

For these reasons my opinion is that the decoder should handle this use case and be able to adjust the sample rate if Implicit HE-AAC is encoutered. Sadly my guess is  this is a non-trivial change.

Comment 4 by r...@mixlr.com, Sep 22 2015

Unfortunately initializing the SourceBuffer with the 'audio/mp4; codecs="mp4a.40.5"' codec results in the behaviour described above.

Is there any other workaround? This would be incredibly useful for us.

Comment 5 by dalecur...@chromium.org, Sep 22 2015

Cc: wolenetz@chromium.org chcunningham@chromium.org
Whoops, actually +wolenetz this time.  On Chrome canary, do you see any errors in the chrome://media-internals log for the created player?

Comment 6 by pavelp.h...@gmail.com, Sep 29 2015

I've got same problem with HE-AAC. IMO it's insane to require us to tell source buffer it's AAC or HE-AAC. How do you expect us to know that stuff in case of implicit signaling? Seriously, do it the way AAC spec tells you to handle it (up-sampling).
Any idea when HE-AAC in MSE is expected to be fixed?

Comment 7 by pavelp.h...@gmail.com, Sep 29 2015

Or at least be compliant to aac spec and if SourceBuffer was configured as LC-AAC then drop all unknown bitstream extensions (wasn't that supposed to be done by the decoder already?!).
At least it would work and not fail to decode.

Comment 8 by chcunningham@chromium.org, Sep 30 2015

Labels: -OS-Mac OS-All
Status: Assigned
Looking at this now. Re:#5, I only see the already-mentioned error: "Append: stream parsing failed. Data size=15634 append_window_start=0 append_window_end=inf"

Repro's on linux, expanding to OS-All

Comment 9 by chcunningham@chromium.org, Sep 30 2015

Owner: chcunningham@chromium.org

Comment 10 by wolenetz@chromium.org, Sep 30 2015

Labels: MSEscrubbedM47
Thanks for taking this one, Chris. At least an improved MEDIA_LOG about the decode error would be helpful ;)

Comment 11 by chcunningham@chromium.org, Oct 1 2015

Cc: acolwell@chromium.org
+acolwell

I've tracked this down, need some guidance on what action to take.

'audio/aac' is the right mime-type. Using 'audio/mp4' is wrong because this will cause Chrome to attempt parsing the aac file as an mp4 container. *But*, explicit signalling does not work with 'audio/aac'.

MediaSource.isTypeSupported('audio/mp4; codecs="mp4a.40.5"')
true
MediaSource.isTypeSupported('audio/aac; codecs="mp4a.40.5"')
false

IIUC, this is part of the spec. The AAC bytes stream registry[1] states 'The "codecs" MIME-type parameter must not be used with these MIME-types'. So the "Please use mp4a.40.5 ..." warning is not helpful for aac files.

For this to work in Chrome, we really have to permit implicit signalling. a...@streamroot.io guessed correctly in #3, this appears to not be a trivial change. See code-review discussion here [2]

So how tough is this change? Is it limited to ffmpeg, or do we also need to signal the audio device? Less familiar with this part. Minimally, I can put out a review to improve the advice for explicit signalling (something like: mp4-> use codec=, aac-> sorry, not supported yet). WDYT?

[1] https://w3c.github.io/media-source/mpeg-audio-byte-stream-format.html#mime-types
[2] https://chromiumcodereview.appspot.com/276503002/diff/1/media/filters/ffmpeg_audio_decoder.cc

Comment 12 by acolwell@chromium.org, Oct 1 2015

This is a non-trivial change because it requires changing the audio code to lazy initialize the audio device. In implicit signalling you don't know SBR is there until you have decoded the first AAC frame. All the container headers have values that look like non-SBR content so decoders that don't understand SBR just work. The SBR section is at the end of a normal AAC frame and there aren't any length fields to help you jump to the SBR part w/o completely understanding how to decode an AAC frame. You might want to double check the AAC spec, but I'm pretty sure it is non-trivial.

To make this work, you basically have to defer audio device initialization until the place in the code where the "implicit signalling" message is logged OR figure out how to turn off SBR support in FFmpeg. I'm pretty sure broadcasters would like you to do the former instead of the later. :) I suppose you could also allow the codecs mimetype parameter, but that doesn't conform to the mimetype definition for audio/aac and I'm not a fan of allowing/encouraging people to use non-standard mimetypes for something like this.

Comment 13 by chcunningham@chromium.org, Oct 1 2015

I'm going to take a stab at enabling the implicit signalling (the non-trivial change). Stand by for prototype...

Comment 14 by wolenetz@chromium.org, Nov 9 2015

This seems related to one of the items in http://www.google.com/url?q=http%3A%2F%2Fengineering.dailymotion.com%2Fintroducing-hls-js%2F&sa=D&sntz=1&usg=AFQjCNE_-GmMfhWWJSgxNhzwn_T1trfyJw :
"[HE-AAC vs AAC] audio switching is not straightforward to handle with MediaSource. Some browsers are incapable of switching properly: this raises a video error and playback freezes. In this case the only way to recover is to tear down and re-instantiate MediaSource... and this can result in a small audio/video glitch."

Comment 15 by wolenetz@chromium.org, Nov 24 2015

Labels: MSEscrubbed

Comment 16 by chcunningham@chromium.org, Dec 4 2015

Cc: tkonch...@chromium.org
 Issue 563197  has been merged into this issue.

Comment 17 by don...@jwplayer.com, Feb 5 2016

Finally found the reason for my troubles!

Is there intention to fix this, and if so, is there a timeline?
Would we be able to help speed this up by providing additional or different test pages?

Comment 18 by chcunningham@chromium.org, Feb 5 2016

Sorry for the delay, I've got some higher priority p2's keeping me busy lately. I intend to at least prototype a fix sometime this quarter... so between now and the end of March. 

Providing extra test pages wont speed up the timeline, but it will ensure that your page is tested when the fix is being developed. More tests are welcome :)

Comment 19 by dalecur...@chromium.org, Feb 17 2016

 Issue 586908  has been merged into this issue.

Comment 20 by am...@microsoft.com, Mar 4 2016

We have had several instances of complaints that playback is not working on Chrome with HE-AAC.  It seems like this issue only happens in Chrome - Edge, IE, Firefox are work.  Is there an update on this issue?

Comment 21 by chcunningham@chromium.org, Mar 4 2016

Status is unchanged from comment 18. Will start on this soon.

Comment 22 by don...@jwplayer.com, Mar 7 2016

Thanks Chris that's great to hear!

We are preparing to move around 1.6bn/month video streams from Flash to HTML5 and this is a requirement before we can go live.

If you'd like test streams or example use-cases we'd love to help. I can even set you up with a test harness showing our usage of MSE with HE-AAC and HE-AACv2 if that can help at all.

Comment 23 by chcunningham@chromium.org, Mar 10 2016

I would love some sample streams! At the moment I just have the one test at the top of the bug.

Comment 24 by don...@jwplayer.com, Mar 11 2016

Glad to help,

//playertest.longtailvideo.com/adaptive/id3/playlist.m3u8
//playertest.longtailvideo.com/adaptive/440k/playlist.m3u8
//playertest.longtailvideo.com/adaptive/pbs/playlist.m3u8
//playlists.ihrhls.com/c5/1469/playlist.m3u8?listenerId=ABC1234567890

We can also set you up with a beta of our implementation/usage of MSE if you'd like me to email you with access.

Comment 25 by chcunningham@chromium.org, Mar 14 2016

Definitely interested. Please email me access :)

Comment 26 by chcunningham@chromium.org, Mar 18 2016

donato, I don't see an email from you yet? 

Also, some notes on these links. For me to use these in chrome (no HLS) I've dropped your URLs into the HLS.js demo here:
http://dailymotion.github.io/hls.js/demo/

I'm finding that the first link doesn't play at all in the HLS.js player. It works in Safari... not seeing any errors in Chrome's media logs. No clue what their player is unhappy about. 

The next two links do play, but they each work in today's chrome (don't seem to have any implicit AAC profile changes). 

The final link doesn't work at all. Let me know if I'm not using these in the manner you expected.

Comment 27 Deleted

Comment 28 Deleted

Comment 29 by don...@jwplayer.com, Mar 20 2016

In regards to the HLS.JS library, as I understand it, they are working around this bug by using a single MediaSource for both audio and video. This solution is not ideal because it requires an extra read/write - equal in size to the entire media file; as well as extra complexity to avoid ridiculous garbage collection.

If one use's two MediaSource's, one for audio and one for video, then Chromium will fail and throw a decode error. I believe this is the intended use of MediaSource, and will have a measurable impact on end-users experience.

Comment 30 by jaredrho...@gmail.com, Mar 21 2016

So is this issue only with he-AAC or is it with all AAC codecs?

Comment 31 by dalecur...@chromium.org, Mar 21 2016

Hmm, do you create an audio and video tag when using two media source objects? If so you will lose audio-video sync capabilities.

Comment 32 by don...@jwplayer.com, Mar 21 2016

@jared - this is only HE-AAC and not AAC

@dale - Sorry I misspoke, our strategy is to use a single MediaSource with multiple sourcebuffers. This is analagous to the Google Shaka project which does the same thing here : https://github.com/google/shaka-player/blob/master/lib/media/media_source_engine.js#L235

Comment 33 by chcunningham@chromium.org, Mar 21 2016

Re: 29, I haven't dug into it too deeply yet, but it seems HLS.js is inspecting the AAC profiles before appending to a source buffer and it makes decisions based on known support limitations [0]. This should not be necessary once I fix this bug :). 

Re: 30, The issue is not so much with a specific profile - the root problem is streams that switch between profiles (LC -> HE or vice versa) without an explicit init segment describing the new configuration. If your stream is entirely one profile (and its init segment describes it correctly), you should be fine as-is. 

[0] https://github.com/videojs/videojs-contrib-hls/blob/master/src/videojs-contrib-hls.js#L548

Comment 34 by don...@jwplayer.com, Mar 21 2016

Re: 33, This is true, but only when the AAC is boxed into an mp4 and used in an audio/mp4 sourcebuffer. This means an extra level of computation putting it into a box, and another for the browser to take it out.

From the view of a developer, my pain point isn't sending additional init segments when the audio profile changes. That's pretty workable.

The real pain point is that we can't use an audio/aac source buffer for HE-AAC audio.

Comment 35 by tsunghung@chromium.org, Mar 23 2016

Cc: qin...@chromium.org tsunghung@chromium.org xhw...@chromium.org ti...@chromium.org
 Issue 596683  has been merged into this issue.

Comment 36 by bugdroid1@chromium.org, Mar 28 2016

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8a400694209ce5d56e013673f8c9e8a2b0a1017c

commit 8a400694209ce5d56e013673f8c9e8a2b0a1017c
Author: chcunningham <chcunningham@chromium.org>
Date: Mon Mar 28 04:13:21 2016

Enable implicit signalling for HE AAC v1 & v2 in ADTS.

This allows MSE to playback HE AAC v1 (Spectral Band Replication) and
v2 (Parametric Stereo) in the ADTS (.acc) container. ADTS is not
capable of explicitly signaling these profiles, so the frame's
metadata will claim the stream is an earlier profile (e.g AAC LC) and
the decoder will later tell us its actually HE AAC. See

For more background, see these sections of 2009 ISO-14496-3
1.6.5.2 Implicit and explicit signaling of SBR
1.6.6.2 Implicit and Explicit Signaling of Parametric Stereo

TEST=new media unit tests and all manual tests in bug report
BUG= 534301 ,  177872 

Review URL: https://codereview.chromium.org/1805013003

Cr-Commit-Position: refs/heads/master@{#383478}

[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/content/browser/renderer_host/render_process_host_impl.cc
[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/base/audio_buffer.cc
[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/base/audio_buffer.h
[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/filters/ffmpeg_audio_decoder.cc
[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/renderers/audio_renderer_impl.cc
[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/renderers/audio_renderer_impl.h
[add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-implicit-he-aac-v1.aac
[add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-implicit-he-aac-v2.aac
[add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-lc-aac.aac
[add] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/data/bear-audio-main-aac.aac
[modify] https://crrev.com/8a400694209ce5d56e013673f8c9e8a2b0a1017c/media/test/pipeline_integration_test.cc

Comment 37 by ashej...@chromium.org, Mar 29 2016

Cc: ashej...@chromium.org
Labels: Needs-Feedback
Retested the above issue on All-OS with chrome version '51.0.2693.2'(383600)  flash version -   21.0.0.204 and still the playback fails on "http://chrome-he-aac-v2-test.s3-eu-west-1.amazonaws.com/test.html".

@chcunningham: Hey, would you mind checking the same ?

Appreciate your help.

Attach is the print screen for the same.
Screen Shot 2016-03-29 at 3.52.42 PM.png
195 KB View Download

Comment 38 by don...@jwplayer.com, Mar 29 2016

Tested across 13 streams on Canary, and works flawlessly for each.

I will continue testing more streams today.

ashejole@ - that AAC file works through our players MSE implementation. The error in yours is most likely that the source buffer should be
    mediaSource.addSourceBuffer('audio/aac'); 
instead of audio/mp4.

Comment 39 by chcunningham@chromium.org, Mar 29 2016

Labels: Merge-Request-50
Status: Verified (was: Assigned)
donato@ is right. See my comment 11.

I've remade that test page with the mime type audio/aac here:
http://storage.googleapis.com/chcunningham-chrome-shared/534301/aac_test.html

The above test page works in canary, 51.0.2693.0


Requesting merge back to M50.

Comment 40 by tin...@google.com, Mar 29 2016

Labels: -Merge-Request-50 Merge-Approved-50 Hotlist-Merge-Approved
Your change meets the bar and is auto-approved for M50 (branch: 2661)

Comment 41 by am...@microsoft.com, Mar 29 2016

I can also confirm that our previous failing playback with MSE and HE-AAC is now working on Canary. Thanks!

Comment 42 by bugdroid1@chromium.org, Mar 29 2016

Project Member
Labels: -merge-approved-50 merge-merged-2661
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6

commit 8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6
Author: Chris Cunningham <chcunningham@chromium.org>
Date: Tue Mar 29 21:26:22 2016

[TO M50] Enable implicit signalling for HE AAC v1 & v2 in ADTS.

This allows MSE to playback HE AAC v1 (Spectral Band Replication) and
v2 (Parametric Stereo) in the ADTS (.acc) container. ADTS is not
capable of explicitly signaling these profiles, so the frame's
metadata will claim the stream is an earlier profile (e.g AAC LC) and
the decoder will later tell us its actually HE AAC. See

For more background, see these sections of 2009 ISO-14496-3
1.6.5.2 Implicit and explicit signaling of SBR
1.6.6.2 Implicit and Explicit Signaling of Parametric Stereo

TEST=new media unit tests and all manual tests in bug report
BUG= 534301 ,  177872 

Review URL: https://codereview.chromium.org/1805013003

Cr-Commit-Position: refs/heads/master@{#383478}
(cherry picked from commit 8a400694209ce5d56e013673f8c9e8a2b0a1017c)

TBR=dalecurtis@chromium.org, ccameron@chromium.org

Review URL: https://codereview.chromium.org/1838203003 .

Cr-Commit-Position: refs/branch-heads/2661@{#424}
Cr-Branched-From: ef6f6ae5e4c96622286b563658d5cd62a6cf1197-refs/heads/master@{#378081}

[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/content/browser/renderer_host/render_process_host_impl.cc
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/base/audio_buffer.cc
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/base/audio_buffer.h
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/filters/ffmpeg_audio_decoder.cc
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/renderers/audio_renderer_impl.cc
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/renderers/audio_renderer_impl.h
[add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-implicit-he-aac-v1.aac
[add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-implicit-he-aac-v2.aac
[add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-lc-aac.aac
[add] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/data/bear-audio-main-aac.aac
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/pipeline_integration_test.cc
[modify] https://crrev.com/8c2780f4c3ffb9bcae756e9d8e761bd85bf9c4b6/media/test/pipeline_integration_test_base.h

Comment 43 by ashej...@chromium.org, Mar 30 2016

Labels: -Needs-Feedback

Comment 44 by dalecur...@chromium.org, Apr 12 2016

 Issue 602731  has been merged into this issue.

Comment 45 by durga.behera@chromium.org, Apr 26 2016

Labels: TE-Verified-M50 TE-Verified-M52 TE-Verified-50.0.2661.87 TE-Verified-52.0.2717.0
Verified the issue on Win 7,Mac 10.11.4 and Ubuntu 14.04 using 52.0.2717.0 and 50.2661.87 and its working fine.
URL used : http://storage.googleapis.com/chcunningham-chrome-shared/534301/aac_test.html
Hence added the respective labels for the same.
534301_April_26.mp4
795 KB Download

Sign in to add a comment