Hmm, probably we just need to update our ffmpeg copy. +wolenetz

 Issue 591845  landed the ffmpeg roll, but following that, I tried and this still repros. With debug logging enabled on a local build containing proprietary codecs and with ffmpeg_branding="ChromeOS", the file still does not play.

... ffmpeg_common.cc(119)] Unknown audio CodecID: 0
... ffmpeg_common.cc(289)] Unknown AVSampleFormat: -1
... webmediaplayer_impl.cc(1100)] OnError
... ERROR:render_media_log.cc(25)] MediaEvent: PIPELINE_ERROR demuxer: no supported streams

--> dalecurtis@ since the roll doesn't appear to include support for this, what's the plan and priority for this?

Oh, I misread this as vp9, I don't think ffmpeg has opus in mp4 support yet so neither do we. We won't have it until someone contributes it to ffmpeg.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available. If you change it back, also remove the "Hotlist-Recharge-Cold" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

dale, the original repro link is not available. Do we support opus now? if not, can you resolve this bug as won't fix?

Works in src= now, => wolenetz@ to make the call on adding support in MSE.

Please do! I might be biased, but I think that having Opus in MSE will simply adoption a lot.
If needed I can provide more samples, since the original in not available any more.

Hi,
Does this work with MSE?

Chrome does not yet support Opus in MP4 via MSE (it does via regular, non-MSE, src=, though):
  MediaSource.isTypeSupported('video/mp4; codecs="opus"') --> false

Further, src='s "canPlayType" indicates lack of support, even if the underlying non-MSE, src= implementation actually supports it:
  document.createElement('video').canPlayType('video/mp4; codecs="opus"') --> ""

+flim, do you know the current state of opus-in-mp4? The spec is still listed as incomplete.

wolenetz: Since we've ended up with av1 in mp4 it'd be good to add support in M70 along with changeType(). Do you have an indication of the amount of work needed here?

It doesn't look too bad from the spec, just add the boxes and parse relevant data from the new Opus box: http://opus-codec.org/docs/opus_in_isobmff.html#4.3.2

I'm happy to pick this up if flim@ thinks the spec is far enough along and you agree with adding it to MSE.

I'm uncertain how well the spec's "4.3.6.2 Pre-roll" integrates with existing Chrome MSE webm-opus. If it's compatible (and if the encoding of that information required in the Opus-in-MP4 spec doesn't violate the overall MSE ISO-BMFF restrictions (no absolute positions, must be fragmented, etc), then it seems like just the following would be needed:

0. Determine if this support needs to be gated by --enable-features=<something>. Also file intent-to-implement(and-ship). Consider go/newchromefeature, though the similar FLAC-in-ISO was only a simple launch IIRC). Also, file a chromestatus entry, etc (as needed by blink intent process).

1. Update StreamParserFactory (and canPlayType internals in mime_util IIRC). Until codec restriction is in place, add something like |has_opus| parameter to MP4StreamParser ctor because Opus-in-ISO has no standardized audio object type IIUC.

2. Parse the opus boxes in //media/formats/mp4. If not |has_opus|, but opus boxes are seen, give parse error.

3. Add browser tests (canPlayType and isTypeSupported)

4. Add test media for mediasource-changetype-play layout test (and add their metadata to the util.js)

5. If we previously had partial appendwindow trimming tests for webm-opus, add some for mp4-opus (e.g. ensure gapless mp4-opus scenario is supported if webm-opus is).

6. Add streamparser and pipeline integration unit tests for mse-mp4-opus

7. Add mse-mp4-opus to the pipeline_integration_fuzzertests and confirm expected code-coverage once the fuzzing starts.

8. I'm uncertain, but media capabilities browsertests/etc might also need updating for mp4-opus


For M-70, this probably won't fit on my plate. I'm uncertain about M-71, considering the other stuff I'm working on.

Tentatively assigning to you, per #13.

Per the ffmpeg implementation it's a 1:1 match with what's in the webm file, so seems like it is simply just adding support for the new box.

For 0, I don't think we need to ship this behind a flag or even go through the simple launch, if it's what I think it is; just dOps box processing. A short intent-to-implement+ship and chromestatus entry like you sent for ISO+MP4 sgtm simply for visibility to developers.

For 5, it doesn't look like we have such tests. 8 shouldn't be necessary.

Given that Firefox has already added support for this: https://bugzilla.mozilla.org/show_bug.cgi?id=1240413 I'll go ahead and put something together.

SGTM then.

Per https://cs.chromium.org/chromium/src/media/filters/chunk_demuxer.cc?rcl=46148d32d76702d301d09bf3a862c6d795f20685&l=271, Opus partial appendWindow trimming is not supported in Chrome MSE.

Be aware that the codec string for opus ("opus" in webm, at least), if interpreting MP4RA strictly, should be "Opus" in MSE ISO-BMFF.

But there's already precedent
1. (e.g. https://github.com/xiph/flac/issues/38#issuecomment-319665714) for using a different case for chars than were used in the FourCC.
2. Firefox uses "opus", not "Opus" for isTypeSupported and canPlayType codec parameter for audio/mp4.

tl;dr: "opus" is ok by me.

Dale, if this is really needed for M-70, I might be able to fit it in on my plate beginning next week. I'm OOO tomorrow (3-day weekend vacation). I2I&S is needed ASAP probably (M-70 feature freeze is tomorrow).

No it's fine I have room on my plate. FWIW, I'm having trouble getting it to work in Firefox, so I wonder if it's actually supported in MSE there.

I've pinged the Firefox bug for clarification. It's possible my ffmpeg is just generating invalid opus+mp4 files. If the Firefox implementation doesn't work though, then yeah it's probably too soon for M70.

I'm not worried about the feature freeze since this would fit the definition of a technical launch which doesn't need to follow the process.

Firefox indicated support broke at some point, but is fixed in nightly. I've verified the test links I've put in the chrome status page work there:

https://www.chromestatus.com/feature/5100845653819392

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d2b7c9f8bca3526cec8ba5e1de0b2bb3bba72144

commit d2b7c9f8bca3526cec8ba5e1de0b2bb3bba72144
Author: Dale Curtis <dalecurtis@chromium.org>
Date: Wed Aug 22 17:06:35 2018

Roll src/third_party/ffmpeg/ ddbe19a87..bfe62e18d (2 commits)

https://chromium.googlesource.com/chromium/third_party/ffmpeg.git/+log/ddbe19a87228..bfe62e18d24f

$ git log ddbe19a87..bfe62e18d --date=short --no-merges --format='%ad %ae %s'
2018-08-21 dalecurtis Update patches and README with ffmpeg roll hash.
2018-08-21 dalecurtis Correct opus-in-mp4 pre-skip to be uint16_t versus int16_t.

Created with:
  roll-dep src/third_party/ffmpeg

Bug:  649438 
Change-Id: I4467ea64338e7f136cd431acd923ae2aaa01048d
Tbr: tguilbert
Reviewed-on: https://chromium-review.googlesource.com/1183848
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Thomas Guilbert <tguilbert@chromium.org>
Commit-Queue: Dale Curtis <dalecurtis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#585105}
[modify] https://crrev.com/d2b7c9f8bca3526cec8ba5e1de0b2bb3bba72144/DEPS

Associating a couple of opus issues that exist already with webm + mse.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/88d5fa137b3569d0297719f7def757731fb37b87

commit 88d5fa137b3569d0297719f7def757731fb37b87
Author: Dale Curtis <dalecurtis@chromium.org>
Date: Fri Aug 24 06:11:42 2018

Implement support for Opus in MP4 using Media Source Extensions.

This adds everything necessary to support MP4+Opus. Specifically:
- The necessary box support such that the Opus and dOps boxes can
be parsed from an MP4 when using MSE.
- Updates to the mime type parsing for canPlayType/isTypeSupported
- Clones all the Opus in WebM tests for Opus in Mp4.

Things that need fixing:
- End-trimming is not correctly specified for opus-in-mp4 test data
yet and no tooling (nor Xiph folks) have correct test clips, this
is tracked by http://crbug.com/876544

Other things fixed:
- Some duplicate tests and media have been removed for Opus Webm.
- Moved opus_constants from media/filters to media/formats/common
- Removed unused constants/functions from opus_constants.
- MockMediaSource now allows SetAppendWindow so we can reuse the
(voluminous) opus audio hashes instead of generating new ones
while the test media is broken.

BUG= 649438 
TEST=new unittests

Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I8dd11f7d08616b551dae1bca14c3e44d5447c92c
Reviewed-on: https://chromium-review.googlesource.com/1178929
Commit-Queue: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#585720}
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/content/browser/media/media_browsertest.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/content/browser/media/media_canplaytype_browsertest.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/base/mime_util_internal.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/filters/BUILD.gn
[delete] https://crrev.com/936898b90312b023c56c5e56b20183488cde89dc/media/filters/opus_constants.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/filters/stream_parser_factory.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/BUILD.gn
[add] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/common/opus_constants.cc
[rename] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/common/opus_constants.h
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/mp4/box_definitions.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/mp4/box_definitions.h
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/mp4/fourccs.h
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/formats/mp4/mp4_stream_parser.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/muxers/BUILD.gn
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/muxers/webm_muxer.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/BUILD.gn
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/data/README
[delete] https://crrev.com/936898b90312b023c56c5e56b20183488cde89dc/media/test/data/bear-opus-end-trimming.webm
[add] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/data/bear-opus.mp4
[add] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/data/opus-trimming-test.mp4
[add] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/data/sfx-opus_frag.mp4
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/mock_media_source.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/mock_media_source.h
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/pipeline_integration_fuzzertest.cc
[modify] https://crrev.com/88d5fa137b3569d0297719f7def757731fb37b87/media/test/pipeline_integration_test.cc

cc +tinskip, I think Thomas worked on Opus in mp4 spec at some point.