New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 747050 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 3
Type: Bug



Sign in to add a comment

Support encrypted FLAC (in mp4 container) on Chrome for Android

Project Member Reported by wolenetz@chromium.org, Jul 20 2017

Issue description

Initial investigation indicates that MSE(flac-in-mp4 - see  bug 666000 ) + EME on Android would probably require further work to:
1) understand flac+MSE+EME use cases (decrypt-only, or decrypt-and-decode) - and if decrypt-only, would FFmpeg decode work? (I'm not an EME expert, so don't know), and
2) Possibly dynamically query the runtime MediaCodec for support for secure "audio/flac" decode if at least decrypt+decode is required.

Note that  bug 666000  should enable unencrypted MSE flac-in-mp4 decode on desktop+Android Chrome using FFmpeg.

For details, see https://chromium-review.googlesource.com/579662; description is copied here from that demo CL:

Though Android list of supported media formats [1] [2] includes FLAC
decoder (Android 3.1+) (with some channelcount, samplerate and
samplesize constraints), it explicitly mentions "FLAC (.flac) only" in
the "Supported File Type(s) / Container Formats" column.

This CL started as an optimistic approach to see if "audio/flac" is
supported by MediaCodec, to perhaps more easily and fully support a
possible use of MSE + EME "audio/flac" on Android in Chrome in the
future (plain src= .flac and upcoming support [3] for MSE flac-in-MP4 in
Chrome use FFmpeg, including on Android, but EME on Android may require
MediaCodec support for flac decode).  Unfortunately, it doesn't appear
to be uniformly supported, as this CL demonstrates.

Details:
On at least a Nexus 7 2013 with Marshmallow, flac is not listed
in /system/etc/media_codecs.xml (nor any of the files it includes) for
decode, just "OMX.google.flac.encoder" is the only mention of flac. On
this same device, attempt to construct a decoder for "audio/flac"
mimetype fails (from local test with this CL):
E ACodec  : Unable to instantiate a decoder for type 'audio/flac' with err 0xfffffffe.
E ACodec  : signalError(omxError 0xfffffffe, internalError -2)

Further investigation on this device with this CL demonstrates that
MediaExtractor, used on a .flac file, produces a MediaFormat with
"audio/raw" mimetype, implying flac decode has already been done IIUC:
D cr_MediaCodecUtil: [MediaCodecUtil.java:305] BIG TODO testfile: /sdcard/chromium_tests_root/media/test/data/sfx.flac
D cr_MediaCodecUtil: [MediaCodecUtil.java:311] BIG TODO track count 1
D cr_MediaCodecUtil: [MediaCodecUtil.java:315] BIG TODO mime audio/raw

Online reports in [4] appear to confirm that general flac codec decode
is not available uniformly on Android 3.1+ via MediaCodec; the only
requirement is that Android support .flac decode (which it appears to at
least on the test device using MediaExtractor). [4] also appears to
corroborate that MediaExtractor is doing FLAC decode work in some cases
where OEM doesn't include explicit FLAC decoder elsewhere, see [5]
("FLACExtractor wraps a C libFLAC parser aka stream decoder").

[1] https://developer.android.com/guide/topics/media/media-formats.html
[2] http://static.googleusercontent.com/media/source.android.com/en/us/compatibility/android-cdd.pdf
[3]  https://crbug.com/666000 
[4] https://bugzilla.mozilla.org/show_bug.cgi?format=default&id=1295886
[5] https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/FLACExtractor.cpp#71

 
xhwang@/dalecurtis@/hbengali@ please help triage this. Thanks!
(I'll continue to focus my current work on the FLAC MSE-in-mp4  bug 666000 .)
There's no easy way to query this from the renderer, so for now I'd just mark this as unsupported. I.e. return false for EME+flac on Android in mime_util_internal. Ideally by updating the PlatformInfo struct with this new variable.
Chatting with xhwang@, and looking at https://codereview.chromium.org/2866473004/ (and the current cdm message filter: https://cs.chromium.org/chromium/src/components/cdm/browser/cdm_message_filter_android.cc?q=cdm_message_filter_android.cc&sq=package:chromium&dr&l=57) indicates that flac-in-mp4 on android is already disabled. Unfortunately, encrypted_media_supported_types_browsertest is disabled for Android, so none of this is auto-tested. In my CL adding unencrypted '{audio,video}/mp4; codecs="flac"' canPlayType support as prereq for fixing  bug 666000 , I'll include a comment similar to the vorbis comment already in https://cs.chromium.org/chromium/src/components/cdm/browser/cdm_message_filter_android.cc?q=cdm_message_filter_android.cc&sq=package:chromium&dr&l=57.

Perhaps some refactoring to make it clear precisely where the determination of mime/key system/codec support per platform is done would help, too, but is out of scope of my MSE work in  bug 666000 .
Project Member

Comment 4 by bugdroid1@chromium.org, Jul 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726

commit b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726
Author: Matt Wolenetz <wolenetz@chromium.org>
Date: Fri Jul 21 18:00:35 2017

Enable src= support for FLAC-in-MP4

Since M56, all Chrome build types that include proprietary codecs and
use the unified media pipeline have been able to demux and decode
FLAC-in-MP4, but canPlayType() hasn't been updated yet.

Further motivation to update that, a prerequisite for supporting
FLAC-in-MP4 with MSE in Chrome is that canPlayType('{audio,video}/mp4,
"flac"') needs to return "probably".

This change updates canPlayType() to indicate FLAC-in-MP4 is supported,
and includes new tests for src= FLAC-in-MP4.

A further complication is that MP4's AudioSampleEntry samplerate field
is fixed point 16.16 bits, so lacks range to handle bitrates above
65535Hz. The FLAC-in-MP4 spec [1] guides how to encode the correct
sample rate when it is outside this scale, and this change includes test
files at both 44.1kHz and 192kHz to verify FFmpegDemuxer extracts the
correct samplerate information.

Also, based on investigation leading to https://crbug.com/747050#c3,
this change includes a comment indicating lack of decode support for
*encrypted* FLAC (in any container) on Android (which would need FLAC
decode support directly in MediaCodec, rather than in MediaExtractor as
some devices appear to do). In the future, perhaps more refined logic
can be used to identify cases where encrypted FLAC-in-MP4 on Android is
supported by the device.

BUG= 666000 ,747050
TEST=Added sfx-flac.mp4 FFmpeg AudioDecoderTest cases, basic media
playback browsertests for bear-flac.mp4 and bear-flac-192kHz.mp4,
FFmpegDemuxerTests, PipelineIntegrationPerfTests, and a
PipelineIntegrationTest; Updated MediaCanPlayTypeTests and
MimeUtilTest.ParseAudioCodecString;

Change-Id: I04df11c34aa539c53dd2c8868257f7ef9608ddc7
Reviewed-on: https://chromium-review.googlesource.com/580575
Reviewed-by: Xiaohan Wang <xhwang@chromium.org>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#488703}
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/components/cdm/browser/cdm_message_filter_android.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/content/browser/media/media_browsertest.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/content/browser/media/media_canplaytype_browsertest.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/base/mime_util_internal.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/base/mime_util_unittest.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/filters/audio_decoder_unittest.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/filters/ffmpeg_demuxer_unittest.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/README
[add] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/bear-flac-192kHz.mp4
[add] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/bear-flac.mp4
[add] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/sfx-flac.mp4
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/pipeline_integration_perftest.cc
[modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/pipeline_integration_test.cc

Owner: wolenetz@chromium.org
Status: Assigned (was: Untriaged)
matt, since you have a CL for this bug, I assigned it to you.
Owner: xhw...@chromium.org
w.r.t. this bug, that CL in #4 just added a comment pointing to this bug: https://chromium.googlesource.com/chromium/src/+/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/components/cdm/browser/cdm_message_filter_android.cc#58

=> xhwang@ (EME) for getting this support updated eventually (depends most likely on Android). Clear (non-EME) MSE and src= FLAC decode on Android is done with FFmpeg.

Xiaohan, send it back my way if you need :)
EME only supports WebM and MP4 (https://w3c.github.io/encrypted-media/format-registry/stream/#registry). I don't know if the FLAC container has mechanisms to signal encryption, which would be required to support "audio/flac" in EME.

Android DRM requires platform support for all codecs. It doesn't have to be a ".secure" codec, but it does need to be decoded in the OS.

@#7, to be clearer, I've landed ( bug 666000 ) MSE support for demux + decode/playback of FLAC-in-ISOBMFF(MP4). I filed this bug when I discovered that our Android pipeline (with the unified pipeline and ffmpeg FLAC decode) should work fine for FLAC-in-ISOBMFF in MSE), but that our EME pipeline's usage of MediaCodec + MediaDrm on Android would necessitate support for FLAC (codec, not container) decode in MediaCodec, which is not-uniformly supported. I wasn't promoting that .flac container needs EME support in Chrome (any platform, including Android).

Comment 9 by xhw...@chromium.org, Oct 10 2017

Status: WontFix (was: Assigned)
Summary: Support encrypted FLAC (in mp4 container) on Chrome for Android (was: Non-uniform MediaCodec support in Android 3.1+ for FLAC decode)
For audio, we don't need a secure decoder. So on Android versions/devices where FLAC codec is supported, we should be able to support EME+Flac (in theory). That being said, given there's no interest in encrpted FLAC, I'll probably just mark this as WontFix. We can reopen this issue if that assumption changes.
Status: Assigned (was: WontFix)

Sign in to add a comment