Support encrypted FLAC (in mp4 container) on Chrome for Android |
|||||
Issue descriptionInitial investigation indicates that MSE(flac-in-mp4 - see bug 666000 ) + EME on Android would probably require further work to: 1) understand flac+MSE+EME use cases (decrypt-only, or decrypt-and-decode) - and if decrypt-only, would FFmpeg decode work? (I'm not an EME expert, so don't know), and 2) Possibly dynamically query the runtime MediaCodec for support for secure "audio/flac" decode if at least decrypt+decode is required. Note that bug 666000 should enable unencrypted MSE flac-in-mp4 decode on desktop+Android Chrome using FFmpeg. For details, see https://chromium-review.googlesource.com/579662; description is copied here from that demo CL: Though Android list of supported media formats [1] [2] includes FLAC decoder (Android 3.1+) (with some channelcount, samplerate and samplesize constraints), it explicitly mentions "FLAC (.flac) only" in the "Supported File Type(s) / Container Formats" column. This CL started as an optimistic approach to see if "audio/flac" is supported by MediaCodec, to perhaps more easily and fully support a possible use of MSE + EME "audio/flac" on Android in Chrome in the future (plain src= .flac and upcoming support [3] for MSE flac-in-MP4 in Chrome use FFmpeg, including on Android, but EME on Android may require MediaCodec support for flac decode). Unfortunately, it doesn't appear to be uniformly supported, as this CL demonstrates. Details: On at least a Nexus 7 2013 with Marshmallow, flac is not listed in /system/etc/media_codecs.xml (nor any of the files it includes) for decode, just "OMX.google.flac.encoder" is the only mention of flac. On this same device, attempt to construct a decoder for "audio/flac" mimetype fails (from local test with this CL): E ACodec : Unable to instantiate a decoder for type 'audio/flac' with err 0xfffffffe. E ACodec : signalError(omxError 0xfffffffe, internalError -2) Further investigation on this device with this CL demonstrates that MediaExtractor, used on a .flac file, produces a MediaFormat with "audio/raw" mimetype, implying flac decode has already been done IIUC: D cr_MediaCodecUtil: [MediaCodecUtil.java:305] BIG TODO testfile: /sdcard/chromium_tests_root/media/test/data/sfx.flac D cr_MediaCodecUtil: [MediaCodecUtil.java:311] BIG TODO track count 1 D cr_MediaCodecUtil: [MediaCodecUtil.java:315] BIG TODO mime audio/raw Online reports in [4] appear to confirm that general flac codec decode is not available uniformly on Android 3.1+ via MediaCodec; the only requirement is that Android support .flac decode (which it appears to at least on the test device using MediaExtractor). [4] also appears to corroborate that MediaExtractor is doing FLAC decode work in some cases where OEM doesn't include explicit FLAC decoder elsewhere, see [5] ("FLACExtractor wraps a C libFLAC parser aka stream decoder"). [1] https://developer.android.com/guide/topics/media/media-formats.html [2] http://static.googleusercontent.com/media/source.android.com/en/us/compatibility/android-cdd.pdf [3] https://crbug.com/666000 [4] https://bugzilla.mozilla.org/show_bug.cgi?format=default&id=1295886 [5] https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/FLACExtractor.cpp#71
,
Jul 20 2017
There's no easy way to query this from the renderer, so for now I'd just mark this as unsupported. I.e. return false for EME+flac on Android in mime_util_internal. Ideally by updating the PlatformInfo struct with this new variable.
,
Jul 20 2017
Chatting with xhwang@, and looking at https://codereview.chromium.org/2866473004/ (and the current cdm message filter: https://cs.chromium.org/chromium/src/components/cdm/browser/cdm_message_filter_android.cc?q=cdm_message_filter_android.cc&sq=package:chromium&dr&l=57) indicates that flac-in-mp4 on android is already disabled. Unfortunately, encrypted_media_supported_types_browsertest is disabled for Android, so none of this is auto-tested. In my CL adding unencrypted '{audio,video}/mp4; codecs="flac"' canPlayType support as prereq for fixing bug 666000 , I'll include a comment similar to the vorbis comment already in https://cs.chromium.org/chromium/src/components/cdm/browser/cdm_message_filter_android.cc?q=cdm_message_filter_android.cc&sq=package:chromium&dr&l=57. Perhaps some refactoring to make it clear precisely where the determination of mime/key system/codec support per platform is done would help, too, but is out of scope of my MSE work in bug 666000 .
,
Jul 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726 commit b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726 Author: Matt Wolenetz <wolenetz@chromium.org> Date: Fri Jul 21 18:00:35 2017 Enable src= support for FLAC-in-MP4 Since M56, all Chrome build types that include proprietary codecs and use the unified media pipeline have been able to demux and decode FLAC-in-MP4, but canPlayType() hasn't been updated yet. Further motivation to update that, a prerequisite for supporting FLAC-in-MP4 with MSE in Chrome is that canPlayType('{audio,video}/mp4, "flac"') needs to return "probably". This change updates canPlayType() to indicate FLAC-in-MP4 is supported, and includes new tests for src= FLAC-in-MP4. A further complication is that MP4's AudioSampleEntry samplerate field is fixed point 16.16 bits, so lacks range to handle bitrates above 65535Hz. The FLAC-in-MP4 spec [1] guides how to encode the correct sample rate when it is outside this scale, and this change includes test files at both 44.1kHz and 192kHz to verify FFmpegDemuxer extracts the correct samplerate information. Also, based on investigation leading to https://crbug.com/747050#c3, this change includes a comment indicating lack of decode support for *encrypted* FLAC (in any container) on Android (which would need FLAC decode support directly in MediaCodec, rather than in MediaExtractor as some devices appear to do). In the future, perhaps more refined logic can be used to identify cases where encrypted FLAC-in-MP4 on Android is supported by the device. BUG= 666000 ,747050 TEST=Added sfx-flac.mp4 FFmpeg AudioDecoderTest cases, basic media playback browsertests for bear-flac.mp4 and bear-flac-192kHz.mp4, FFmpegDemuxerTests, PipelineIntegrationPerfTests, and a PipelineIntegrationTest; Updated MediaCanPlayTypeTests and MimeUtilTest.ParseAudioCodecString; Change-Id: I04df11c34aa539c53dd2c8868257f7ef9608ddc7 Reviewed-on: https://chromium-review.googlesource.com/580575 Reviewed-by: Xiaohan Wang <xhwang@chromium.org> Reviewed-by: Dale Curtis <dalecurtis@chromium.org> Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org> Cr-Commit-Position: refs/heads/master@{#488703} [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/components/cdm/browser/cdm_message_filter_android.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/content/browser/media/media_browsertest.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/content/browser/media/media_canplaytype_browsertest.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/base/mime_util_internal.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/base/mime_util_unittest.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/filters/audio_decoder_unittest.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/filters/ffmpeg_demuxer_unittest.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/README [add] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/bear-flac-192kHz.mp4 [add] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/bear-flac.mp4 [add] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/data/sfx-flac.mp4 [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/pipeline_integration_perftest.cc [modify] https://crrev.com/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/media/test/pipeline_integration_test.cc
,
Jul 27 2017
matt, since you have a CL for this bug, I assigned it to you.
,
Jul 27 2017
w.r.t. this bug, that CL in #4 just added a comment pointing to this bug: https://chromium.googlesource.com/chromium/src/+/b6be2ef08a6cc5ae86dcc4a9cd2b3e3066353726/components/cdm/browser/cdm_message_filter_android.cc#58 => xhwang@ (EME) for getting this support updated eventually (depends most likely on Android). Clear (non-EME) MSE and src= FLAC decode on Android is done with FFmpeg. Xiaohan, send it back my way if you need :)
,
Aug 3 2017
EME only supports WebM and MP4 (https://w3c.github.io/encrypted-media/format-registry/stream/#registry). I don't know if the FLAC container has mechanisms to signal encryption, which would be required to support "audio/flac" in EME. Android DRM requires platform support for all codecs. It doesn't have to be a ".secure" codec, but it does need to be decoded in the OS.
,
Aug 3 2017
@#7, to be clearer, I've landed ( bug 666000 ) MSE support for demux + decode/playback of FLAC-in-ISOBMFF(MP4). I filed this bug when I discovered that our Android pipeline (with the unified pipeline and ffmpeg FLAC decode) should work fine for FLAC-in-ISOBMFF in MSE), but that our EME pipeline's usage of MediaCodec + MediaDrm on Android would necessitate support for FLAC (codec, not container) decode in MediaCodec, which is not-uniformly supported. I wasn't promoting that .flac container needs EME support in Chrome (any platform, including Android).
,
Oct 10 2017
For audio, we don't need a secure decoder. So on Android versions/devices where FLAC codec is supported, we should be able to support EME+Flac (in theory). That being said, given there's no interest in encrpted FLAC, I'll probably just mark this as WontFix. We can reopen this issue if that assumption changes.
,
Apr 24 2018
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by wolenetz@chromium.org
, Jul 20 2017