The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/92607542c7cfde619e78d1b29490b1e1a13bb2da

commit 92607542c7cfde619e78d1b29490b1e1a13bb2da
Author: xhwang <xhwang@chromium.org>
Date: Fri Apr 28 01:44:33 2017

media: Refactor VideoFrameStreamTest

This is in preparation for updating/adding tests for the change to support
run-time switch of decoders.

- Move decoders to be selected to a vector to avoid duplicate code.
- Add |decoder_| to track the currently selected decoder.
- Remove DECODER_INIT pending state since this is rarely used.
- Allow FakeVideoDecoder to use different display names. This is needed
  to check what decoder has been selected.

BUG= 695595 
TEST=Refactor only. No changes to existing tests.

Review-Url: https://codereview.chromium.org/2835203006
Cr-Commit-Position: refs/heads/master@{#467809}

[modify] https://crrev.com/92607542c7cfde619e78d1b29490b1e1a13bb2da/media/filters/decoder_stream.cc
[modify] https://crrev.com/92607542c7cfde619e78d1b29490b1e1a13bb2da/media/filters/decoder_stream.h
[modify] https://crrev.com/92607542c7cfde619e78d1b29490b1e1a13bb2da/media/filters/fake_video_decoder.cc
[modify] https://crrev.com/92607542c7cfde619e78d1b29490b1e1a13bb2da/media/filters/fake_video_decoder.h
[modify] https://crrev.com/92607542c7cfde619e78d1b29490b1e1a13bb2da/media/filters/fake_video_decoder_unittest.cc
[modify] https://crrev.com/92607542c7cfde619e78d1b29490b1e1a13bb2da/media/filters/video_frame_stream_unittest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1492be9e0d583b0110f22d7e5f2ded139611f8bd

commit 1492be9e0d583b0110f22d7e5f2ded139611f8bd
Author: xhwang <xhwang@chromium.org>
Date: Fri May 05 03:01:21 2017

media: Support better decoder switching

Today on reinitialization error, or decode error of the first buffer,
DecoderStream will fall back to decoders left in the DecoderSelector.
Since we will have less and less decoders in the DecoderSelector, the
ability to switch decoders is limited.

This CL updates DecoderSelector so that it takes a callback to create
a list of decoders, instead of taking the list of decoders directly.
This allows the DecoderSelector to select a decoder that has been
tried or selected before, such that upon decoder reinitialization error
(e.g. switching from clear to encrypted config), or upon decode error of
the first buffer, we always have the full list of decoders to select
from.

Two mechanisms are added to avoid trying the same failing decoder again:

1. Blacklist

When SelectDecoder() is called due to reinitialization failure, or
decoding failure of the first buffer, the existing decoder is listed as
the "blacklisted decoder" such that DecoderSelector will not even try
it.

There is one exception though. When DecryptingDemuxerStream is selected,
since the input steam is changed (encrypted -> clear), the blacklisted
decoder is ignored. For example, FFmpegVideoDecoder is slected first for
a clear stream. Then on a config change, the stream becomes encrypted so
FFmpegVideoDecoder fails to reinitialize and we need to select a new
decoder. After DecryptingDemuxerStream is selected, we should still be
able to use FFmpegVideoDecoder to decode the decrypted stream.

2. Fall back at most once

If fallback has already happened and decode of the first buffer failed
again, we don't try to fallback again. This is to avoid the infinite
loop of "select decoder 1 -> decode error -> select decoder 2 -> decode
error -> select decoder 1".

BUG= 695595 
TEST=Updated/Added unittests.

Review-Url: https://codereview.chromium.org/2837613004
Cr-Commit-Position: refs/heads/master@{#469579}

[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/base/audio_decoder.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/base/audio_decoder_config.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/base/mock_filters.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/base/mock_filters.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/base/video_decoder.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/base/video_decoder_config.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/audio_decoder_selector_unittest.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/decoder_selector.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/decoder_selector.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/decoder_stream.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/decoder_stream.h
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/fake_video_decoder.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/video_decoder_selector_unittest.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/filters/video_frame_stream_unittest.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/renderers/audio_renderer_impl.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/renderers/video_renderer_impl.cc
[modify] https://crrev.com/1492be9e0d583b0110f22d7e5f2ded139611f8bd/media/renderers/video_renderer_impl_unittest.cc

wolenetz: Now our media pipeline can switch among decoders freely at runtime. This means that if MSE allows codec switch, it should just work in our media pipeline. This could be useful for use cases like ads insertion because ads and video might be encoded separately using different codecs. Is this something we can consider from MSE's perspective?

This should also fix issues like b/37326648 around switching between clear and encrypted streams.

@#6, yes, though MSE spec would need changing. There are spec bug(s) around this type of scenario already - the question would be priority of doing this versus the current "polyfill" where apps use multiple media elements.

Further to #6,9 (MSE codec switching):

Doing this would require a spec change in addition to implementation change.
In addition to codec switching, would container switching also need to be required? (e.g. splice mp4:h264+aac ad into the middle of a webm:vp9+opus stream?)

In the spec, the initialization segment received algorithm disallows codec changes. Such an initialization segment identifies the codec parameters for upcoming media segment appends to that SourceBuffer.
The spec allows for multiple initialization segments to be appended over time to a SourceBuffer (e.g. to accomplish stream adaptation), but disallows for the initialization segments to differ their codec. Paste from pertinent spec text:
> Verify the following properties. If any of the checks fail then run the append error algorithm and abort these steps.
>  The number of audio, video, and text tracks match what was in the first initialization segment.
>  The codecs for each track, match what was specified in the first initialization segment.
>  If more than one track for a single type are present (e.g., 2 audio tracks), then the Track IDs match the ones in the first initialization segment.

Rough idea of what might need to be done:
1. Determine if container switching is also desired, and if so, how to signal such a switch normatively to the SourceBuffer (SourceBuffer.abort() might not be enough by itself; "sniffing" which container bytestream type is appended following a parser reset might not be a complexity that user agents want to be required to do. Alternatively, a mechanism - perhaps a parameter to the abort() call itself, or a distinct method - could be added and required of apps to directly signal which container bytestream will be appended next.

2. Modify the spec (and incubate in WICG with interested browsers and other web authors) to
2a. Include a mechanism by which the API can be queried proactively to determine support for codec (and maybe container) switching. Not all implementations must allow such switching, right?
2a1. Perhaps a new method on MediaSource, akin to isTypeSupported()? Or maybe modify isTypeSupported() (and possibly the non-MSE canPlayType()) enough to provide definitive response to the app about?
2a2. Or perhaps something in the MediaCapabilities spec?
2b. Update the spec for MediaSource.isTypeSupported()
2c. Update the spec for the initialization segment received algorithm.
3. Modify Chrome's implementation to experiment through incubation and eventually ship it on by default.

And further to #6,9,10 - a MSE spec issue tracking 'Support "seamless" cross-codec track switching' is related, and reveals an additional piece necessary to accomplish this for MSE: if "seamless" container switching is required, then either a mechanism for proactively requesting track switching is needed, or proactively requiring the application to identify which tracks across the different containers map to the same logical stream. The former is probably simpler to do with the way media track in HTMLMediaElement is currently described.

But if container switching isn't required, just codec switching within the same container track, then both spec and implementation change scope is reduced to accomplish just "seamless" codec switching.

I'll copy #10 and this comment into that spec bug and update its title to be more reflective of the larger discussion (seamless codec switching, possibly with seamless container switching).

I've updated the related MSE spec issue (https://github.com/w3c/media-source/issues/155#issuecomment-299552809) with discussion points I raised in #10-11.

For spec-related discussion, I highly recommend it be done on that spec bug.

Update from the MSE spec discussion: See https://www.w3.org/Bugs/Public/show_bug.cgi?id=22785#c19 for a possible general HTML5 media route to do declarative automated track switching using special timed text track metadata cues. (This would require far less change for MSE, and would work for all of HTML5 playbacks; it would need full media track and text track implementation, as well as additional incubation around the special metadata cues and so forth.)

@#13, I have a newer, MSE-focused API proposal I'll share publicly soon (and link into the currently active  bug 605134 .