Dropped frame that is outside append window with negative dts in mp4 and 2 frames with 0 pts.
Reported by
apasyn...@vewd.com,
Apr 12 2018
|
||||||
Issue description
UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Example URL:
Steps to reproduce the problem:
1. Use ShakaPlayer (or do that in other way) to play mp4 audio with negative dts in MSE way.
2. Observe logs.
What is the expected behavior?
All audio frames in mp4 are played.
What went wrong?
During playback of mp4 content in ShakaPlayer with following audio packets:
[PACKET]
codec_type=audio
stream_index=0
pts=-2048
pts_time=-0.042667
dts=-2048
dts_time=-0.042667
duration=N/A
duration_time=N/A
convergence_duration=N/A
convergence_duration_time=N/A
size=341
pos=9156
flags=K
[/PACKET]
[PACKET]
codec_type=audio
stream_index=0
pts=-1024
pts_time=-0.021333
dts=-1024
dts_time=-0.021333
duration=N/A
duration_time=N/A
convergence_duration=N/A
convergence_duration_time=N/A
size=341
pos=9497
flags=K
[/PACKET]
[PACKET]
codec_type=audio
stream_index=0
pts=0
pts_time=0.000000
dts=0
dts_time=0.000000
duration=N/A
duration_time=N/A
convergence_duration=N/A
convergence_duration_time=N/A
size=379
pos=9838
flags=K
[/PACKET]
I observe following behavior in logs:
[1:1:0412/095417.653047:760697626845:VERBOSE3:mp4_stream_parser.cc(809)] Emit audio frame: track_id=1, key=1, dur=21, dts=0, cts=-42, size=341
[1:1:0412/095417.653091:760697626888:VERBOSE3:mp4_stream_parser.cc(809)] Emit audio frame: track_id=1, key=1, dur=21, dts=21, cts=-21, size=341
[1:1:0412/095417.653131:760697626929:VERBOSE3:mp4_stream_parser.cc(809)] Emit audio frame: track_id=1, key=1, dur=21, dts=42, cts=0, size=379
At this point data in audio frames are wrong (cts(aka pts) are negative as in original content parsed with ffmpeg but dts starts from 0).
In next step FrameProcessor drops this frames and append wrong preroll buffer which starts at 0 pts.
[1:1:0412/095417.660290:760697634088:VERBOSE3:frame_processor.cc(654)] ProcessFrame: Processing frame Type=1, TrackID=1, PTS=-42666us, DTS=0us, DUR=21333us, RAP=1
[1:1:0412/095417.660308:760697634105:VERBOSE2:frame_processor.cc(696)] ProcessFrame: WARNING: Frame DTS(0us) > PTS(-42666us), frame type=audio
[1:1:0412/095417.660315:760697634113:VERBOSE3:frame_processor.cc(848)] Dropping frame that is outside append window.
[1:1:0412/095417.660321:760697634119:VERBOSE3:frame_processor.cc(654)] ProcessFrame: Processing frame Type=1, TrackID=1, PTS=-21333us, DTS=21333us, DUR=21333us, RAP=1
[1:1:0412/095417.660341:760697634138:VERBOSE2:frame_processor.cc(696)] ProcessFrame: WARNING: Frame DTS(21333us) > PTS(-21333us), frame type=audio
[1:1:0412/095417.660347:760697634144:VERBOSE3:frame_processor.cc(848)] Dropping frame that is outside append window.
[1:1:0412/095417.660352:760697634150:VERBOSE3:frame_processor.cc(654)] ProcessFrame: Processing frame Type=1, TrackID=1, PTS=0us, DTS=42666us, DUR=21333us, RAP=1
[1:1:0412/095417.660363:760697634160:VERBOSE2:frame_processor.cc(696)] ProcessFrame: WARNING: Frame DTS(42666us) > PTS(0us), frame type=audio
[1:1:0412/095417.660369:760697634167:VERBOSE1:frame_processor.cc(562)] Attaching audio preroll buffer [-21333us, 0us) to 0us
In end result ChunkDemuxer return following 2 frames with pts=0
[1:1:0412/095417.664798:760697638595:VERBOSE2:source_buffer_stream.cc(1669)] GetNextBuffer AUDIO: handled next buffer with preroll, returning status kSuccess
[1:1:0412/095417.664803:760697638600:VERBOSE2:chunk_demuxer.cc(410)] CompletePendingReadIfPossible_Locked: returning kOk, type 1, dts 0.042666, pts 0, dur 0.021333, key 1
[1:19:0412/095417.664931:760697638728:VERBOSE2:source_buffer_stream.cc(1669)] GetNextBuffer AUDIO: handled next buffer with preroll, returning status kSuccess
[1:19:0412/095417.664935:760697638733:VERBOSE2:chunk_demuxer.cc(410)] CompletePendingReadIfPossible_Locked: returning kOk, type 1, dts 0.042666, pts 0, dur 0.021333, key 1
Responsibility of reading dts is on TrackRunIterator
track_run_iterator.cc
const TrackFragment& traf = moof.tracks[i];
......
int64_t run_start_dts = traf.decode_time.decode_time;
uint64_t sample_count_sum = 0;
for (size_t j = 0; j < traf.runs.size(); j++) {
const TrackFragmentRun& trun = traf.runs[j];
TrackRunInfo tri;
tri.track_id = traf.header.track_id;
tri.timescale = trak->media.header.timescale;
tri.start_dts = run_start_dts;
But TrackFragmentDecodeTime structure in TrackFragment contains uint64_t(unsigned type) for decode time:
box_definitions.h
struct MEDIA_EXPORT TrackFragmentDecodeTime : Box {
DECLARE_BOX_METHODS(TrackFragmentDecodeTime);
uint64_t decode_time;
};
Did this work before? N/A
Is it a problem with Flash or HTML5? HTML5
Does this work in other browsers? N/A
Chrome version: 65.0.3325.181 Channel: stable
OS Version: ubuntu 16.04
Flash Version:
Contents of chrome://gpu:
,
Apr 12 2018
apasyniuk@vewd.com: w.r.t. the dropped frames and the 2 frames at time 0: those are dropped (compliantly per MSE) due to AppendWindow trimming logic. At least part of those frames' presentation intervals (in PTS) is outside the appendWindow (which is [0,+Infinity) by default, can cannot be set to start earlier than 0) -- in the example, both frames are completely outside the appendWindow. Chrome attempts to retain the most recent frame that is dropped to use it to preroll the decoder for a frame that isn't dropped that is immediately subsequent to it in append and decode order. In the logs, the preroll buffer inherits the timestamps of the subsequent buffer, so what is actually happening is the -21ms (PTS) buffer is being used to preroll the decoder such that the real buffer beginning at 0ms (PTS) can begin with stable output. Note that this PTS-based appendWindow trimming (including partial audio frame trimming in Chrome) along with audio preroll behavior in Chrome enables gapless audio playback scenarios. w.r.t. the DTS calculation in Chrome logs: I believe this is an artifact of the old (currently default) incompliant buffering-by-dts behavior, where we elected to not shift dts backwards by ELST offset because that would make more streams fail due to negative DTS. The PTS (cts) *is* calculated correctly, and used for append window trimming (even in buffering-by-dts behavior). Once MseBufferByPts feature ships on-always, removing some of these old logs will be part of the extensive clean-up involved in removing the legacy buffering-by-DTS code. Those items aside, I'm quite curious: Does your scenario not work? Can you describe what is broken?
,
Apr 12 2018
,
Apr 13 2018
I use custom decoder in my scenario which do not implement special handling of preroll frame. Main problem was that two frames have exactly same pts(in this case 0). From my understanding of your description of trimming and preroll: - first audio frame(which is preroll buffer) should have discard_padding=kInfiniteDuration, - such frame should be only decoded but not presented Am I correct?
,
Apr 13 2018
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 16 2018
@wolenetz: Could you please check comment#4 commented by reporter. Also please let us know does this require any triage from TE end?
,
Apr 20 2018
A Gentle ping... wolenetz@ request you to please respond to comment #6, which will help in further triaging of the issue. Thanks..
,
Apr 24 2018
As there is no update for comment #6 & #7, adding 'TE-NeedsTriageHelp' label and requesting someone from Internals>Media>Source team to look into this issue and help in further triaging. Thanks..
,
May 22 2018
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by dalecur...@chromium.org
, Apr 12 2018Owner: wolenetz@chromium.org
Status: Assigned (was: Unconfirmed)