Hello,

FYI, the person who needs to take a look at this is Matthew Wolenetz:
wolenetz@chromium.org
Why are you not assigning it to him?

There are two ways to reproduce this issue:

1. We can send you a link to a webpage, similar to http://umediaserver.net/umediaserver/demohtml5player.html
where you can see this issue, but that needs to be coordinated with us, because we cannot have the encoder encoding all the time. When you are ready to take a look, drop us an email at contact@umediaserver.net and we will start the stream and send you a link to webpage.

2. You can reproduce this issue by yourself by installing Open Broadcaster Software (OBS) classic, and Unreal Media Server on Windows OS (virtual machine is fine too). With OBS advanced settings, make Key-frame frequency 1 second; make sure you encode with x264. Select some video source like local webcam, configure RTMP push to Unreal Media Server and start streaming.
Now you can create a webpage that uses Unreal HTML5 player, similar to 
http://umediaserver.net/umediaserver/demohtml5player.html
that links to your stream. You will see this issue in Chrome. IE and Edge work fine. Again, if you want us to help you configuring this by yourself, drop us an email.

#3 - 
1. No point in assigning if the issue is not reproduced yet.
2. I am sure Matthew is not the only one working on that area, so it is better to put the right component and let the usual triage take its course (unless a specific revision is found to cause the issue and Matthew is its author, in which case, it will be assigned to Matthew).

Thanks for reporting this issue.

This may not be specific to MSE (because decoder seems likely problem).

To assist our repro, please share a bytestream that reproduces this issue and the parameters to addSourceBuffer(..) used with that bytestream.

Also, a few other items will help:

1) does this repro only on Windows? (Does it work on Mac, for instance?)
2) Is it using GPU or software decode? (please share a chrome://media-internals "Save Log" output from a repro)
3) Is it 100% reproducible using the bytestream; are there any seeks or other operations needed to induce repro?

This sounds more like general Windows H/W decode/render issue. Over to sandersd@ to attempt repro once bytestream for repro is available.

Thanks for looking into it.

1) We didn't check on MAC, but Chrome on Android works fine, so for now - yes. seems like Windows-only issue.

2) Attached is media-internals.txt

3) It is 100% reproducible using the bytestream, you start to see the issue from the beginning, no other operations are needed.

We will provide the bytestream later today.

Deleted: media-internals.txt 16.4 KB

media-internals.txt 16.4 KB View Download

Thank you for providing more feedback. Adding requester "wolenetz@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

OK, I am attaching a bytestream. It's a fragmented .mp4; the first segment is init segment and then media segments follow.
All players like VLC or WMP happily play this .mp4 file.
But in HTML5 player in Chrome, when I was playing, I saw constant video shuddering.

Here is the addSourceBuffer call:
addSourceBuffer("video/mp4; codecs="avc1.64001E, mp4a.40.2"") 

Video is encoded with OBS, high profile, veryfast preset. Unreal Media Server receives the stream from OBS, does mp4 packaging and streams via Websocket to HTML5 player.

One note about this mimetype parameter to addSourceBuffer: Chrome is the only browser who cares about "codecs" part in this string. Why? The exact codec profiles and levels will come in init segment, and that's where it should be taken from, not from this string. Just strange. So Unreal Media Server may not be exact about formatting the "avc1.64001E" string - the actual h.264 level that you will find in init segment may be different. I hope it doesn't matter. I think Chrome uses this string just for basic validation, not more. Correct me if I am wrong. Would be absolutely crazy if that causes the issue being discussed. 

Tomorrow I will try to provide a  webpage with live stream containing b-frames
so you can see the issue live.

Deleted: UFragmented-23-22-04.zip 9.7 MB

UFragmented-23-22-04.zip 9.7 MB Download

OK, here is a live stream with this issue:

http://umediaserver.net/umediaserver/bframesissue.html

It is a live stream: OBS captures a player window, encodes with x264 High profile, 1-sec K-frame frequency, and does RTMP push to Unreal Media Server. 

The timestamps in this MP4 are incorrect; it has PTS = DTS for every frame. For H.264 in MP4 specifically, the PTS order (in the MP4) must match the PicOrderCnt order (in the H.264 stream).

The artifacts you are seeing are cased by disagreement between the decoder (which only sees the H.264 stream) and the player (which only sees the MP4 metadata) on frame order.

Taking this on as a task to improve logging; we can log a message in chrome://media-internals when this problem is observed.

So please explain why HTML5 player in IE and Edge play OK?
And why every player like VLC and WMP play this .mp4 OK?
And why Chrome 43.0.2357.65 plays OK?

If we have to recalculate PTS on the server side, the latency will increase and the whole idea here is to play with low latency.
 

I can't comment on how other players work; presumably they are disregarding the MP4 timestamps and relying on the H.264 stream directly. This isn't a reasonable course in general, since the H.264 stream does not (usually) include frame duration information, so additional layers of translation would need to be involved.

Chrome 43 predates Project Butter (https://youtube-eng.googleblog.com/2015/11/smoother-in-chrome.html), which introduced the most recent frame selection algorithm.

I don't understand this concern about latency--the use of B-frames is already incurring at least 33ms of latency, compared to which PTS computation is trivial, and the correct PTS values should already be an output of the video encoding process.

"presumably they are disregarding the MP4 timestamps and relying on the H.264 stream directly. This isn't a reasonable course in general, since the H.264 stream does not (usually) include frame duration information, so additional layers of translation would need to be involved."

Well, looks like all other players disagree with you, and I don't agree as well. It's a decoder's task to recalculate PTS for b-frames; the decoder is the one who knows where b-frame belong.

All RTMP publishing encoders like FMLE and OBS provide RTMP streams and RTMP protocol has only one timestamp. So we on the server side, would have to do the same job that h.264 decoder does, to provide different PTS and DTS.

Regarding increasing latency - we will need to buffer the last GOP, to recalculate PTS, hence we are one GOP-late. Well, currently the decoder in Edge does the same so in this sense the result is the same...
We do see higher latency with streams containing b-frames, that's inevitable.

> Well, looks like all other players disagree with you

Since it was handy, I just tested in Safari, and it jitters exactly like Chrome.

Note that it's also plausible that other players are just dropping all the out-of-order B-frames, which is what Chrome would have done in M43.

> it's also plausible that other players are just dropping all the out-of-order B-frames

Doesn't look like it - try Edge and IE - they play smoothly.
Chrome 43 also plays smoothly and doesn't drop frames - we would see it.

I am turning off the live stream with b-frames. Let me know if you need it - I can turn it on.

1. I want to correct my mistake in the previous post:

> All RTMP publishing encoders like FMLE and OBS provide RTMP streams and RTMP protocol has only one timestamp.

In fact, RTMP protocol does provide composition times for H.264 NALUs, so we probably can provide valid PTS values for streams coming over RTMP (provided encoders send these composition times correctly)

2. However, in general the requirement of having exact PTS values in .mp4 makes it very hard to work with live streams containing b-frames with legacy systems that only provide one timestamp to a coded frame (usually DTS). So these systems will continue to use Flash instead of MSE (Flash player plays streams with b-frames just fine, given only a single timestamp for frame). For example, streams received via RTSP protocol and streams received from DirectShow encoders, only have one timestamp for coded frame and there is no way in general to pass additional composition time.
And, of course, to reconstruct PTS from DTS and bytestream is not feasible on the server side that receives bytestreams from live encoders over IP streaming protocols.

So your decision to go with this philosophy limits the usability of MSE comparing to Flash. At least with Chrome and Safari, as Edge and IE don't have this problem.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/44cce866bb03e9424ac86ac4cdff8500fd906567

commit 44cce866bb03e9424ac86ac4cdff8500fd906567
Author: sandersd <sandersd@chromium.org>
Date: Mon Mar 27 20:51:16 2017

Log out-of-order frames in chrome://media-internals.

BUG= 703019 

Review-Url: https://codereview.chromium.org/2770253004
Cr-Commit-Position: refs/heads/master@{#459871}

[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/content/renderer/media/webmediaplayer_ms.cc
[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/content/renderer/media/webmediaplayer_ms_compositor.cc
[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/content/renderer/media/webmediaplayer_ms_compositor.h
[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/media/filters/video_renderer_algorithm.cc
[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/media/filters/video_renderer_algorithm.h
[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/media/filters/video_renderer_algorithm_unittest.cc
[modify] https://crrev.com/44cce866bb03e9424ac86ac4cdff8500fd906567/media/renderers/video_renderer_impl.cc

Back to wolenetz@ for an opinion on handling timestamps in MSE.

MSE requires PTS and DTS determinable from the appended streams to correctly report buffered ranges (in PTS time) and to manage buffering correctly (using both PTS and DTS).

sandersd@, what does regular src= playback do with such a stream (and what does the ffmpeg demuxer report for DTS/PTS of such legacy streams?)

Also, in Chrome's decoding and rendering, is there a viable work-around to prevent janky playback of such streams with missing/incorrect PTS values?

Also, mershstein@, please elaborate on your statement, if possible: "to reconstruct PTS from DTS and bytestream is not feasible on the server side that receives bytestreams from live encoders over IP streaming protocols".

Also, is it infeasible to reconstruct corrected PTS information in your JS web app before appending the media to MSE? JS transmuxers are becoming quite common due to things like legacy HLS being adapted to play with MSE.

sandersd@, it might also be good to understand how frequently streams hit your logging path / this edge case.

As I wrote before, a typical streaming server that would want to send streams to MSE, does not do live encoding by itself but rather receives encoded streams from live software/hardware encoders over various streaming protocols. Not all these protocols carry both DTS and PTS (for example, RTSP cannot).

So if the H.264 stream containing b-frames and having DTS timestamps only, is sent to:

Flash player, VLC player over RTMP, VLC player over MPEG2-TS, DirectShow players, hardware STBs via MPEG2-TS, MSE players in IE and Edge:

The result is: playback is fine. So obviously decoders in these players are able to resort frames and readjust PTS.

It's only MSE players in Chrome and Safari (not sure about Firefox) that require proper mp4 filling of PTS and DTS in order to play correctly.
(Chrome 43 also played OK).

Reconstructing PTS on the server side, when you have DTS only, would require parsing and analyzing the H.264 bytestream and thus would increase latency and server's CPU load. Parsing H.264 with Javascript on the client side is kind of crazy idea - the webpage would have to load heavy javascript and eat CPU as well. And that is for each web viewer.