I forgot to note that this also happens regardless of sourceBuffer.mode = 'sequence' or sourceBuffer.mode = 'segments'

I muxed another MP4 at 50fps instead of 100fps and this one behaves a bit differently. See the attached zip.

Due to the FPS change I had to remove this line:

sourceBuffer.timestampOffset = 1; 

And then it seems to work slightly better. It will render the first frame and also the second frame, but then it doesn't render any more frames until about frame 45 (holding down the ] key). From that point it seems to be working 100%. But that's still far off from the behavior of Chrome Beta and Canary as of last week.

Deleted: SMPTE_50fps_dashinit.zip 10.1 MB

SMPTE_50fps_dashinit.zip 10.1 MB Download

Confirmed with the 50fps zip. Canary (52.0.2717.0) is definitely showing different behavior from earlier version I tested (52.0.2715.0).

In general I don't think we guarantee frame accuracy in the current version of the MSE spec, but it is interesting to see things changed so much in canary. Dale, do you know whats changed? Maybe the avsync stuff you landed (https://codereview.chromium.org/1863373002)?

The new low delay mode takes frame timestamps into account, if they drift too far behind the decoder will wait for the first frame that is within an acceptable amount of drift of the media clock. I'd guess your timestamping is off such that the next frame is not considered valid.

Dale, yeah I am using MSE in a non-standard way where I decode on demand up to 60FPS with non sequential frames (ie. frame 10, 3, 200, 25, etc.) at non-constant time intervals. Though I've tested my code on Firefox and it does follow the old Chrome behavior. 

See here for some more info on my use case http://stackoverflow.com/questions/36364943/frame-by-frame-decode-using-media-source-extension - Essentially FPS/timestamps don't matter for my implementation, but I understand my use of MSE is rather outside of its intended scope. 

Regarding Low Delay Mode - I was able to replicate this issue in both modes. See the attached files. Most of the time it will increment in 4 frames, but sometimes it increments in 3 or 5 frames.

If this code is staying around for future Chrome versions, do you have any recommendations on how I can achieve frame by frame non sequential decode at random intervals? What would the new code do if there were no individual frame timestamps? Would it be possible to set the TimeStampOffset as a workaround for each frame?

Thanks,
Dustin

Deleted: NonLowDelayMode.zip 6.0 MB

NonLowDelayMode.zip 6.0 MB Download

You'll have to standardize the time intervals between your frames or set current time appropriately with each [] advancement such that the latest advanced frame is considered valid. It definitely looks like currentTime is getting ahead of where you think it is and thus the frame you want to display is no longer valid.

Thanks Dale. I was able to get my code working by adding video.currentTime += .01 every time before I append to the buffer. 

Do you feel that's a relatively safe fix going forward? Any other thoughts/opinions on the robustness of my implementation would be much appreciated given your knowledge of the Chromium roadmap.

Oh, and feel free to close this issue.

Thanks,
Dustin

Yeah that should be a safe fix; obviously this use case is a bit unexpected so I can't guarantee it'll work forever, but nothing on our schedule (the latest changes weren't on a schedule fwiw) should break this.

Thanks Dale.

Hey Dale,

I pinged Matt Wolenetz earlier today but figured I'd also post on this thread as it contains an easy to repro case.

In today's OSX Beta, Dev, and Canary builds I am no longer able to achieve frame accurate decoding using MSE. Windows and Android do not seem to be affected, and Chrome 52 and 53 stable on OSX are fine. You can reproduce using the attached files above or via this JSFiddle - http://jsfiddle.net/t5ypom6k/ - The SMPTE video should display on load and then you should be able to increment/decrement frames using the [] keys. 

Are you aware of any changes on the OSX MSE code that could explain this? Should I add a bug for 54.x Beta on OSX? Unfortunately this will be a breaking bug for my use-case so any insight would be appreciated.

Thanks,
Dustin

sandersd@ has made some changes here recently; over to him to take a look. Does this work correctly with --disable-accelerated-video-decode ?

Thanks for the quick reply Dale. 

Yes, it does work correctly with --disable-accelerated-video-decode

Don't think so, that's for all platforms.

Yah, sorry about that. I just deleted that comment as I realized the same thing. It's definitely specific to OSX.

+self and chcunningham@ to watch this w.r.t. previous email with Dustin.

One interesting thing about what I'm seeing is that OSX Chrome (v54+) is able to play the mp4 directly when I drag the file onto the browser. It only fails when I play it using my implementation of MSE - http://jsfiddle.net/t5ypom6k/ - If GPU decode is disabled then it works in all cases.

Another piece that could be influencing the behavior is that I am using custom build of MP4Box that sets trex->track->Header->duration = 0 and movie->moov->mvex->mehd->fragment_duration = 0 so that I invoke Chrome's "Low Delay" mode - see here for more info on that - http://stackoverflow.com/questions/36364943/frame-by-frame-decode-using-media-source-extension

Let me know if I can help in any way.

dalecurtis@: This would all be much easier if seeking wasn't necessary. Perhaps there is a better way to handle this case when in low-delay mode?

I'm seeing in the decoder that Reset() is called before frames complete decoding, causing them to be dropped.

What's happening is a sequence of events like this:
  - seeking begins, and schedules a sequence of asynchronous flush/reset operations.
  - appendBuffer() runs, and a frame is send for decoding.
  - the async flush/reset hits the decoder, and decoding is dropped.

How MSE should be scheduling this is not something I understand, but AFAICT the hardware decoder is operating correctly.

Assigning back to wolentz@ for MSE analysis.

sandersd@ and I briefly discussed this this morning. Back to him with the following context:
1) If the JS is doing: appendBuffer(frames at next seek target time)+seek(next seek target time) (either with or without awaiting updateend before issuing that seek), and there's no buffer fed to the decoder after there are frames buffered at the seek target time, that sounds like a bug.
2) If the JS is pausing playback and then doing the append+seek, yet the renderer never displays the prerolled, paused frame at the seek target time, that sounds like a bug.
3) (Versus #1 and #2, above): If playback is not paused, and JS appends and seeks, and decoder is fed compressed frame at seek target time, but the video renderer algorithm drops the decode result due to time having progressed beyond it, it's a bit iffy if that is a bug. In this use case, it would be better for at least first-video-frame-after-seek-preroll to *not* be dropped even if time has progressed beyond it (at least, in the low-latency mode).

dustin...@: Do you still experience this issue? Enough code has changed here that I can't be sure, and I was never able to reproduce it myself.

This is working as expected now. Feel free to close this issue. Thanks!