Cc'ing mlamouri@ for help in investigating this further as this could be related to  Issue 593273 .

Thank you!

per.h.hagglund@gmail.com,can you provide a repro url?
matt@, is this a known issue to you?

After the video frame freezes (and doesn't automatically resume), and the audio continues for a short while, does the audio eventually stop, too (and before the end of the last audio segment?) I suspect audio also stalls, just we have added a little video-underflow leeway in the renderer that allows audio to continue playing a little beyond the last buffered video.

Also, in the repro case, do the audioSourceBuffer.buffered() show a continuous single range, the videoSourceBuffer.buffered() show two ranges separated by a gap, and the resulting videoTag.buffered() show the intersection of those audio & video buffered ranges?

friendly ping:

per.h.hagglund@gmail.com : I believe this is working-as-intended in Chrome where we allow a small amount of video underflow in an audio/video playback prior to stalling the playback to reduce playback jankiness in some situations. Is this behavior posing  a significant problem that can't be worked around?

Sorry for the high latency response, I've been really busy this week. I'll try to make time for making an example of the problem the coming week.

Just to inform: the bug is still active in 49 (and in beta/canary).

I've tested some more and noticed a few more weird stuff that might help. Read the answers to your questions below.

wolenetz@chromium.org : 
"After the video frame freezes (and doesn't automatically resume), and the audio continues for a short while, does the audio eventually stop, too (and before the end of the last audio segment?)"
- No, it does not eventually stop, it just continues as long as we've got audio. Weird fact 1: If, after any number of video segments has been dropped (or during the drop), any number of audio segments are dropped the video will be resumed correctly without any pause/play on a seek (which is the expected behavior with video as well).
 
"I suspect audio also stalls, just we have added a little video-underflow leeway in the renderer that allows audio to continue playing a little beyond the last buffered video."
- The problem is that this leeway is way too long in that case. If I would drop multiple video segments (200 ms each), the video still freezes and does not resume when seeking to a place where both audio and video exists (unless explicitly calling pause/play, or manually dropping an audio segment).
And even if a long leeway is ok, the video should not be freezed when seeking to a valid time (where we have video/audio).

"Also, in the repro case, do the audioSourceBuffer.buffered() show a continuous single range, the videoSourceBuffer.buffered() show two ranges separated by a gap, and the resulting videoTag.buffered() show the intersection of those audio & video buffered ranges?"
-Yes to all three questions, audio shows a continuous single range, video shows two ranges, the resulting buffered shows the intersection.

To summarize: as I see it, either 
-keep the "leeway" but fix the freezeframe issue where it does not unfreeze when seeking to a valid location where we have both audio and video (which would probably be what most (including me and the companies I represent) would appreciate the most, thanks to the continous audio playback)
or
-remove the leeway, trigger any kind of "can-not-play" state to allow the developer to handle the issue appropiatly. (this is how firefox handles it)

Either one of these solutions fixes the problem.

This behavior is posing a big threat to our streaming solution since the video.pause();video.play() does not always solve the freeze frame issue. And the worst thing is that there is no way to programmatically detect if the video is "freezed" with playing audio, since the video element thinks it's playing correctly.

Just comment if you want me to explain or test anything else. I'll respond faster next time. I'll also try to set up some code so you could test this (or you can try to follow the steps). 

This looks like the same issue than an issue I opened a while ago: https://bugs.chromium.org/p/chromium/issues/detail?id=423801

In my opinion, the leeway introduces a lot of problems and is not the desired behaviour: simply said, most MSE based players implement a workaround by setting the playbackRate to 0 when reaching the end of the buffer, and also emulate a 'waiting' event (see below):
- dash.js: https://github.com/Dash-Industry-Forum/dash.js/blob/4c0d599a4914406bbab6defb5af40270362200c5/src/streaming/models/VideoModel.js#L144
- rx-player: https://github.com/canalplus/rx-player/blob/684c926395e682d64f6fa7327e7fac1b7a35a510/src/core/stream.js#L430
- shaka-player: https://github.com/google/shaka-player/blob/19973d63f439537528f8c4d255d8b75d8fa7e643/lib/media/playhead.js#L154

I guess the fact the the 'waiting' event is not triggered properly in case of buffer shortage is also a consequence of this. 
This issue was open 2 1/2 years ago, and the resolution would be so simple if you just had a clear definition of "nothing left in buffer": https://bugs.chromium.org/p/chromium/issues/detail?id=279213

This creates all kinds of problems, mostly developpers have to implement all sort of workaround to get their player skin and QoS analytics right. Like this polyfill that we implemented just for chrome: https://github.com/streamroot/video-event-polyfill/blob/master/lib/video-event-polyfill.js
It looks simple enough, except once in a while we spend a couple of days trying to fix a corner case because some player we integrate with also implements some kind of workaround for this issue. And I'm sorry to say that only chrome has this wrong, while bugs around these issues have been opened for a couple years awaiting resolution.

I'm sorry for the rant, and think you guys are doing an amazing job with MSE (I've been using it since it first started working in Canary). But I would appreciate it very much if you would just keep buffer shortage behaviour simple and finally get it to work according to spec.

Thanks for the comment, I've assigned the issue of events which I agree we should be doing a better job with. The 3 second delay may not be going anywhere though since it has measurable benefits as reported by YouTube and Netflix.

What would be your ideal event delivery pattern if we kept the 3 second delay?

Hi! 

I've just posted a reproducable example code repo, which also could help some when debugging. It has prechunked h264 segments and everything.

You can access that repository here: 
https://github.com/RealSprint/chunked-mse-bug-593271
and if you want to see it live without downloading the repo:
http://dev.realsprint.com:3000/
I've also attached a zip of that repo in this post.

In our opinion, the most pressing issue isn't to follow the spec or to have the correct events (even if that would be very awesome). The most pressing issue is to get the video to unfreeze after a drop and a seek. It's easy to understand what I mean if you check the demo.

Deleted: bug-repro.zip 961 KB

bug-repro.zip 961 KB Download

@#8 Thanks for providing a demo. I'll take a look soon.

Any updates on this? This issue is currently labeled as "needs feedback". Let us know if there is anything regarding testing or something that you need help with.

Seems to be the same as a bug just reported on Shaka Player, but specifically affecting Android: https://github.com/google/shaka-player/issues/438

Any updates?

On Windows and Android this seems to also happen when the video buffer runs out and we still have audio buffered. In this case the audio will keep playing while the video freezes.

On Windows it stops the audio after a second or two, but on Android the audio keeps playing until audio buffer is also exhausted.

Both are undesirable behaviors, but the audio not stopping at all on Android is quite worrying, as I can imagine it happening a lot on our typical usage environment: slow mobile networks.

Current best idea of a workaround is to monitor the video.buffered ranges at very short intervals and calling video.pause() when we detect buffer running out. 

To reproduce, you can use Chrome DevTools network throttling.

* go to https://shaka-player-demo.appspot.com/demo/ on desktop Chrome
* open Chrome DevTools network panel
* choose Sintel 4k stream
* let it buffer some of the video, then switch Chrome's network throttling on to GRPS setting
* wait for the buffer to empty

Expected: video and audio stops at the same time the buffer runs out
Actual: video freezes, but audio continues for a few seconds (or on Android, stop only after all of audio buffer is exhausted)

wolenetz, I'll take this. It dovetails with the changes* I'm making to fire the "waiting" event. 

*https://codereview.chromium.org/2115143002/

I've dug into the initial report and the repro attached to comment 8. Re: other comments, I'll respond shortly (I also dislike the leeway).


High level comment - I note that your repro attempts to advance past gaps using an "checkOff" interval. I'm not sure if this is part of your production streaming solution, but if so I strongly recommend instead filling the gaps in your buffered segments for smoother/more-interoperable playback. I recognize that our lack of support for the waiting event may make this more difficult (apologies, fix in-progress). I typically see streaming apps do their own monitering of buffered vs currentTime - always ensuring some room ahead and filling gaps to prevent a stall. 


From initial report:
> When a new video segment arrives, the video does not resume until you do "myVideo.pause();myVideo.play();" which is really weird, since the state has not really changed.

I'm guessing by "new video segment" you mean a segment that is *beyond* the gap rather than filling the gap. If so this is not expected to automatically resume playback - the gap must be filled to resume at the current position. (If I've guessed wrong, pls attach a repro of this). I'm not able to repro the resume from video.pause/play - video is still frozen for me after calling play again. LMK if you're still seeing this and can give more instructions on reproducing. 


RE: leeway being too long (seemingly for length of audio):
This is a separate bug which is *now fixed* and should ship out in M52. Basically, when rendering live streams in our "low delay mode" (which we are in for your repro), we fail to reliably detect video underflow. You can see the fix live now in Dev or Canary channels.  


RE: video does not resume when seeking:
In your repro, seeking is done via the checkOff function. Seeking is only performed when you notice a significant drift between wall clock time (using date.now()) and mediaElement.currentTime. The subtelty here is that, due to the leeway-too-long bug above, in the case of a video-only gap the mediaElement.currentTime never actually stops advancing. This means that checkOff never detects a gap in this case and never actually performs a seek. I've confirmed this by adding logs to your repro. I've also confirmed that if I manually set currentTime=currentTime after the video freezes (advancing past the gap), it does actually unfreeze. 

Responding to comments 6 and 7:

I'm also not a fan of the 3 second delay for video-only gaps. As pointed out in 6, it causes confusion and leads to hacks. But also, I think its really a bandaid over  a bigger issue: Chrome's renderer doesn't currently know the difference between demuxer buffer underflow (e.g. buffered gap) vs decoder underflow (decoder can't decode in time, regardless of how much encoded data is buffered). Looking at the change** where the 3 second delay was added, decoder underflow is mentioned as the motivation. Its easy to see why YouTube/Netflix would be happier with this change - it avoids pausing audio for a momentary glitch with the video decoder.

IMHO the correct solution is to do away with the delay and immediately pause whenever reading from the demuxer buffers comes up empty (gaps). Decoder underflow should *never* trigger a pause - it should instead simply drop frames until things recover. Apparently, things usually recover in 3 seconds or less. This should still make YouTube/Netflix happy - audio won't pause just because video decoding is a little slow. 

This proposal does introduce a new possibility that, for decoder underflow, perhaps the decoder will never be able to catch back up (e.g. it used to benefit from the eventual pause). I think this is a pretty rare case, but even still we can add back a pause for decoder underflow if needed.

This is a non-trivial change - hence the bandaid.

**https://codereview.chromium.org/475863002/

@#17 +1 to differentiating decoder underflow vs pre-decode buffering underflow, and stopping renderer sooner for the latter case.

Re comment 13 (Android):

I was not able to repro the adio-plays-forever-while-video-freezes using the shaka demo player + network throttling. In all of my attempts I found that video and audio paused essentially at the same time (if you expand the '> info' tab, this was when the feild "Buffered:" read " -**s / +0s").

When I use the repro script uploaded in comment 8, I see the same behavior as on desktop. With version 51, audio plays on forever while video is frozen. With newer versions (where the low delay bug is fixed), audio stops 3 seconds after the video freezes (working as intended). 

I'm not entirely satisfied though because the shaka player's sintel stream is not low-delay mode... which is the critical detail to explain why audio never pauses. Does your repro with shaka player still work? 

> but if so I strongly recommend instead filling the gaps in your buffered segments for smoother/more-interoperable playback.

is it possible to build MP4 segment what fill specified gap with audio silence? this segment must be universal, i.e. not contain encoded audio data ('mdat' box is empty), only matadata.

I'm not sure I follow you. The original report has gaps in the video stream, so filling with audio silence is not reasonable (video gaps must be filled with video content). 

If you're meaning to fill audio gaps, you can definitely fill them with silence, but I don't know how to do this with just metadata. 

The best course is to ensure the media you're using has accurate timestamps/durations and does not create unexpected gaps.

 Issue 687639  has been merged into this issue.