Matt, Dan, Chris; can you take a look here?

tl;dr:
I was unable to repro on Linux.
Dan, can you try this out on GPU/other decoders/platforms than Linux please?

Detail:
I put together a simple MSE player that appends all 6 of those segments and appears to play (A/V synchronized) continuously (once I seeked the element to begin playback at 15 seconds, which is shortly after the beginning of the buffered range.

Result:
currentTime = 29.904088, element's buffered ranges = { [15.027500, 29.981000) }
test.html:67   video buffered ranges = { [15.027500, 30.055000) }
test.html:68   audio buffered ranges = { [14.981000, 29.981000) }

This is on Linux, with Chrome 57.0.2987.133 (Official Build) (64-bit

PFA my test page.

Also, Dan reported offline that the edit lists in the mp4 stream (which Chrome currently doesn't observe for MSE) would result in a shift earlier of just 0.0566seconds for the video.
I don't think the lack of observing edit lists by Chrome is implicated in any "freezing", especially since I couldn't repro it.

Deleted: test.html 4.0 KB

test.html 4.0 KB View Download

In c#3, s/shortly after/very near/ :)

I was not able to reproduce any issue on Canary or with hardware decode.

The .webm metadata says that the duration is ~1:30, much longer than there is video data for. Perhaps this is a UI issue with your player?

Also, the only anomalies I see when I tried to repro this on Linux in media-internals were repeated messages about the mp4 video frames' DTS being slightly higher than their PTS. These are just debug messages, and they roughly correlate to the missing edit list offset processing in Chrome MSE currently (which would reduce those DTS to being at or before PTS, IIUC). I did not see any errors in local repro attempts in chrome://media-internals on Linux.

One other attempt I tried was to set mode = 'sequence' on each of the audio and video sourcebuffers, to detect potential case of discontinuity within the appended segments. The net result was again no repro, and of course no need to seek to 15 seconds since the 'sequence' mode aligned the SourceBuffers to begin at time 0. No discontinuity seems to be present/detected by MSE parser within either of the sets of segments provided in the bug report:

('sequence' appendMode):
currentTime = 14.914486, element's buffered ranges = { [0.000000, 15.000000) }
test.html:67   video buffered ranges = { [0.000000, 15.027500) }
test.html:68   audio buffered ranges = { [0.000000, 15.000000) }

> The .webm metadata says that the duration is ~1:30, much longer than there is video data for. Perhaps this is a UI issue with your player?

webkitDecodedFrameCount stopped increasing when it hit the issue. Let me try to get another sample with more complete data.

So, I played around with this a little more. If I take the same content and append it in a simple MSE harness, it does not create the same behavior.

I used to see a small "gap" in the video buffered range at the location the rendering would freeze. But outside of the context of the full Facebook app, this does not seem to be reproducing. In fact, it seems to stop reproducing when we stop using the Web Audio API. Could Web Audio API have any impact on the video's sourceBuffer buffered attribute?

+rtoy in case he has any ideas. It should be simple to add WebAudio redirection to a test page. This might come back to the GC issue on the other thread. w/ WebAudio we might end up evicting ahead of playtime and thus freeze on a gap.

You might check the reported ranges on the source buffer at the time of freeze.

@dale: Yeah I just came to basically the exact same conclusion. But I am not sure if it's actually evictions that are causing the gap. It's only 200ms long, at the beginning of a segment boundary.

videoDebug.VideoElement.buffered.length
2
videoDebug.VideoElement.buffered.end(0)
20.019954
videoDebug.VideoElement.buffered.start(1)
20.211519

We found the reason for the gaps: sourceBuffer.timestampOffset is being changed between segment appends. It seems to remain though that in some cases, with small gaps, audio playback (and videoNode.currentTime, et al) will continue to play, but the video will freeze and not continue to play (as reported by webkitDecodedFrameCount).

Maybe we can reproduce it by slowly incrementing timestampOffset between appends to have an ever increasing gap between segments to try to reproduce?

This fiddle reproduces the issue: https://jsfiddle.net/dbaulig/uw5zspyn/6/

I am appending audio segments without a timestampOffset and video segments with an increasing timestampOffset. This creates a fragmented video stream with gaps. Playing the video will at some point stop progressing because the gaps are too large. Note though that starting at currentTime ~5.642223 video frames will stop rendering and the audio track will continue playing (with currentTime continuing to update) until it reaches a larger gap at currentTime ~8 where playback stalls entirely (as expected).

Note that the content URLs will expire at some point. If that should be the case, please ping me and I will provide new ones.

Yes, (on the GC thread, I explained in detail), timestampOffset usage can cause buffered range gaps.

Is the actual duration of continued audio playback beyond the video stall more than about 2 seconds? I think our rendering pipeline allows for small video underflows to smooth playback, so long as there is still audio, in muxed A/V playback.

If you find that the duration of continued audio playback beyond the video stall is significantly larger than 2 seconds, it would likely be a bug in our decode/renderer portion of the pipeline.

In the fiddle the audio continues playing for less than two seconds, but I believe this is likely due to the continued increase of gap sizes eventually causing the stall. In the actual application the audio playback continued significantly longer than 2 seconds after the video froze (probably till the end of the stream if it wasn't for the GC issue causing a not properly handled QuotaExceedErr).

Let me see if I can change the fiddle to reproduce this more accurately.

In the log provided by servolk@ in the GC thread, I found that GC's "media_time" had progressed about 65-70 seconds beyond the video stall in an audio/video playback. This explains the eventual QuotaExceededError (because Chrome doesn't auto-evict buffers between the currently playing GOP and the last appended GOP, when the last appended GOP is later in the timeline than the currently playing GOP (and no seek is pending).

So this appears to confirm that Chrome is somehow allowing significant video stall to occur while audio is left to continue playing for much longer than 2 seconds.

Dale, can you take a look please? (Dan requested I loop you in on this investigation at this point.)

If we reach EOS on the video stream we'll continue to let the audio play -- otherwise I'd be very surprised that this occurs. You can verify this by checking to see if we fire the ended notification in VideoRendererImpl. I won't have time to look today, maybe tomorrow or Friday.

At least in the fiddle provided above I can see that we hit underflow for the whole pipeline after reporting the video underflow. Matt, which sample are you looking at?

Dale: Yeah, I've been trying to mess around with the fiddle to get it to reproduce the issue so that it continues past the video underflow, but wasn't able to up until now.

I've definitely seen it progress much further in the actual application before. I think Matt is referring to a sample that was captured in the actual application.

@#20-21, I was using servolk@'s repro log from the GC email thread. There is info on how to get that private repro info on the thread.

I wonder if this is exacerbated by having the element linked to WebAudio: might that trigger the media pipeline to decode a bunch more audio (and consequently let the pipeline's audio clock (which is currentTime in an A/V player) race forward? Perhaps update the jsfiddle to use WebAudio might enable repro there, too?

Iw as able to get the fiddle into a state that it reproduces the issue: https://jsfiddle.net/dbaulig/uw5zspyn/8/

I am not quite sure why it throws an exception upon setting timestampOffset but apparently it does (now?), but if you catch that exception and then continue appending video segments, it produces the exact behavior I've seen on the full application.

Thanks, will look at it tomorrow. From a quick look at the trace I can see that the video renderer thinks playback has ended (note "VideoPlayback" section ends), which since we don't see a BUFFERING_HAVE_NOTHING means playback gracefully ended from the perspective of the VRI.

Ah, I messed up: the reason for the exception was that I was using the URL of the wrong representation, cutting the requests/appends at the wrong byte boundaries.

Here's an updated version of the fiddle that uses the correct URL and uses a small Web Audio dummy, but that doesn't change anything. The audio stream still stops playing after approximately two seconds.

I just verified that in the actual application playback indeed continues and that even though no endOfStream is invoked there.

Note that there was a change in production that will prevent the issue from being reproduced on the test user I have provided on the email thread. I am currently working on enabling the broken behavior on that user again.

Forgot to update here that the test user was opted into the broken experience again.

Looks like URL signatures expired. Can you reupload the test case?

Any chance you guys can use the production user account and link provided in the email thread to reproduce? That's probably the more scalable solution than having to provide new CDN links every other day. 

Sorry I've been OOO so haven't had time to get to this. I tried today using the production account and was not able to reproduce the issue. dbaulig@ are you still able to hit this issue? I've asked Matt to take another look as well since he had a working repro.

Some (rough) notes for if/when I can obtain a repro (thanks Dale):

After stall reading from demuxer (e.g., on video read), current time would only go further if the other track data (e.g., audio) were continuing to be provided and eos (e.g., on video) was signaled. (If you can repro otherwise it's a bug.)

Or we only ever got one frame (we don't know how long the frame should be shown in that case).

dbaulig@, is the test URL from email message April 18 still valid for reproducing this issue? When I tried a current tip-of-tree Chromium debug build (containing mp4 support) with that URL (before even logging in to FB), MSE API failed a debug check on the appended bytestream:
[1:1:0601/150235.762913:VERBOSE3:avc.cc(214)] nal_unit_type 6
[1:1:0601/150235.763039:VERBOSE3:avc.cc(214)] nal_unit_type 9
[1:1:0601/150235.763151:VERBOSE1:avc.cc(219)] Unexpected AUD in order_state 1
[1:1:0601/150235.763297:FATAL:avc.cc(113)] Check failed: AVC::IsValidAnnexB(*buffer, *subsamples). 

This check is skipped in release builds, but it could indicate something is wrong with the bytestream. If that URL is otherwise valid for reproducing this issue with release builds, the debug-build checks are making repro+diagnosis more difficult.
 

servolk@ indicated in that email thread similar problem with debug build IsValidAnnexB, and was able to repro the issue after commenting that DCHECK out. Today, with a tip-of-tree release build, I've been unable to reproduce this issue (no buffered range gaps, so no stalls of one or the other stream) using the URL provided in email on April 18. Note, the login credentials from that email appear to be invalid (undergoing server maintenance), but playback proceeds sometimes in the background without freezing.

dbaulig@ - to help us see if there is indeed a Chrome issue where playback of audio is allowed to continue after stall of video without EOS being signalled, can you provide a URL that demonstrates the original issue?

@#34 dbaulig@ can you provide a URL that demonstrates the original issue?

Note, this may also be caused or exacerbated by Chrome's current lack of compliance around PTS/DTS: it reports buffered ranges by DTS timestamps, but currentTime by PTS. The MSE GC heuristic takes currentTime as one of its inputs, so it's possible that the heuristic might have GC'ed the GOP that was in process of being fed to the decoder. I'm landing PTS/DTS compliance fixes soon for experimentation (see  issue 718641 ).

wolenetz@: I don't have a URL handy, but I did attach the actual segment files to the issue originally when I filed it. Unless something else has changed they should continue to reproduce the issue. Will those be sufficient for you?

Alternatively, please use the test user as described above to grab some production URLs from the actual implementation.