Is the issue https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=328512#2093, https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=328512#2094, https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=328512#2095 ?

@#2, The log entry:
  "(Log limit reached. Further similar entries may be suppressed): Warning: using MSE 'sequence' AppendMode for a SourceBuffer with multiple tracks may cause loss of track synchronization. In some cases, buffered range gaps and playback stalls can occur. It is recommended to instead use 'segments' mode for a multitrack SourceBuffer."

is important. It leads to the eventual error in playback:
 "Failed to reconcile encoded audio times with decoded output."

'sequence' mode for multi-track SourceBuffer is very likely to be deprecated soon in both MSE spec and Chrome implementation.

Rather, just use the default 'segments' appendMode for the SourceBuffer. If you need to adjust where segments get appended into the timeline, there is the SourceBuffer "timestampOffset" [1] which can be adjusted to shift subsequently appended media segments' coded frames' timestamps by that offset.

[1] https://www.w3.org/TR/media-source/#dom-sourcebuffer-timestampoffset

I'm keeping this bug open though, because this situation should result in a tab crash. I suspect it might be related to the HTMLMediaElement being captured-from having reached an error state, but I could be wrong.

@#3 replace "should result in a tab crash" with "should NOT result in a tab crash"

@#4 Using "segments" mode at Chromium results in no media being played following "updateend" event https://plnkr.co/edit/E8OlvwiUmCwIUTOKSNkv?p=preview. Similar to what previously described at https://bugs.chromium.org/p/chromium/issues/detail?id=737757#c9 at previous attempts to use MediaSource to achieve the current use case of recording multiple media fragments having arbitrary MIME type or codecs at a single webm or other media type file. Not certain how to effectively use timestampOffset to remedy the issue while using "segments" mode. Asked how to do so several times at comments and at issues filed at GitHub; including https://github.com/w3c/media-source/issues/190; https://github.com/whatwg/html/issues/2824; https://github.com/w3c/media-source/issues/191; timestampOffset is used at this approach https://stackoverflow.com/questions/46379009/how-to-use-segments-mode-at-sourcebuffer-of-mediasource-to-render-same-result, last updated at https://github.com/guest271314/MediaFragmentRecorder/blob/master/demos/recordMediaFragments.html, which uses ts-ebml to set seekable cues at resulting webm file though is also inconsistent as to actually reaching the final portion of the code - the tab might crash; only the first video is rendered at playing of all concatenated videos; errors occur when setting timestampOffset that prevents code from proceeding.

Ideally, we could use a similar approach as Web Audio API uses with OfflineAudioContext to get video data from an array buffer - without necessarily needing to play the media first or at all - though we might still face the issue of synchronizing the audio and video data at subsequent recording and playback.

The above described use case is possible when utilizing canvas.captureStream() and AudioContext.createMediaStreamDestination() with MediaRecorder.pause and MediaRecorder.resume, as proven by Kaiido at this Answer https://stackoverflow.com/a/45343042 at https://stackoverflow.com/questions/45217962/how-to-use-blob-url-mediasource-or-other-methods-to-play-concatenated-blobs-of, with slight adjustments to the code. Full credit goes to https://stackoverflow.com/users/3702797/kaiido for the approach and proof-of-concept, see https://plnkr.co/edit/E8OlvwiUmCwIUTOKSNkv?p=preview at version 2.

Still have the sense that am not fully gathering the intended functionality of MediaSource, and with adjustments, the implementation can be used to achieve the use case requirement.

The purpose of the code at this issue is to revisit the possibility of utilizing MediaSource with MediaRecorder to determine if this approach is at all viable, or all hope should be abandoned as to further trying to marry Chromium's implementation of MediaRecorder and MediaSource to achieve the requirement of the use case.

Deleted: recordArbitraryMediaFragmentsToASingleVideoWithCanvasCaptureStreamAndAudioContextMediaStreamDestination.html 6.3 KB

recordArbitraryMediaFragmentsToASingleVideoWithCanvasCaptureStreamAndAudioContextMediaStreamDestination.html 6.3 KB View Download

@#4 Including canonical usages of timestampOffset (with "segments" mode) at https://www.w3.org/TR/media-source/#examples might be helpful.

@#5 thank you for the additional context.

First, regarding the tab crash (which I believe this crbug should continue to track):
On a local tip-of-tree Chromium debug build with proprietary codecs included, I repro'ed a DCHECK failure (on release builds, I didn't get a good stack trace yet, but in one attempt there was memory corruption detected)

[1:1:0309/122751.321056:FATAL:html_video_element_capturer_source.cc(139)] Check failed: canvas_->imageInfo().width() == resolution.width (768 vs. 480)

--> dalecurtis@ to assist in triage of what appears to be a Blink->MediaStream->CaptureFromElement issue. I'll close the original report on the MSE spec GH, since this is a crbug.

Second: Regarding the applicability/usability/ergonomics of the HTML5 media API for use in offline decode, capture, and splicing - that is an interesting use case (though not best tracked by this crbug.
'segments' mode bufferes appended media directly at the media's timestamps as offset by timestampOffset, and as cropped by appendWindowStart and appendWindowEnd. Those three attributes can only be adjusted when !(SourceBuffer.updating || SourceBuffer is in the middle of parsing a media segment) -- the SourceBuffer must also be sane (attached to a MediaSource, etc..). So long as the API hasn't reached a fatal error state, you can query the 'updating' state of the SourceBuffer directly (and there are associated events). You can force the reset of the parser to ensure it's not in the middle of parsing a media segment by using the SourceBuffer.abort() operation. Using this guidance, you shouldn't encounter synchronous errors adjusting those three attributes.

When appending your captured fragments of blob playbacks standalone to MSE (e.g. just 1 "chunk" in your original example), you can example the resulting buffered ranges to assist debugging what timestampOffset adjustment might have been best used prior to the append of that segment. This may help determine a general algorithm that works for you. Note that if some of your captures have only a subset of two tracks (e.g. just video or just audio), then the intersected buffered range will show a gap. Playback of such would stall at that gap  -- if this is indeed the case then you'll need to append silence or no-op video to fill the gap.

I'm also a bit confused on the use case: is it meant to accommodate varying mime-type/codec media fragements' concatenation into a single resulting stream for later use? Or is there some other purpose.

I feel the latter part of this post should be continued on https://github.com/w3c/media-source/issues/190, as it is more of a spec use case question.

Capture issues may be related to  issue 819914 .

@#7 Will try to parse the first part of https://bugs.chromium.org/p/chromium/issues/detail?id=820489#c7 and compose appropriate code.

The technical overview, debugging and provided feedback are helpful.

Adjusting the code to use a different <video> element at MediaSource portion of code does not crash tab. 

However, the results are inconsistent; the video playback infrequently stop at stalled event; also trying to record the media after MediaSource endOfStream results in GET blob:https://run.plnkr.co/1908afd7-5f13-4306-b620-0380313b505a 416 (Requested Range Not Satisfiable) when setting resulting Blob as value passed to URL.createObjectURL() set at a different <video> element than <video> element where MediaSource is set as resource. Getting to that point requires media playback no less than three times - and still the video is not recorded using captureStream.

Deleted: recordHTMLMediaElementWithSrcSetToMediaSourceWithMediaRecorder-1-1.html 7.9 KB

recordHTMLMediaElementWithSrcSetToMediaSourceWithMediaRecorder-1-1.html 7.9 KB View Download

If Chromium wont fix the production of seekable WebM files with Chromium implementation MediaRecorder API, why does the use of MediaRecorder throw an error when recording files?

From chrome://media-internals

Timestamp    Property    Value
00:00:00 773	error	 Failed to reconcile encoded audio times with decoded output.
00:00:09 503	error	 Failed to send video packet for decoding: timestamp=4167000 duration=0 size=734 side_data_size=0 is_key_frame=0 encrypted=0 discard_padding (ms)=(0, 0)
00:00:09 503	error	 video decode error
00:00:09 575	pipeline_error	PIPELINE_ERROR_DECODE

Or is there another reason calling HTMLMediaElement.captureStream() where the media resource provider is a MediaSource attached to a HTMLMediaElement crashes the tab?

Same on windows. It works 1-2 times and then crash the tab.