I tweaked my own recording test app according to mimic the reported behavior, but I don't see the problem on my Linux machine. In theory you should be able to
1. Load the attached app
2. Start and choose record tab
3. Pick a tab and do a recording
4. Stop and download the file.

Can the reporter test this app?

Deleted: RecordingApp.zip 887 KB

RecordingApp.zip 887 KB Download

I've installed the RecordingApp through the chrome extensions tab.
On Mac Chrome 56 it works and I can play the webm and download, size and fps look good.
On Mac Chrome 57 the app crashes after about 2 to 3 seconds of recording.

Deleted: Screen Shot 2017-03-13 at 2.46.47 PM.png 33.4 KB

	Screen Shot 2017-03-13 at 2.46.47 PM.png 33.4 KB View Download

Btw we tried the Screen Recorder app from the web store (https://chrome.google.com/webstore/detail/screen-recorder/gdamcnkmddbfhaadidkhahllkabienpk) on Mac Chrome 57 and it crashes also after a few seconds.

Please find the bisect result below :


You are probably looking for a change made after 441318 (known good), but no lat
er than 441319 (first known bad).
CHANGELOG URL:
  https://chromium.googlesource.com/chromium/src/+log/db04bc3039a04e5a26c13cfe63818b3f170f01cf..5740fa8801d59cfd99bac735ef483f1360e11492


Note : Given this is regression which started in M57, I am not 100% sure if this would block M57 release since this doesn't break anything but a bad caputure. Plese correct me if I am wrong.

Thanks for the updates.
Please understand that a 'bad capture' is a big deal for our users, stuck in Chrome 57.

Can you work around the problem by using maxWidth:window.screen.width, maxHeight:window.screen.height ?

The bisect (comment 4) doesn't make any sense: That change wouldn't have affected the capture resolution or scaling. The only thing it could effect is the timing of when the first frame is captured.

Perhaps the recording impl broke due to the pattern of the video resolution changing? Seems like mcasas@ or emircan@ should take a look at this.

Given the fact that this causes crashing, making it RBS for M58.

Any chance of getting a crash Id (chrome://crashes)

Crash ID 1480ffbb-588b-4cd5-90af-c25b71429354 (Server ID: 508efce480000000)

This is for the RecordingApp crash. The original case doesn't crash but gives the funky  video.

>> Can you work around the problem by using maxWidth:window.screen.width, maxHeight:window.screen.height ?

Not sure I understand how that would help?

I tried it, and it is not bad if the tab is (almost) fullscreen, but there is still a size pop of letter boxing due to the window tabs at the top.
But for smaller tabs where the tab width/height ratio is not close to the screen width/height ratio one still gets a significant size change (distortion) about 2 seconds into the recorded video (like attached video on the original report).

Btw during screen capture we also show a small video feedback (via URL.createObjectURL from the captured stream) and that stays in the proper ratio. So it looks MediaRecorder related.

Cheers.

The crash might be a different problem, but here's an ID: 51f602e480000000

Why do we end up in VEA for a Vp8 recording?

The crash and ratio problem has one thing in common, something strange happens after ~2secs.

According to [1], only a few cases of this crash have shown in the
last Canaries, so marking it RBS might be overestimating it (or 
am I looking at the wrong crash stack?).  

It seems to be something funky in VEA, probably we're trying
to re-create it because of a resolution change? (around [2] ?)
emircan@, wdyt?


[1] 
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27content%3A%3A%60anonymous%20namespace%5C%27%3A%3AVEAEncoder%3A%3AEncodeOnEncodingTaskRunner%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion

[2] https://cs.chromium.org/chromium/src/content/renderer/media_recorder/video_track_recorder.cc?sq=package:chromium&type=cs&l=834

If I disable hardware encode (using decode flag...) I can repro the problem initially reported. You need to make the tab quite narrow to easily see it.

 Issue 700855  has been merged into this issue.

Re 13, I cannot access crash link right now, but lets make a new bug about crash as there are multiple issues here. [2] doesn't point to VEA but VPX, and the initial report is VP9 on Mac, so VEA wouldn't be related to the scaling issue. 

I quickly ran the scaling issue by changing niklase@s extention to use vp9. It seems to be related to below lines where resolution changes. It seems to be handled correctly, but maybe the WebmMuxer is not handling it properly since we initialize segent based on the first frame's size, see https://cs.chromium.org/chromium/src/media/muxers/webm_muxer.cc?rcl=51fad280650c0155706ba734848b1e50864893ed&l=146.

[91690:775:0313/232445.103888:ERROR:video_track_recorder.cc(1207)] InitializeEncoder1280x1280
[91690:31239:0313/232445.113987:ERROR:video_track_recorder.cc(805)] EncodeOnEncodingTaskRunner
...
[91690:31239:0313/232448.188271:ERROR:video_track_recorder.cc(805)] EncodeOnEncodingTaskRunner
[91690:31239:0313/232448.188318:ERROR:video_track_recorder.cc(872)] Destroying/Re-Creating encoder for new frame size: 1280x1280 --> 1280x710 vp9
[91690:31239:0313/232448.295343:ERROR:video_track_recorder.cc(805)] EncodeOnEncodingTaskRunner 

gdehan@: I have tried playing back your video in the issue
description.  If I play it inside Chrome, I see a (the) strange
stretching, but this doesn't happen in my VLC Mac.  Webm container
is agnostic re. changes in the Video Frame resolution. It's the
player that can handle gracefully frame resolution changes or
not.  Can you try reproducing with an external VLC and write
back your experiences here?


> Also the framerate is not set.

Correct, the frame rate is not set because is variable.  MSE
and other players derive it from 

... 
the individual frames/audio packets timestamps.

But that the resolution changes is the actual problem reported here. You don't need to rescale the tab to trigger this.

Also, gdehaan@ (sorry for the misspell before), could you plz
try other combinations of videoBitsPerSecond and codec type?
I.e. IIUC you are running

   mediaRecorder = new MediaRecorder(stream, {
         videoBitsPerSecond : 2500000,
         mimeType: 'video/webm; codecs=vp9'
    });

which essentially forces a very specific behaviour. I wonder what
would happen if you leave it fully unconstrained (should
boil down to H264/AV1 in Mac), or command 'codecs=VP8', or
if you were to increase the encode bitrate.

Let's keep this bug focused on the resolution change. miu@, after your CL it seems like tab capture gets a change in resolution after 2 seconds. I've verified this by removing the recorder completely from my test app. is that expected?

@mcasas: The video plays in VLC, but it does change size, however it does maintain the correct aspect ratio (ie no pixel stretching).
This is part of a web app so we (want to) play it on Chrome. Actually, we convert the webm to mp4 with ffmpeg (v3.2.2 and 3.2.4) for browser compatibility and it can't handle the size change nor the variable fps (ie reads it as 1000fps so the standard output mp4 is rather big and doesn't play very well).
We tried using vp8, but I think it did not change the resolution change thing. We tested lots of combinations of maxWidth, etc, so I don't remember exactly, but it was not a workaround.

Just to confirm: we are not resizing the captured tab, the resolution change seems to be spontaneous. 

niklase: There is nothing broken w.r.t. tab capture here. The user is specifying a maxWidth and maxHeight, and no minWidth/Height. That is interpreted as "any resolution as long as it's less-than-or-equal-to maxWidth by maxHeight." WAI. :)

My change seems to be exposing an existing bug. Take a look at it: It's a very simple patch; and all it could be doing is causing extra frames to be captured. Speculating, maybe the "refresh frame" is causing an extra resolution change. Or, perhaps this is somehow affecting the logic that computes what the resolution should be, given how it measures system resource utilization; and so it is choosing something different than before. Nevertheless, it's not violating any API contract from what I can tell.

For those interested, here's how tab capture chooses resolutions: https://www.chromium.org/developers/design-documents/auto-throttled-screen-capture-and-mirroring

The question is whether this is a bug in the player or the recording API impl. The Chrome video player does a lot to remain compatible with all kinds of broken formats found across the web. So, I would tend to start looking at the newer feature first. ;-) Long ago, I brought up concerns about how resolution changes would be handled by the recording API, and there were issues with WebM not even having a "resolution change" concept; but I don't recall how we resolved this. Perhaps, on each resolution change, a new WebM segment must be started with the new resolution; just to make all players do the right thing?

For now, the user can work around this by specifying a minWidth and minHeight that are the same as the max values. Then, the resolution would not change. However, for windows smaller than 1280x1280, the capture impl will up-scale, which is a waste of system resources (larger data volume to handle, and no improvement in quality).

Looked a bit more at this. What happens is that for the first seconds we always get a video that has maxWidth x maxHeight dimensions, and then it scales down to the actual dimensions. miu@, even if this doesn't directly violate any spec, do you think this behaviour is what any user wants?

Also, is the throttling even active for the GetUserMedia use case? Is for example encode time wired up?

Hmm...The initial capture resolution should be either the user-provided max resolution, or if the tab content area is smaller, the size of the tab content area.

Based on your description, it seems that the size of the tab content area might not be known until after the first frame has been captured. If that's the case, the reason it takes "seconds" for the resolution to change is because the capture pipeline doesn't allow resolution changes to occur more often than once every 3 seconds. This is to avoid stressing encoders with a huge flood of size changes (e.g., when the user resizes the browser window), which would otherwise causes a flood of large-sized key frames to be emitted downstream.

This seems to explain why my change exposed this problem: The initial "refresh frame" is forcing a capture before tasks have run that provide the capture logic with the resolution of the tab content area.

So, it seems there are two things to fix:

1. The initial "refresh frame" capture should be delayed until the size of the tab content area is known (or maybe canceled?).
2. The Video Recorder needs to handle resolution changes.

I can take-on #1. Seems like mcasas@ should probably take #2?

Note: Even once #1 is fixed, #2 will still be an issue (e.g., if the user resizes the browser window while recording tab capture).

Agree. 2 is already done as well it can be (afaik), it's just all players doesn't support it. This issue also revealed a crash for uneven resolutions for Mediastream recorder with HW encode, but that's already fixed. 

This issue is also visible in Google Meet tab sharing. Encoding and decoding handles it, but we end up sending mega-sized frames initially - which might be a problem om lower bandwidth networks. It would be great to get this fixed quickly so we can merge to 58.

Re 2nd fix, I tried changing track size dynamically in MediaRecorder but it didn't help, see https://codereview.chromium.org/2750233003/.

#25.2: MediaRecorder _does_ support resolution changes
caveat: such resolution change implies destroying-(re)constructing
the encoder, so it's not something you want to do too often.  The 
recorded video in the issue description is perfectly playable as I
wrote in #17;  Chrome playback makes some strange assumptions, but
webm-wise everything looks OK as you can see by playing back in VLC.
#27: emircan@ WDYMean taht "it didn't help" ?

Re #28, I mean the output is still the same: playing in Chrome you see stretched video. I just wanted to note that so it is not tried again.

mcasas: Sure, Media Recording might be "bug free." However, can something be done to make it work on more players? Or, can something be done to fix Chrome playback of the video recorded with this API?

vigneshv@, do you have recommendation on how to set up webm files when there is mid-stream video resize?

chcunningham@, can we do something about #29/30? (When reproducing the
webm in the issue description, Chrome doesn't respect the resolution
change and distorts the output whereas VLC resizes OK).

> Re 2nd fix, I tried changing track size dynamically in MediaRecorder but it didn't help, see https://codereview.chromium.org/2750233003/.

This will certainly not help because once the headers are written out (it's done when the first frame is added), setting the track width and height has no effect.

> do you have recommendation on how to set up webm files when there is mid-stream video resize?

VP9 has a way of changing file sizes mid-stream. AFAIK, we don't specify that in the container. We merely put the first frame's size in the container and let the players deal with size changes (as these size changes will be known only *after* decoding the frame).

Just by looking at the code, it seems like Chromes should handle it fine because it uses the width/height values from the decoded frame to set he VideoFrame dimensions [1]. So it's possible that it could be that the downstream layers of code don't handle this change appropriately.

Chris or Dale might know more about this.

[1] https://cs.chromium.org/chromium/src/media/filters/vpx_video_decoder.cc?l=815

MSE requires you to append a new init segment [0] to describe the new frame size. Here's an example [1]

[0] https://www.w3.org/TR/mse-byte-stream-format-webm/#webm-init-segments
[1] http://w3c-test.org/media-source/mediasource-config-change-webm-v-framesize.html

Sorry, I realize your question in 32 is not specifically about MSE - I don't think midstream resolution changes for regular src=file.webm are really well specified. 

To be pedantic, when you describe a given frame size in the video track, and then the file suddenly doesn't have that frame size, I'm not sure if this qualifies as a spec-conforming webm video. 

I think Emerican's demo in comment #27 is probably a no-op in terms of what actually gets written to the file - IIUC he's changing the track size info after the track has already been written.

Could we support this sort of src= file... maybe - I'm not sure what all is involved yet. 

Vignesh - do you think its valid to change frame size in the middle of a webm file? Any way to explicitly signal this? Any idea if ffmpeg and/or libvpx support it?

Dale - do you have a stance on implicit midstream config changes for src= video?

OOC, why is a file not just a stream with a known end position? Why would we treat files any differently than a stream, except to provide a seek control (that would need to know the total duration of the video)?

Also, note that some src=URLs will be streams (i.e., the HTTP response will not provide a content-size header).

IIUC src=file vs stream is not an important distinction here. If you _stream_ webm that claims in its track info to be a certain size, and then actual send frames of a different size, you experience the same troubles described above. 

Perhaps I'm missing your intent.

Or by "stream", are you really asking about MSE "streaming" vs src=?

> do you think its valid to change frame size in the middle of a webm file? Any way to explicitly signal this? Any idea if ffmpeg and/or libvpx support it?

Yes, it is valid for frame size to change in the middle of the WebM file. There is no way to signal this in the container. Both ffmpeg and libvpx can handle such streams.

Also, since this issue is likely because of the way vpx_video_decoder wrapper handles it, i don't think the behavior would be different between MSE and <video> tag paths.

Also, like i pointed out before, to me it looks like Chromium is in fact handling it correctly. We are setting the size of the VideoFrame based on the size of the *decoded* frame and not based on the container in vpx_video_decoder wrapper.

So as long as the downstream layers use the size of the VideoFrame to render, this should be handled just fine. So it looks like the issue is there.

Apologies for missing your comments above! 

I notice that the video frame gets created with the width and height from the decode output, but the natural size is still being set from the config [0]. Will see what happens if we update this to use r_w/r_h (Vignesh, yell if thats not right)

[0] https://cs.chromium.org/chromium/src/media/filters/vpx_video_decoder.cc?rcl=1852c3e4976e2ac0473350f8aa4589e753926934&l=817

Early findings:

Change proposed in 42 *mostly* fixes it. There's a very brief flash of the stretched image at the transition boundary, but then its back to letterbox.

I also notice we're hitting the dcheck (with and without my change)
[1:13:0317/180556.371567:FATAL:gpu_memory_buffer_video_frame_pool.cc(377)] Check failed: gfx::Rect(video_frame->coded_size()).Contains(gfx::Rect(output)).

Will spend more time on Monday

We have a test file that changes frame sizes [1]. I thought Chrome used to have more test files that would change sizes as well.


[1] https://chromium.googlesource.com/chromium/src/media/+/refs/heads/master/test/data/frame_size_change-av_enc-v.webm

Should this be bumped to M59?

Let's keep this for 58 for now for the issue mentioned in #25. miu@, assigning to you for that change unless you want to keep a separate bug for that. 

I'm working on a fix. Have it mostly working as described in 44. 

We need to fork this issue... Chris, you're working on a playout change right? We still want to fix the problem that we always get max resolution for the first 2 seconds and then it falls down to native. This is undesirable for for example Hangouts even if it's not a problem for recorder/playout. I'll create a separate bug for that then.

Sounds good. I'll take a look at the forked bug...

> Chris, you're working on a playout change right?

Right

> We have a test file that changes frame sizes

I now realize that the way vpx_video_decoder fixes natural size is intentional. Chrome long ago made the decision to take the natural size from the container metadata and scale the frames to match that. 
https://bugs.chromium.org/p/chromium/issues/detail?id=117629

We can re-evaluate that stance, but changing it does have some interesting side effects. Check out scaling [0] vs not [1] below.

[0] https://storage.googleapis.com/chcunningham-chrome-shared/with_scaling_frame_size_change.webm

[1] https://storage.googleapis.com/chcunningham-chrome-shared/without_scaling_frame_size_change.webm


Should we just stick with scaling? 

Do we want to support video with frame size changes that also vary the aspect ratio (like the one from reporter)?

To be clear, scaling => stretching whenever frames aspect ratio does not match that of containers natural size (exactly whats happening in reporters video). But I personally think letting the video bounce around by letting natural size fluctuate is the worse of two evils. 

+1 scaling it is. I didn't realize your change would affect this.

In that case, from playback POV this is working as intended. niklase@, other issues are tracked separately right? OK to close this?

niklase@ - ping for comment 54. 

OK, fine to close.