Video layer / surface transforms are weird for rotated MP4 video |
|
Issue descriptionMP4 metadata can specify a video rotation (see media/base/video_rotation.h). e.g. ffmpeg.exe -i input.mp4 -metadata:s:v rotate=90 -vcodec copy -acodec copy output.mp4 Both VideoLayerImpl and VideoFrameResourceProvider (used for video surfaces) build the quad and transform this way: 1) Transform = rotation + translation to bring the video back into bounds 2) Quad rect (and visible rect etc.) is built from rotated size <- this is weird because the transform has the rotation already so it doesn't intuitively make sense for the quad rect to also be rotated pre-transform There's also a squashing factor applied by some code I haven't figured out yet (which is why it works correctly in the end - details below). VideoLayerImpl: https://cs.chromium.org/chromium/src/cc/layers/video_layer_impl.cc?rcl=6c58db1772c85aae93604b11906d6576671814d5&l=124 VideoFrameResourceProvider: https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/graphics/video_frame_resource_provider.cc?g=0&l=75 Both call into VideoResourceUpdater to append quads: https://cs.chromium.org/chromium/src/media/renderers/video_resource_updater.cc?rcl=51276962033b3e9b02804d178c829e88e6849c6d&l=424 So for example, 1920x1080 video content generates a quad of ~1080x1920 and the final matrix looks like this (row major): (0, 0.55, 0, t_x); (-1.72, 0, 0, t_y); (0, 0, 1, 0); (0, 0, 0, 1) Above t_x and t_y are x and y translation. The plain 90deg rotation matrix would have 1 instead of 0.55 and 1.72. The squashing factor is what makes it non-unitary. So the applying the matrix on the 1080x1920 quad produces 0.55 * 1920 x 1.72 * 1088 which is 1088x1920 again so it's effectively equivalent to an identity matrix for this case. Now the problem is that for direct composition we'd like to get a transform from video's content space (1920x1080) to screen space, but it looks like quad_to_target_transform is not that. However, the rendering result is correct, and that's because of this piece of code which was meant to do scaling, but also handles axes swap by accident: https://cs.chromium.org/chromium/src/gpu/ipc/service/direct_composition_surface_win.cc?rcl=51276962033b3e9b02804d178c829e88e6849c6d&l=881 I think we can make the transform look like it intuitively makes sense, but I need to figure out where the squashing happens (probably happens to preserve video aspect ratio somehow).
,
Nov 10
For video surfaces, the squashing happens here: https://cs.chromium.org/chromium/src/components/viz/service/display/surface_aggregator.cc?rcl=81dfa8e54a460006adeb46cdcdd18f15dc2b6cc3&l=268 gfx::Transform scaled_quad_to_target_transform( source_sqs->quad_to_target_transform); scaled_quad_to_target_transform.Scale(SK_MScalar1 / layer_to_content_scale_x, SK_MScalar1 / layer_to_content_scale_y); This scale is a pre-scale i.e. scale the content first before applying the quad_to_target_transform, but the scale factor is in surface layer's target space. I think if we changed this to a post-scale (i.e. ConcatTransform(scale_transform), then we could build the video quad's rect and transform in the intuitive way that we expect it to work.
,
Nov 12
The quad rect and transform for VideoLayerImpl are as you'd expect i.e. rect is in content space pre-transform (ex. 1920x1080), the transform is a plain rotation + translation, and there's no squashing at any stage.
This is because in the following code, |rotated_size| is meant to be in pre-rotation space e.g. 1920x1080, and bounds() is post-rotation e.g. 1080x1920:
void VideoLayerImpl::AppendQuads(viz::RenderPass* render_pass,
AppendQuadsData* append_quads_data) {
DCHECK(frame_.get());
gfx::Transform transform = DrawTransform();
gfx::Size rotated_size = bounds();
switch (video_rotation_) {
case media::VIDEO_ROTATION_90:
rotated_size = gfx::Size(rotated_size.height(), rotated_size.width());
transform.Rotate(90.0);
transform.Translate(0.0, -rotated_size.height());
break;
case media::VIDEO_ROTATION_180:
transform.Rotate(180.0);
transform.Translate(-rotated_size.width(), -rotated_size.height());
break;
case media::VIDEO_ROTATION_270:
rotated_size = gfx::Size(rotated_size.height(), rotated_size.width());
transform.Rotate(270.0);
transform.Translate(-rotated_size.width(), 0);
break;
case media::VIDEO_ROTATION_0:
break;
}
...
The opposite holds true for VideoFrameResourceProvider i.e. rotated_size is post-rotation e.g. 1080x1920:
void VideoFrameResourceProvider::AppendQuads(
viz::RenderPass* render_pass,
scoped_refptr<media::VideoFrame> frame,
media::VideoRotation rotation,
bool is_opaque) {
TRACE_EVENT0("media", "VideoFrameResourceProvider::AppendQuads");
DCHECK(resource_updater_);
DCHECK(resource_provider_);
gfx::Transform transform = gfx::Transform();
gfx::Size rotated_size = frame->coded_size();
switch (rotation) {
case media::VIDEO_ROTATION_90:
rotated_size = gfx::Size(rotated_size.height(), rotated_size.width());
transform.Rotate(90.0);
transform.Translate(0.0, -rotated_size.height());
break;
case media::VIDEO_ROTATION_180:
transform.Rotate(180.0);
transform.Translate(-rotated_size.width(), -rotated_size.height());
break;
case media::VIDEO_ROTATION_270:
rotated_size = gfx::Size(rotated_size.height(), rotated_size.width());
transform.Rotate(270.0);
transform.Translate(-rotated_size.width(), 0);
break;
case media::VIDEO_ROTATION_0:
break;
}
...
The fix is simple:
1) Fix VideoFrameResourceProvider to produce quads with pre-rotation rect and transform containing the rotation.
2) Fix SurfaceAggregator's scaling to a post-scale instead of pre-scale.
,
Nov 12
FWIW video layer's bounds comes from PipelineMetadata::natural_size which has video rotation applied to it according to the comments here: https://cs.chromium.org/chromium/src/media/base/pipeline_metadata.h?rcl=3fbba49dc9e193e72b4c8680776404803b2d7a92&l=33
,
Nov 13
Turns out even my last observation was incorrect. The actual cause of the bug is that VideoFrameSubmitter sets the output_rect of its RenderPass to the pre-rotation size which is then used by SurfaceAggregator to calculate scale factors. I fixed that (and the quad rect), and it worked!
,
Nov 15
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1dfd4418faf858a63bd901ef0485c7e54e5e69fb commit 1dfd4418faf858a63bd901ef0485c7e54e5e69fb Author: Sunny Sachanandani <sunnyps@chromium.org> Date: Thu Nov 15 20:29:28 2018 Fix video quad rect and surface output rect Video surface output rect didn't take rotation into account so when SurfaceAggregator would stretch content to fill bounds, it would end up with the wrong scaling factors (squash in one dimension and expand in another) since the surface quad's bounds (in the embedder) were rotated. To work around this (or by accident), VideoFrameResourceProvider was passing the rotated size as the quad's rect which doesn't make sense intuitively. This also worked by accident with DirectComposition overlays because of applying another scaling which would fix the aspect ratio again. This change makes it possible to use the |quad_to_target_transform| as is, without having to apply an aspect ratio correcting scaling, and to assume that the quad's rect is in the pre-transform space. Bug: 904035 Change-Id: Ia55e44f1f2b49b8d368a97af54f3ce9d90a81234 Reviewed-on: https://chromium-review.googlesource.com/c/1334971 Reviewed-by: enne <enne@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Frank Liberato <liberato@chromium.org> Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> Cr-Commit-Position: refs/heads/master@{#608495} [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/cc/layers/video_layer_impl.cc [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/media/renderers/video_resource_updater.cc [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/media/renderers/video_resource_updater.h [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/third_party/blink/renderer/platform/graphics/video_frame_resource_provider.cc [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/third_party/blink/renderer/platform/graphics/video_frame_submitter.cc [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/third_party/blink/renderer/platform/graphics/video_frame_submitter.h [modify] https://crrev.com/1dfd4418faf858a63bd901ef0485c7e54e5e69fb/third_party/blink/renderer/platform/graphics/video_frame_submitter_test.cc
,
Nov 15
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/de12bc444834cea208a809629ca347fcec39e93f commit de12bc444834cea208a809629ca347fcec39e93f Author: Ella Ge <eirage@chromium.org> Date: Thu Nov 15 20:51:44 2018 Revert "Fix video quad rect and surface output rect" This reverts commit 1dfd4418faf858a63bd901ef0485c7e54e5e69fb. Reason for revert: <INSERT REASONING HERE> Original change's description: > Fix video quad rect and surface output rect > > Video surface output rect didn't take rotation into account so when > SurfaceAggregator would stretch content to fill bounds, it would end up > with the wrong scaling factors (squash in one dimension and expand in > another) since the surface quad's bounds (in the embedder) were rotated. > > To work around this (or by accident), VideoFrameResourceProvider was > passing the rotated size as the quad's rect which doesn't make sense > intuitively. This also worked by accident with DirectComposition > overlays because of applying another scaling which would fix the aspect > ratio again. > > This change makes it possible to use the |quad_to_target_transform| as > is, without having to apply an aspect ratio correcting scaling, and to > assume that the quad's rect is in the pre-transform space. > > Bug: 904035 > Change-Id: Ia55e44f1f2b49b8d368a97af54f3ce9d90a81234 > Reviewed-on: https://chromium-review.googlesource.com/c/1334971 > Reviewed-by: enne <enne@chromium.org> > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Reviewed-by: Frank Liberato <liberato@chromium.org> > Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> > Cr-Commit-Position: refs/heads/master@{#608495} TBR=kbr@chromium.org,enne@chromium.org,sunnyps@chromium.org,liberato@chromium.org Change-Id: I99727abc125468581caa0cf9e2ddea74351dd1f9 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 904035 Reviewed-on: https://chromium-review.googlesource.com/c/1338206 Reviewed-by: Ella Ge <eirage@chromium.org> Commit-Queue: Ella Ge <eirage@chromium.org> Cr-Commit-Position: refs/heads/master@{#608501} [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/cc/layers/video_layer_impl.cc [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/media/renderers/video_resource_updater.cc [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/media/renderers/video_resource_updater.h [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/third_party/blink/renderer/platform/graphics/video_frame_resource_provider.cc [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/third_party/blink/renderer/platform/graphics/video_frame_submitter.cc [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/third_party/blink/renderer/platform/graphics/video_frame_submitter.h [modify] https://crrev.com/de12bc444834cea208a809629ca347fcec39e93f/third_party/blink/renderer/platform/graphics/video_frame_submitter_test.cc
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/56f3274d1545526815280622044716606e869e62 commit 56f3274d1545526815280622044716606e869e62 Author: Sunny Sachanandani <sunnyps@chromium.org> Date: Fri Nov 16 00:18:39 2018 Reland "Fix video quad rect and surface output rect" This is a reland of 1dfd4418faf858a63bd901ef0485c7e54e5e69fb Original change's description: > Fix video quad rect and surface output rect > > Video surface output rect didn't take rotation into account so when > SurfaceAggregator would stretch content to fill bounds, it would end up > with the wrong scaling factors (squash in one dimension and expand in > another) since the surface quad's bounds (in the embedder) were rotated. > > To work around this (or by accident), VideoFrameResourceProvider was > passing the rotated size as the quad's rect which doesn't make sense > intuitively. This also worked by accident with DirectComposition > overlays because of applying another scaling which would fix the aspect > ratio again. > > This change makes it possible to use the |quad_to_target_transform| as > is, without having to apply an aspect ratio correcting scaling, and to > assume that the quad's rect is in the pre-transform space. > > Bug: 904035 > Change-Id: Ia55e44f1f2b49b8d368a97af54f3ce9d90a81234 > Reviewed-on: https://chromium-review.googlesource.com/c/1334971 > Reviewed-by: enne <enne@chromium.org> > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Reviewed-by: Frank Liberato <liberato@chromium.org> > Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> > Cr-Commit-Position: refs/heads/master@{#608495} TBR=enne@chromium.org,kbr@chromium.org,liberato@chromium.org Bug: 904035 Change-Id: I278a0ea2c5507f01ea13b31d2090b5f99537c7c3 Reviewed-on: https://chromium-review.googlesource.com/c/1338418 Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org> Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> Cr-Commit-Position: refs/heads/master@{#608596} [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/cc/layers/video_layer_impl.cc [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/media/renderers/video_resource_updater.cc [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/media/renderers/video_resource_updater.h [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/third_party/blink/renderer/platform/graphics/video_frame_resource_provider.cc [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/third_party/blink/renderer/platform/graphics/video_frame_submitter.cc [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/third_party/blink/renderer/platform/graphics/video_frame_submitter.h [modify] https://crrev.com/56f3274d1545526815280622044716606e869e62/third_party/blink/renderer/platform/graphics/video_frame_submitter_test.cc
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/957673e4c587f72aff5fde1a8b9f46597c1a97d5 commit 957673e4c587f72aff5fde1a8b9f46597c1a97d5 Author: Kentaro Hara <haraken@chromium.org> Date: Fri Nov 16 03:46:19 2018 Revert "Reland "Fix video quad rect and surface output rect"" This reverts commit 56f3274d1545526815280622044716606e869e62. Reason for revert: This broke VideoFrameSubmitterTest on Linux ChromiumOS MSan Tests. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20ChromiumOS%20MSan%20Tests/9636 Original change's description: > Reland "Fix video quad rect and surface output rect" > > This is a reland of 1dfd4418faf858a63bd901ef0485c7e54e5e69fb > > Original change's description: > > Fix video quad rect and surface output rect > > > > Video surface output rect didn't take rotation into account so when > > SurfaceAggregator would stretch content to fill bounds, it would end up > > with the wrong scaling factors (squash in one dimension and expand in > > another) since the surface quad's bounds (in the embedder) were rotated. > > > > To work around this (or by accident), VideoFrameResourceProvider was > > passing the rotated size as the quad's rect which doesn't make sense > > intuitively. This also worked by accident with DirectComposition > > overlays because of applying another scaling which would fix the aspect > > ratio again. > > > > This change makes it possible to use the |quad_to_target_transform| as > > is, without having to apply an aspect ratio correcting scaling, and to > > assume that the quad's rect is in the pre-transform space. > > > > Bug: 904035 > > Change-Id: Ia55e44f1f2b49b8d368a97af54f3ce9d90a81234 > > Reviewed-on: https://chromium-review.googlesource.com/c/1334971 > > Reviewed-by: enne <enne@chromium.org> > > Reviewed-by: Kenneth Russell <kbr@chromium.org> > > Reviewed-by: Frank Liberato <liberato@chromium.org> > > Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#608495} > > TBR=enne@chromium.org,kbr@chromium.org,liberato@chromium.org > > Bug: 904035 > Change-Id: I278a0ea2c5507f01ea13b31d2090b5f99537c7c3 > Reviewed-on: https://chromium-review.googlesource.com/c/1338418 > Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org> > Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> > Cr-Commit-Position: refs/heads/master@{#608596} TBR=kbr@chromium.org,enne@chromium.org,sunnyps@chromium.org,liberato@chromium.org Change-Id: I55b589ea5325317bc271928d9d2b388bec344174 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 904035 Reviewed-on: https://chromium-review.googlesource.com/c/1338586 Reviewed-by: Kentaro Hara <haraken@chromium.org> Commit-Queue: Kentaro Hara <haraken@chromium.org> Cr-Commit-Position: refs/heads/master@{#608653} [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/cc/layers/video_layer_impl.cc [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/media/renderers/video_resource_updater.cc [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/media/renderers/video_resource_updater.h [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/third_party/blink/renderer/platform/graphics/video_frame_resource_provider.cc [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/third_party/blink/renderer/platform/graphics/video_frame_submitter.cc [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/third_party/blink/renderer/platform/graphics/video_frame_submitter.h [modify] https://crrev.com/957673e4c587f72aff5fde1a8b9f46597c1a97d5/third_party/blink/renderer/platform/graphics/video_frame_submitter_test.cc
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1ab59f89eaa1967589498e8338e4e36b57347fdf commit 1ab59f89eaa1967589498e8338e4e36b57347fdf Author: Sunny Sachanandani <sunnyps@chromium.org> Date: Fri Nov 16 20:28:31 2018 Reland "Reland "Fix video quad rect and surface output rect"" This is a reland of 56f3274d1545526815280622044716606e869e62 Fix uninitialized |rotation_| in VideoFrameSubmitter. Original change's description: > Reland "Fix video quad rect and surface output rect" > > This is a reland of 1dfd4418faf858a63bd901ef0485c7e54e5e69fb > > Original change's description: > > Fix video quad rect and surface output rect > > > > Video surface output rect didn't take rotation into account so when > > SurfaceAggregator would stretch content to fill bounds, it would end up > > with the wrong scaling factors (squash in one dimension and expand in > > another) since the surface quad's bounds (in the embedder) were rotated. > > > > To work around this (or by accident), VideoFrameResourceProvider was > > passing the rotated size as the quad's rect which doesn't make sense > > intuitively. This also worked by accident with DirectComposition > > overlays because of applying another scaling which would fix the aspect > > ratio again. > > > > This change makes it possible to use the |quad_to_target_transform| as > > is, without having to apply an aspect ratio correcting scaling, and to > > assume that the quad's rect is in the pre-transform space. > > > > Bug: 904035 > > Change-Id: Ia55e44f1f2b49b8d368a97af54f3ce9d90a81234 > > Reviewed-on: https://chromium-review.googlesource.com/c/1334971 > > Reviewed-by: enne <enne@chromium.org> > > Reviewed-by: Kenneth Russell <kbr@chromium.org> > > Reviewed-by: Frank Liberato <liberato@chromium.org> > > Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#608495} > > TBR=enne@chromium.org,kbr@chromium.org,liberato@chromium.org > > Bug: 904035 > Change-Id: I278a0ea2c5507f01ea13b31d2090b5f99537c7c3 > Reviewed-on: https://chromium-review.googlesource.com/c/1338418 > Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org> > Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> > Cr-Commit-Position: refs/heads/master@{#608596} TBR=enne@chromium.org Bug: 904035 Change-Id: I0f94cd8d71c7c9af60d3d57ac593d2f509e566b0 Reviewed-on: https://chromium-review.googlesource.com/c/1339200 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org> Cr-Commit-Position: refs/heads/master@{#608923} [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/cc/layers/video_layer_impl.cc [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/media/renderers/video_resource_updater.cc [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/media/renderers/video_resource_updater.h [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/third_party/blink/renderer/platform/graphics/video_frame_resource_provider.cc [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/third_party/blink/renderer/platform/graphics/video_frame_submitter.cc [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/third_party/blink/renderer/platform/graphics/video_frame_submitter.h [modify] https://crrev.com/1ab59f89eaa1967589498e8338e4e36b57347fdf/third_party/blink/renderer/platform/graphics/video_frame_submitter_test.cc
,
Dec 4
|
|
►
Sign in to add a comment |
|
Comment 1 by zmo@chromium.org
, Nov 10Status: Assigned (was: Untriaged)