New issue
Advanced search Search tips

Issue 718823 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: ----



Sign in to add a comment

getUserMedia interprets differently video width depending on phone orientation

Project Member Reported by beaufort...@gmail.com, May 5 2017

Issue description

Chrome: 60.0.3089.0
Device name: Google Pixel Phone 7.1.2

Steps to reproduce:
1. Hold device in landscape mode
2. Go to https://beaufortfrancois.github.io/sandbox/media/hd.html
3. getUserMedia Constraints are 1920x1080 and I get a beautiful 1920x1080 video. Yeah!
4. Hold device in portrait mode now
5. New getUserMedia constraints are 1080x1920...

Expected result:
I should get 1080x1920 video

Actual result:
I get a 1080x1080 video

Note that Chrome Stable (58.0.3029.83) currently returns smaller videos (640x480 and 480x640).
 
Cc: mcasas@chromium.org
Components: Blink>GetUserMedia>Webcam
This doesn't look like a bug.

If your constraints say {height: 1920, width: 1080} and the resolution is 1080x1920, you will get 1080x1080 since you say you want width closest to 1080 (you get it) and height closest to 1920 (you get 1080, which is as close as you can get).

Perhaps you can try something like:
{advanced: [{height: 1920, width: 1080}, {height:1080, width: 1920}]}

Then it will first try exactly 1920x1080, if not possible, exactly 1080x1920. If not possible, it will try the default, which is 640x480.

You can also use:

{basic: {height:1920, width:1080}, advanced: [{height: 1920, width: 1080}, {height:1080, width: 1920}]}

Note that these "naked" values are interpreted as ideal in the basic section and as exact in the advanced section.
It will first try 1920x1080 exact, then 1080x1920, then you'll get whatever is closest to 1920x1080.
You can add more advanced sections to try other resolutions.

Labels: -Pri-3 Pri-2
Owner: guidou@chromium.org
Status: Assigned (was: Unconfirmed)
I see that the problem is that 1080x1920 is not reported as a supported resolution.
On phones that do not report supported resolution we use a fallback list and that list only includes the landscape resolutions.
I'll prepare a patch to add portrait resolutions on Android.
I've updated https://beaufortfrancois.github.io/sandbox/media/hd.html to include track settings.
Here are some stats I've gathered on my Pixel phone:


Firefox 53.0.2
--------------

Portrait:

Video element: 768x1024
Track settings: 1024x768

Landscape:

Video element: 1024x768
Track settings: 1920x1080


Chrome Canary 60.0.3094.0
-------------------------

Portrait:

Video element: 1080x1080
Track settings: 1080x1080

Landscape:

Video element: 1920x1080
Track settings: 1920x1080


Note: Chrome Beta and Chrome Dev are the same results as Chrome Canary.


Chrome Stable 58.0.3029.83
--------------------------

Portrait:

Video element: 480x640
Track settings: N/A

Landscape:

Video element: 640x480
Track settings: N/A
Firefox Beta 54.0b6
-------------------

Portrait:

Video element: 1080x1920
Track settings: 1920x1080

Landscape:

Video element: 1920x1080
Track settings: 1920x1080

Comment 7 by guidou@chromium.org, May 10 2017

Please disregard my comment #3.

You should not use 1080x1920, but just use 1920x1080. The implementation should deal with the rotation.

Because you are using 1080x1920, the track settings in Firefox are 1024x768 and 1080x1080 in Chrome. Since 1080x1920 those are the closest resolutions they can give you, given that you specify a maximum width of 1080.

The bug in Chromium is in VideoTrackAdapter::CalculateTargetSize, which is not using the rotation to make the computation.
When the phone is rotated the input frame is 1080x1920, but CalculateTargetSize thinks the maximum it should be is 1920x1080 and crops it to 1080x1080.
The fix is to incorporate the rotation to CalculateTargetSize.

Comment 8 by guidou@chromium.org, May 10 2017

I tried to make a quick fix by looking at the rotation metadata in the frame, but they do not have the rotation in the metadata.

mcasas/chfremer: Do you know a good way to pass the rotation information (frame metadata or other mechanism) ?

Comment 9 by guidou@chromium.org, May 10 2017

I tried to make a quick fix by looking at the rotation metadata in the frame, but they do not have the rotation in the metadata.

mcasas/chfremer: Do you know a good way to pass the rotation information (frame metadata or other mechanism) ?
I believe the rotation information is available in VideoCaptureDeviceClient at [1], but is not included alongside the frame from there on its path to the consumers.

Where in the path does the constraint processing happen?

I guess one option would be to pass the information alongside the frame all the way from VideoCaptureDeviceClient. Assuming that the rotation information originates from a device API, would it be a viable alternative to have the constraint processing module obtain the rotation directly from the device API and assume that it matches with the rotation of incoming frames?

[1] https://cs.chromium.org/chromium/src/media/capture/video/video_capture_device_client.cc?dr=CSs&l=138
Thanks for the information. The rotation is not necessary for processing constraints,  but for processing frames after the track has started. 
The rotation information is required in VideoTrackAdapter. 
I'll try to come up with a patch tomorrow and will get back to you. 
After consulting with foolip@, passing the rotation in the frames would be wrong because that would mean the frames need to be rotated to be displayed correctly, which is not the case here.

Using a device API like chfremer@ suggests would be better, but I think I can produce something that works by passing the expected native resolution of the frames to the VideoTrackAdapter.
Project Member

Comment 14 by bugdroid1@chromium.org, May 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0ac21bec152a95d8edb8e759f765a512d552357c

commit 0ac21bec152a95d8edb8e759f765a512d552357c
Author: guidou <guidou@chromium.org>
Date: Tue May 16 10:18:51 2017

Detect frames from rotated devices in VideoTrackAdapter on Android.

On Android, when a device is rotated, frames from the camera arrive with
width and height swapped. The VideoTrackAdapter was unable to detect this
and instead cropped them incorrectly because it treated the rotated
dimension as a violation of the maximum size settings.

This CL provides a simple mechanism to detect this problem on Android.
If the frame arrives from the camera with width and height swapped from
what was expected, treat it as a valid frame that does not need to be
adjusted.

BUG= 718823 

Review-Url: https://codereview.chromium.org/2870413004
Cr-Commit-Position: refs/heads/master@{#472057}

[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_constraints_util.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_constraints_util.h
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_constraints_util_unittest.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_constraints_util_video_content.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_constraints_util_video_device.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_video_source.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_video_source_unittest.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/media_stream_video_track_unittest.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/video_track_adapter.cc
[modify] https://crrev.com/0ac21bec152a95d8edb8e759f765a512d552357c/content/renderer/media/video_track_adapter.h

Status: Fixed (was: Assigned)
Labels: Merge-Request-59
Project Member

Comment 17 by sheriffbot@chromium.org, May 22 2017

Labels: -Merge-Request-59 Hotlist-Merge-Approved Merge-Approved-59
Your change meets the bar and is auto-approved for M59. Please go ahead and merge the CL to branch 3071 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), gkihumba@(ChromeOS), Abdul Syed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 18 by bugdroid1@chromium.org, May 22 2017

Labels: -merge-approved-59 merge-merged-3071
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a110813cbb7d0f8ff2395a6a4138183e97946ea5

commit a110813cbb7d0f8ff2395a6a4138183e97946ea5
Author: Guido Urdaneta <guidou@chromium.org>
Date: Mon May 22 15:42:01 2017

Detect frames from rotated devices in VideoTrackAdapter on Android.

On Android, when a device is rotated, frames from the camera arrive with
width and height swapped. The VideoTrackAdapter was unable to detect this
and instead cropped them incorrectly because it treated the rotated
dimension as a violation of the maximum size settings.

This CL provides a simple mechanism to detect this problem on Android.
If the frame arrives from the camera with width and height swapped from
what was expected, treat it as a valid frame that does not need to be
adjusted.

BUG= 718823 

Review-Url: https://codereview.chromium.org/2870413004
Cr-Original-Commit-Position: refs/heads/master@{#472057}
Review-Url: https://codereview.chromium.org/2897993002 .
Cr-Commit-Position: refs/branch-heads/3071@{#652}
Cr-Branched-From: a106f0abbf69dad349d4aaf4bcc4f5d376dd2377-refs/heads/master@{#464641}

[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_constraints_util.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_constraints_util.h
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_constraints_util_unittest.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_constraints_util_video_content.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_constraints_util_video_device.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_video_source.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_video_source_unittest.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/media_stream_video_track_unittest.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/video_track_adapter.cc
[modify] https://crrev.com/a110813cbb7d0f8ff2395a6a4138183e97946ea5/content/renderer/media/video_track_adapter.h

Verified fix on Pixel/ N running Chrome "60.0.3107.3"
Labels: M-59

Sign in to add a comment