Issue metadata
Sign in to add a comment
|
GPU decoding of h264 mp4 with "Original Aspect Ratio" results in incorrect display aspect ratio
Reported by
dustin.k...@gmail.com,
Mar 29 2017
|
||||||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 Example URL: https://panomoments.com/u/dustinkerstein/m/grand-central Steps to reproduce the problem: 1. Enable Hardware-accelerated video decode 2. Go to https://panomoments.com/u/dustinkerstein/m/grand-central 3. Verify that the HD/SD qualities work correctly 4. Switch to UHD 5. Note that the image is distorted and shifted incorrectly 6. Disable Hardware-accelerated video decode and redo step 4 7. Note that it displays correctly What is the expected behavior? It should display the same way as the SD and HD qualities. What went wrong? Not sure when this broke (maybe 57 or 56) but it appears to only affect files that are being played back with Media Source Extensions and hardware video decoding. If I attempt to just drag the full video file into Chrome, it displays correctly (as a square): https://data.panomoments.com/processed/5849a51bc26d08000b62f9f8/58a5d20affe5e3000bdb34c0/uhd_dashinit.mp4 The main difference between the UHD and HD/SD files are that the UHD is encoded at a resolution of 3840x2160 using the following ffmpeg command: ffmpeg -y -framerate 1 -i ./%6d.jpg -c:v libx264 -g 1 -vf "scale=3840:2160" -crf 30 -pix_fmt yuv420p _uhd.mp4 This sets the "Original Aspect Ratio" = 1.0 so that the player knows that even though it's a non-square video, it should be played back using a 1:1 square aspect ratio. Feel free to compare the UHD file to the HD file: https://data.panomoments.com/processed/5849a51bc26d08000b62f9f8/58a5d20affe5e3000bdb34c0/hd_dashinit.mp4 Did this work before? Yes I think 56 but it could have been 55. Is it a problem with Flash or HTML5? N/A Does this work in other browsers? N/A Chrome version: 57.0.2987.110 Channel: stable OS Version: 10.0 Flash Version: Shockwave Flash 25.0 r0 Contents of chrome://gpu: Graphics Feature Status Canvas: Hardware accelerated Flash: Hardware accelerated Flash Stage3D: Hardware accelerated Flash Stage3D Baseline profile: Hardware accelerated Compositing: Hardware accelerated Multiple Raster Threads: Enabled Native GpuMemoryBuffers: Software only. Hardware acceleration disabled Rasterization: Software only, hardware acceleration unavailable Video Decode: Hardware accelerated Video Encode: Hardware accelerated VPx Video Decode: Software only, hardware acceleration unavailable WebGL: Hardware accelerated WebGL2: Hardware accelerated Driver Bug Workarounds clear_uniforms_before_first_program_use decode_encode_srgb_for_generatemipmap disable_discard_framebuffer disable_framebuffer_cmaa exit_on_context_lost force_cube_complete msaa_is_slow scalarize_vec_and_mat_constructor_args texsubimage_faster_than_teximage Problems Detected GPU rasterization should only be enabled on NVIDIA Pascal and Maxwell, Intel Broadwell+, and AMD RX-R2 GPUs for now.: 643850 Disabled Features: gpu_rasterization Accelerated VPx decoding is hanging on some videos.: 654111 Disabled Features: accelerated_vpx_decode Some drivers are unable to reset the D3D device in the GPU process sandbox Applied Workarounds: exit_on_context_lost TexSubImage is faster for full uploads on ANGLE Applied Workarounds: texsubimage_faster_than_teximage Clear uniforms before first program use on all platforms: 124764, 349137 Applied Workarounds: clear_uniforms_before_first_program_use Always rewrite vec/mat constructors to be consistent: 398694 Applied Workarounds: scalarize_vec_and_mat_constructor_args ANGLE crash on glReadPixels from incomplete cube map texture: 518889 Applied Workarounds: force_cube_complete On Intel GPUs MSAA performance is not acceptable for GPU rasterization: 527565 Applied Workarounds: msaa_is_slow Framebuffer discarding can hurt performance on non-tilers: 570897 Applied Workarounds: disable_discard_framebuffer Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198 Applied Workarounds: disable_framebuffer_cmaa Disable KHR_blend_equation_advanced until cc shaders are updated: 661715 Decode and Encode before generateMipmap for srgb format textures on Windows: 634519 Applied Workarounds: decode_encode_srgb_for_generatemipmap Native GpuMemoryBuffers have been disabled, either via about:flags or command line. Disabled Features: native_gpu_memory_buffers Version Information Data exported 3/29/2017, 1:04:50 PM Chrome version Chrome/57.0.2987.110 Operating system Windows NT 10.0.14393 Software rendering list version 12.13 Driver bug list version 9.29 ANGLE commit id c1a5d16e964a 2D graphics backend Skia/57 ae9cc5d3588d52f4b371b55845704b25d88cf06d Command Line Args Files (x86)\Google\Chrome\Application\chrome.exe" --flag-switches-begin --enable-touch-drag-drop --touch-events=enabled --flag-switches-end Driver Information Initialization time 220 In-process GPU false Passthrough Command Decoder false Sandboxed false GPU0 VENDOR = 0x8086, DEVICE= 0x0a16 Optimus false AMD switchable false Desktop compositing Aero Glass Diagonal Monitor Size of \\.\DISPLAY1 10.6" Diagonal Monitor Size of \\.\DISPLAY2 10.6" Diagonal Monitor Size of \\.\DISPLAY3 10.6" Diagonal Monitor Size of \\.\DISPLAY1 24.0" Driver vendor Intel Corporation Driver version 20.19.15.4331 Driver date 11-18-2015 Pixel shader version 5.0 Vertex shader version 5.0 Max. MSAA samples 8 Machine model name Machine model version GL_VENDOR Google Inc. GL_RENDERER ANGLE (Intel(R) HD Graphics Family Direct3D11 vs_5_0 ps_5_0) GL_VERSION OpenGL ES 3.0 (ANGLE 2.1.0.c1a5d16e964a) GL_EXTENSIONS GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc_srgb GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object Disabled Extensions GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent Window system binding vendor Google Inc. (adapter LUID: 0000000000007cad) Window system binding version 1.4 (ANGLE 2.1.0.c1a5d16e964a) Window system binding extensions EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource Direct rendering Yes Reset notification strategy 0x8252 GPU process crash count 0 Compositor Information Tile Update Mode One-copy Partial Raster Enabled GpuMemoryBuffers Status ATC Software only ATCIA Software only DXT1 Software only DXT5 Software only ETC1 Software only R_8 Software only RG_88 Software only BGR_565 Software only RGBA_4444 Software only RGBX_8888 Software only RGBA_8888 Software only BGRX_8888 Software only BGRA_8888 Software only YVU_420 Software only YUV_420_BIPLANAR Software only UYVY_422 Software only Diagnostics 0 b3DAccelerationEnabled true b3DAccelerationExists true bAGPEnabled true bAGPExistenceValid true bAGPExists true bCanRenderWindow true bDDAccelerationEnabled true bDriverBeta false bDriverDebug false bDriverSigned false bDriverSignedValid false bNoHardware false dwBpp 32 dwDDIVersion 12 dwHeight 1200 dwRefreshRate 59 dwWHQLLevel 0 dwWidth 1920 iAdapter 0 lDriverSize 35016288 lMiniVddSize 0 szAGPStatusEnglish Enabled szAGPStatusLocalized Enabled szChipType Intel(R) HD Graphics Family szD3DStatusEnglish Enabled szD3DStatusLocalized Enabled szDACType Internal szDDIVersionEnglish 12 szDDIVersionLocalized 12 szDDStatusEnglish Enabled szDDStatusLocalized Enabled szDXVAHDEnglish Supported szDXVAModes ModeMPEG2_A ModeMPEG2_C ModeWMV9_C ModeVC1_C szDescription Intel(R) HD Graphics Family szDeviceId 0x0A16 szDeviceIdentifier {D7B78E66-4956-11CF-4F67-192AB7C2D935} szDeviceName \\.\DISPLAY1 szDisplayMemoryEnglish 2160 MB szDisplayMemoryLocalized 2160 MB szDisplayModeEnglish 1920 x 1200 (32 bit) (59Hz) szDisplayModeLocalized 1920 x 1200 (32 bit) (59Hz) szDriverAssemblyVersion 20.19.15.4331 szDriverAttributes Final Retail szDriverDateEnglish 11/17/2015 8:00:00 PM szDriverDateLocalized 11/17/2015 20:00:00 szDriverLanguageEnglish English szDriverLanguageLocalized English szDriverModelEnglish WDDM 2.0 szDriverModelLocalized WDDM 2.0 szDriverName igdumdim64.dll,igd10iumd64.dll,igd10iumd64.dll,igd12umd64.dll szDriverNodeStrongName oem140.inf:5f63e5341cc65b69:iHSWM_w10:20.19.15.4331:pci\ven_8086&dev_0a16&subsys_0a161414 szDriverSignDate Unknown szDriverVersion 20.19.0015.4331 szKeyDeviceID Enum\PCI\VEN_8086&DEV_0A16&SUBSYS_0A161414&REV_0B szKeyDeviceKey \Registry\Machine\System\CurrentControlSet\Control\Video\{8C70D41C-0AAB-4D10-8CC0-4A12473090EE}\0000 szManufacturer Intel Corporation szMiniVdd unknown szMiniVddDateEnglish Unknown szMiniVddDateLocalized unknown szMonitorMaxRes Unknown szMonitorName Surface Display Panel szNotesEnglish No problems found. szNotesLocalized No problems found. szOverlayEnglish Supported szRankOfInstalledDriver 00D10001 szRegHelpText Unknown szRevision Unknown szRevisionId 0x000B szSubSysId 0x0A161414 szTestResultD3D7English Not run szTestResultD3D7Localized Not run szTestResultD3D8English Not run szTestResultD3D8Localized Not run szTestResultD3D9English Not run szTestResultD3D9Localized Not run szTestResultDDEnglish Not run szTestResultDDLocalized Not run szVdd unknown szVendorId 0x8086 Log Messages GpuProcessHostUIShim: The GPU process exited normally. Everything is okay.
,
Mar 29 2017
Sorry, it should be "Original Display Aspect Ratio". Also, the reason "Original Display Aspect Ratio" = 1.0 is because the original source frame is 1:1 aspect ratio.
,
Apr 13 2017
Links have changed - https://my.panomoments.com/u/dustinkerstein/m/grand-central
,
Apr 13 2017
,
Apr 13 2017
Can you be more specific about what you are expecting vs. what you are observing? I see that the encoded content is 3840x2160, and the MP4 metadata records the display size as 2160x2160. For both src= and MSE playback, Chrome reports 2160x2160 as the videoWidth/videoHeight. I believe that you are using texImage2D() to obtain textures from these frames. I would expect those textures to be 3840x2160 via either src= or MSE, but there may be / have been paths that would have scaled to 2160x2160. I realize that the WebGL spec is ambiguous about how aspect ratio should affect texImage2D(). Certainly the 3840x2160 option is faster and preserves more quality, but it has been a source of confusion. I expect that, if anything, the spec will be clarified to allow both. Are you observing anything different from what I expect?
,
Apr 14 2017
I just did a little bit more testing with a simplified non-MSE version of the player and I can reproduce this same issue. It still won't happen if I drag the file directly onto Chrome. It seems to be strictly an issue with WebGL textures as you are suggesting. My expectation is that the texture that gets returned in the UHD case to match the dimensions in accordance with the "Original Display Aspect Ratio". Given my 3840x2160 example, I would expect that a 3840x3840 texture would be returned if the "Original Display Aspect Ratio" == 1.0 - It's my impression that the video decoder should be the one responsible for ensuring the dimensions / aspect ratio of the output are correct, mainly because I don't believe there is anything exposed in the HTML5 video decoding framework that provides this "Original Display Aspect Ratio" that would allow me to resize the texture to the intended dimension. This is how Chrome used to behave prior to v57/v56 when hardware video decoding is enabled, and is still how Chrome behaves when hardware video decoding is disabled. Do you see any recent changes that could have affected this behavior? Do you believe this new behavior as intended? Thanks, Dustin
,
Apr 14 2017
And if possible, we should edit the title of this ticket to remove the reference to Media Source Extensions. Let me know if you'd prefer me to create an entirely new ticket. Thanks.
,
Apr 14 2017
Thank you for providing more feedback. Adding requester "sandersd@chromium.org" to the cc list and removing "Needs-Feedback" label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 14 2017
> My expectation is that the texture that gets returned in the UHD case > to match the dimensions in accordance with the "Original Display > Aspect Ratio". This is logical, although in general this is not how decode APIs work. It is the player's responsibility to handle aspect ratio. Perhaps that's not how the web APIs should work, but it does have performance and quality advantages. Usually you can't tell the difference; you map UVs from (0, 0) to (1, 1) to a box of size videoWidth x videoHeight, and you get correct aspect ratio regardless of the underlying texture size. > I don't believe there is anything exposed in the HTML5 video decoding > framework that provides this "Original Display Aspect Ratio" videoWidth/videoHeight. > Do you see any recent changes that could have affected this behavior? See issue 672895 and issue 701060 . > Do you believe this new behavior as intended? Yes, although the current path is still being developed.
,
Apr 14 2017
It looks like those two issues are definitely in the same vein and are very likely related to my issue. The problem with using videoWidth/videoHeight is that these values are equal to 3840x2160 in the above UHD case (as reported by chrome://media-internals), which represent the encoded width/height. However, those parameters don't represent the true aspect ratio of the video. I don't believe there exists any way for me to query the "Original Display Aspect Ratio" using the exposed HTML5 video API, so it is impossible for me to scale the texture at the time of display. Does that make sense? If this is the new intended behavior, then either the "Original Display Aspect Ratio" parameter must be exposed via the HTML5 video API, or the Width/Height video parameters need to take into consideration the "Original Display Aspect Ratio". Do you think this will be resolved in the revert/code-change coming in issue 701060 ? Thanks, Dustin
,
Apr 14 2017
I tested M57 and M59 on Linux (software decode), Mac (hardware decode), and Windows (hardware decode). All report 2160x2160. The revert was for M58, the fix for copying failures was deemed to large for merging, so instead the original cause of the issue is being reverted. M59 will continue to include both changes.
,
Apr 14 2017
Ah yes! I was just looking at media internals. You are correct, pulling in videoHeight and videoWidth do report the correct aspect ratio. Sorry for the confusion there. Going forward I guess I will need to manually scale the video as you've proposed. I think it's safe to mark this issue as designed. Thanks, Dustin PS - as a side note, with it reporting as 2160x2160 that means the texture is being downsized, so it's likely there's very little reason for me to be using the 3840x2160 resolution to store a square texture. If it was scaling to 3840x3840 then it'd at least be preserving the original resolution. But I guess that's a totally different question. Let me know if you have any thoughts on that though.
,
Apr 14 2017
There are two paths for computing the natural size from MP4 headers: - tkhd width and height; here your media explicitly lists 2160x2160 - pasp ratio (overrides tkhd in Chrome's MSE implementation): here your media lists 9:16. This goes through GetNaturalSize() (https://chromium.googlesource.com/chromium/src/+/9efcad1ba88ddc3e0161ba50fdb3a7c7dbe61932/media/base/video_util.cc#54), which computes a width to maintain the height. Also 2160x2160 for your media. I would recommend not including a pasp box for your use case. I agree in general that it is a waste to encode more pixels, but note that since you do get a 3840x2160 texture now, you are no longer losing any data to a resize operation.
,
Apr 17 2017
Sounds good. Thanks! |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 Deleted