New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 719095 link

Starred by 16 users

Issue metadata

Status: Verified
Merged: issue 717884
Owner:
Last visit > 30 days ago
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug

Blocking:
issue 716558


Show other hotlists

Hotlists containing this issue:
Hotlist-1


Sign in to add a comment

webrtc: Chrome displaying blank video if H264 stream has SEI NAL packets

Reported by hitz.inf...@gmail.com, May 5 2017

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36

Example URL:

Steps to reproduce the problem:
1. Dial WebRTC call from Chrome to native software client on Windows using Intel Media SDK hardware video encoder
2. Chrome displays blank video while software client receives proper video
[Note: We are trying this in our internal setup]

What is the expected behavior?
Chrome displays remote video

What went wrong?
Chrome doesn't display remote video.

Did this work before? N/A 

Is it a problem with Flash or HTML5? N/A

Does this work in other browsers? N/A

Chrome version: 58.0.3029.96  Channel: stable
OS Version: OS X 10.12.2
Flash Version: Shockwave Flash 25.0 r0

Contents of chrome://gpu: 
Graphics Feature Status
Canvas: Hardware accelerated
Flash: Hardware accelerated
Flash Stage3D: Hardware accelerated
Flash Stage3D Baseline profile: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Enabled
Native GpuMemoryBuffers: Hardware accelerated
Rasterization: Hardware accelerated
Video Decode: Hardware accelerated
Video Encode: Hardware accelerated
VPx Video Decode: Hardware accelerated
WebGL: Hardware accelerated
WebGL2: Hardware accelerated
Driver Bug Workarounds
add_and_true_to_loop_condition
adjust_src_dst_region_for_blitframebuffer
decode_encode_srgb_for_generatemipmap
disable_framebuffer_cmaa
disable_multimonitor_multisampling
disable_webgl_rgb_multisampling_usage
emulate_abs_int_function
get_frag_data_info_bug
init_two_cube_map_levels_before_copyteximage
msaa_is_slow
pack_parameters_workaround_with_pack_buffer
rebind_transform_feedback_before_resume
regenerate_struct_names
remove_invariant_and_centroid_for_essl3
reset_base_mipmap_level_before_texstorage
rewrite_texelfetchoffset_to_texelfetch
scalarize_vec_and_mat_constructor_args
set_zero_level_before_generating_mipmap
unfold_short_circuit_as_ternary_operation
unpack_alignment_workaround_with_unpack_buffer
unpack_image_height_workaround_with_unpack_buffer
use_intermediary_for_copy_texture_image
use_unused_standard_shared_blocks
Problems Detected
Multisampling is buggy on OSX when multiple monitors are connected: 237931
Applied Workarounds: disable_multimonitor_multisampling
Unfold short circuit on Mac OS X: 307751
Applied Workarounds: unfold_short_circuit_as_ternary_operation
Always rewrite vec/mat constructors to be consistent: 398694
Applied Workarounds: scalarize_vec_and_mat_constructor_args
Mac drivers handle struct scopes incorrectly: 403957
Applied Workarounds: regenerate_struct_names
On Intel GPUs MSAA performance is not acceptable for GPU rasterization: 527565
Applied Workarounds: msaa_is_slow
glGenerateMipmap fails if the zero texture level is not set on some Mac drivers: 560499
Applied Workarounds: set_zero_level_before_generating_mipmap
Pack parameters work incorrectly with pack buffer bound: 563714
Applied Workarounds: pack_parameters_workaround_with_pack_buffer
Alignment works incorrectly with unpack buffer bound: 563714
Applied Workarounds: unpack_alignment_workaround_with_unpack_buffer
copyTexImage2D fails when reading from IOSurface on multiple GPU types.: 581777
Applied Workarounds: use_intermediary_for_copy_texture_image
Multisample renderbuffers with format GL_RGB8 have performance issues on Intel GPUs.: 607130
Applied Workarounds: disable_webgl_rgb_multisampling_usage
Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198
Applied Workarounds: disable_framebuffer_cmaa
glGetFragData{Location|Index} works incorrectly on Max: 638340
Applied Workarounds: get_frag_data_info_bug
glResumeTransformFeedback works incorrectly on Intel GPUs: 638514
Applied Workarounds: rebind_transform_feedback_before_resume
glTexStorage* are buggy when base mipmap level is not 0: 640506
Applied Workarounds: reset_base_mipmap_level_before_texstorage
Result of abs(i) where i is an integer in vertex shader is wrong: 642227
Applied Workarounds: emulate_abs_int_function
Rewrite texelFetchOffset to texelFetch for Intel Mac: 642605
Applied Workarounds: rewrite_texelfetchoffset_to_texelfetch
Rewrite condition in for and while loops for Intel Mac: 644669
Applied Workarounds: add_and_true_to_loop_condition
Decode and encode before generateMipmap for srgb format textures on macosx: 634519
Applied Workarounds: decode_encode_srgb_for_generatemipmap
Init first two levels before CopyTexImage2D for cube map texture on Intel Mac 10.12: 648197
Applied Workarounds: init_two_cube_map_levels_before_copyteximage
Insert statements to reference all members in unused std140/shared blocks on Mac: 618464
Applied Workarounds: use_unused_standard_shared_blocks
Tex(Sub)Image3D performs incorrectly when uploading from unpack buffer with GL_UNPACK_IMAGE_HEIGHT greater than zero on Intel Macs: 654258
Applied Workarounds: unpack_image_height_workaround_with_unpack_buffer
adjust src/dst region if blitting pixels outside read framebuffer on Mac: 644740
Applied Workarounds: adjust_src_dst_region_for_blitframebuffer
Mac driver GL 4.1 requires invariant and centroid to match between shaders: 639760, 641129
Applied Workarounds: remove_invariant_and_centroid_for_essl3
Disable KHR_blend_equation_advanced until cc shaders are updated: 661715
Version Information
Data exported	5/5/2017, 6:17:29 PM
Chrome version	Chrome/58.0.3029.96
Operating system	Mac OS X 10.12.2
Software rendering list version	12.20
Driver bug list version	9.36
ANGLE commit id	461d9a3060e3
2D graphics backend	Skia/58 4c81ba6ba3a3270db809bf7d4c3bc782694a56a4
Command Line Args	Chrome.app/Contents/MacOS/Google Chrome --flag-switches-begin --flag-switches-end
Driver Information
Initialization time	67
In-process GPU	false
Passthrough Command Decoder	false
Sandboxed	true
GPU0	VENDOR = 0x8086, DEVICE= 0x0a2e *ACTIVE*
Optimus	false
Optimus	false
AMD switchable	false
Driver vendor	
Driver version	10.22.25
Driver date	
Pixel shader version	4.10
Vertex shader version	4.10
Max. MSAA samples	8
Machine model name	MacBookPro
Machine model version	11.1
GL_VENDOR	Intel Inc.
GL_RENDERER	Intel Iris OpenGL Engine
GL_VERSION	4.1 INTEL-10.22.25
GL_EXTENSIONS	GL_ARB_blend_func_extended GL_ARB_draw_buffers_blend GL_ARB_draw_indirect GL_ARB_ES2_compatibility GL_ARB_explicit_attrib_location GL_ARB_gpu_shader_fp64 GL_ARB_gpu_shader5 GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_occlusion_query2 GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_separate_shader_objects GL_ARB_shader_bit_encoding GL_ARB_shader_subroutine GL_ARB_shading_language_include GL_ARB_tessellation_shader GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_cube_map_array GL_ARB_texture_gather GL_ARB_texture_query_lod GL_ARB_texture_rgb10_a2ui GL_ARB_texture_storage GL_ARB_texture_swizzle GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_vertex_attrib_64bit GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_EXT_debug_label GL_EXT_debug_marker GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_texture_compression_s3tc GL_EXT_texture_filter_anisotropic GL_EXT_texture_sRGB_decode GL_APPLE_client_storage GL_APPLE_container_object_shareable GL_APPLE_flush_render GL_APPLE_object_purgeable GL_APPLE_rgb_422 GL_APPLE_row_bytes GL_APPLE_texture_range GL_ATI_texture_mirror_once GL_NV_texture_barrier
Disabled Extensions	GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent
Window system binding vendor	
Window system binding version	
Window system binding extensions	
Direct rendering	Yes
Reset notification strategy	0x0000
GPU process crash count	0
Compositor Information
Tile Update Mode	Zero-copy
Partial Raster	Enabled
GpuMemoryBuffers Status
ATC	Software only
ATCIA	Software only
DXT1	Software only
DXT5	Software only
ETC1	Software only
R_8	GPU_READ_CPU_READ_WRITE, GPU_READ_CPU_READ_WRITE_PERSISTENT
RG_88	Software only
BGR_565	Software only
RGBA_4444	Software only
RGBX_8888	Software only
RGBA_8888	GPU_READ, SCANOUT, SCANOUT_CPU_READ_WRITE
BGRX_8888	GPU_READ, SCANOUT, SCANOUT_CPU_READ_WRITE
BGRA_8888	GPU_READ, SCANOUT, SCANOUT_CPU_READ_WRITE, GPU_READ_CPU_READ_WRITE, GPU_READ_CPU_READ_WRITE_PERSISTENT
YVU_420	Software only
YUV_420_BIPLANAR	GPU_READ_CPU_READ_WRITE, GPU_READ_CPU_READ_WRITE_PERSISTENT
UYVY_422	GPU_READ_CPU_READ_WRITE, GPU_READ_CPU_READ_WRITE_PERSISTENT

We have observed from Chromium logs that when SEI packets are present in received H264 stream, then even the first Keyframe is getting discarded. H264 NALUs have to be in following sequence to reproduce this issue: SPS, PPS, SEI, IDR, SEI, Delta-Frame, SEI, Delta-Frame, SEI, Delta-Frame,...... SPS, PPS, SEI, IDR, Delta-Frame. That is when SEI packets are present with every IDR frame, then Chrome won't display remote video.
We have also observed occasional video freeze issues when H264 NALUs are being received in this sequence: SPS, PPS, IDR, Delta-Frame, Delta-Frame, SPS, PPS, SEI, IDR, Delta-Frame. That is when SEI packets are not present in all of the IDR frames.
Please find attached network trace with RTP packets having H264 stream for the blank video scenario. Also attaching event log taken on Chrome for the call and Chromium logs with additional logging around frames processing.
 
Windows_SoftClient_MSDK_Hardware_Encoder_H264StreamWithSEINALPackets.pcapng
142 KB Download
event_log_sei_nalus.20.1
870 KB Download
windows_client_msdkencdoer_chrome_debug.log
275 KB View Download
We are seeing blank video issue on Windows Chrome as well. Following is our analysis as per Chromium logs:
- SEI packets arrive as part of first IDR frame, assuming packet sequence numbers are SPS (1), PPS (2), SEI (3), IDR (4 - 7)
- Then Chrome marks SEI packet as Delta Frame and append IDR frame to it
- Later before passing frame to decoder entire SEI + IDR Frame (Frame with packet numbers from 3 - 7) gets discarded because Chrome expects an IDR frame before a Delta Frame
- As a result Chrome never passes IDR frames to the decoder and displays blank remote video

We have tried few changes to handle SEI packet as a complete frame instead of Delta frame and it seems to be working. We have checked out Chromium from master branch on April 20th with commit revision number 118a3735394b55da76127370471a592d870c4b4c. Attaching diff file with the fix, created in chromium/src/third_party/webrtc directory. Please confirm our analysis above and suggest whether this fix is the right way to handle SEI packets. Thanks.
sei_nalus_handling_fix.diff
4.1 KB Download
Mergedinto: 717884
Status: Duplicate (was: Unconfirmed)
This issue looks similar to the issue id: 717884. Hence, merging into the issue id: 717884.
Please feel free to undupe the same if not the case.

Thanks...!!
Issue we are facing is different than reported in 717884, as we don't see errors observed in that ticket. As updated in the ticket description above that for H264 stream which doesn't have SEI NAL packet in first IDR frame, we do see video on Chrome and thereafter freeze issues whenever SEI packets arrive. So video does decode for our H264 stream, it is just the handling of SEI packets in the H264 stream which causes IDR frames to get discarded. We see blank video on Chrome when SEI packets are there with every IDR frame from the beginning.
We have been updating https://groups.google.com/forum/#!msg/discuss-webrtc/dt_A8pv_nBU/KEW1-kpCBQAJ forum post for this issue. Comment from Philip in this thread talk about handling of SEI packets and review request https://codereview.chromium.org/2769863007/. Fix we have tried also touches code modified in this review request for handling of SEI packets.

Hence requesting to track this as separate issue. Sorry but I don't seem to have permission to unmark this issue as duplicate. Also is Blink>WebRTC>Video right component for this issue?

Thanks.

Comment 4 by hbos@chromium.org, May 11 2017

Cc: emir...@chromium.org krajshree@chromium.org sprang@chromium.org
Components: Blink>WebRTC>Video
Status: Available (was: Duplicate)

Comment 5 by sprang@chromium.org, May 12 2017

Cc: philipel@chromium.org
Cc: niklase@chromium.org
Owner: philipel@chromium.org
Status: Assigned (was: Available)
Hi,

Any update on this issue? Please let us know if fix for this issue is going to be part of M59 release. Thanks.
Hi,

We are working on improving support for H264, hopefully it will be done next week.
Thanks for the update.

Hitesh

Labels: -OS-Mac M-59 OS-All
Hitesh, could you please verify that the fix landed in #11. It is part of canary build 60.0.3108.0 or newer.
I will verify and update. Thanks.
We found fix to be working. Thanks.

Blank video issue has been resolved for the scenario when SEI packet comes with every IDR & Delta frame. Video freeze issue also doesn't happen frequently too for the scenario when SEI packet arrives randomly every few seconds.
We did observe occasional video freeze (like every few minutes) but not like what it used to be before. We need to do more testing around this, as occasional video freeze could be due to packet loss.
We were seeing Aw,snap/crash issue yesterday occasionally with Canary v60.0.3109.0, but no more seeing it today with v60.0.3110.0.

Is this fix going to be available as part of M59 beta builds?

Regards,
Hitesh

Labels: Merge-Request-59
Thanks for verifying!

Hopefully we'll be able to get this into 59.
Project Member

Comment 16 by sheriffbot@chromium.org, May 25 2017

Labels: -Merge-Request-59 Merge-Review-59 Hotlist-Merge-Review
This bug requires manual review: We are only 11 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), gkihumba@(ChromeOS), Abdul Syed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Since we are only less than a week way from Stable, can you please confirm following 

1)if this is super urgent or if it can wait until M60?
2)Is it a safe merge, verified in canary and dev, and with enough automated test coverage?
3)We are already way past branch point for M59, and our bar is very high for merges. 
We need this fix to support basic Video calls with Desktop clients using Intel Media SDK hardware video encoder and improve Video quality with any client which sends SEI NAL packets in H264 video stream.

Thanks,
Hitesh

I think it would be very good to get this into 59 since many users have been suffering from this in 58. It has been in Canary for some time and been verified by some of the affected users. 

It should have reached dev by now. hitz.inifitech, could you verify this in dev channel as well?

FYI, we'd also have to merge https://bugs.chromium.org/p/chromium/issues/detail?id=725494, which has already been approved.
We are still seeing blank video issue with Dev builds on Windows and Mac with Version 60.0.3107.4 (Official Build) dev (64-bit). Looks like fix is still not available in Dev Channel.

Thanks,
Hitesh


Labels: -Merge-Review-59 Merge-Rejected-59
Based on comment 20, seems like this issue is still occurring. Rejecting M59 merge.
Labels: -Merge-Rejected-59 Merge-Request-59
The fix in #11 did not make it into  60.0.3107.4. The WebRTC roll with this fix was made on May 22.
Project Member

Comment 23 by sheriffbot@chromium.org, May 26 2017

Labels: -Merge-Request-59 Merge-Review-59
This bug requires manual review: We are only 10 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), gkihumba@(ChromeOS), Abdul Syed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
If this has not been tested in Dev yet, I'm not comfortable taking this merge in this late in the cycle, especially with M59 stable next week. My recommendation is to wait until M60. 
Blocking: 716558
Labels: -Pri-2 Pri-1
Cc: hdodda@chromium.org
 Issue 722516  has been merged into this issue.
Labels: -Merge-Review-59 Merge-Approved-59
Per chat, discussed this is a safe merge, well tested and verified, and it is impacting a sizeable number of users. Approving merge for M59. 
Project Member

Comment 29 by bugdroid1@chromium.org, May 30 2017

Labels: merge-merged-59
The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/b9ca68003f9dd88241dd2022dc9c077b91bd2ea2

commit b9ca68003f9dd88241dd2022dc9c077b91bd2ea2
Author: philipel <philipel@webrtc.org>
Date: Tue May 30 21:12:14 2017

Check H264 NALUs for frametype and insert SPS/PPS packets into the PacketBuffer.

BUG= chromium:719095 

Review-Url: https://codereview.webrtc.org/2889163003
Cr-Original-Commit-Position: refs/heads/master@{#18214}
Review-Url: https://codereview.webrtc.org/2916433003 .
Cr-Commit-Position: refs/branch-heads/59@{#14}
Cr-Branched-From: 10d095d4f743bc16f8e486e156c48a6d023b32c5-refs/heads/master@{#17657}

[modify] https://crrev.com/b9ca68003f9dd88241dd2022dc9c077b91bd2ea2/webrtc/modules/video_coding/frame_object.cc
[modify] https://crrev.com/b9ca68003f9dd88241dd2022dc9c077b91bd2ea2/webrtc/modules/video_coding/h264_sps_pps_tracker.cc
[modify] https://crrev.com/b9ca68003f9dd88241dd2022dc9c077b91bd2ea2/webrtc/modules/video_coding/h264_sps_pps_tracker_unittest.cc

Labels: -Merge-Approved-59
I faced this issue with M58 on linux x64 machine. will https://chromium.googlesource.com/external/webrtc.git/+/b9ca68003f9dd88241dd2022dc9c077b91bd2ea2 work out on H264 blank issue?

Comment 32 Deleted

derriclyns@,

If you have a stream where every frame start with a SEI then that CL should fix it.
 Issue 717884  has been merged into this issue.
We have verified the fix with latest Chrome Version 59.0.3071.86 (Official Build) (64-bit).

Thanks,
Hitesh


Status: Verified (was: Assigned)
Setting to Verified based on #35.

Sign in to add a comment