New issue
Advanced search Search tips

Issue 860420 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Aug 31
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug

Blocked on:
issue 867520



Sign in to add a comment

MEDIA_LOG when AVC IDR slice occurs in a frame, but the container didn't mark the frame as a keyframe

Reported by jeroen.t...@theoplayer.com, Jul 5

Issue description

UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36

Example URL:
https://cdn.theoplayer.com/bugreport/index.html

Steps to reproduce the problem:
1. Go to https://cdn.theoplayer.com/bugreport/index.html or open the index.html file in the attached zip file.
2. Click the 2nd or 4th button to trigger the playback issue. To check what scenario each button triggers, please read https://cdn.theoplayer.com/bugreport/README.md or README.md file in the attached zip file.

What is the expected behavior?
All the cases should result in normal playback.

What went wrong?
We add certain segments to the buffer. Based on whether we add the segments individually, add a mp4 creating by concatenating the segments to the buffer or specify that mp4 as the source of the video element, the behavior differs. The 2nd and 4th scenario fail to start playback as the video element reports an empty buffer. All other scenarios do work.

Did this work before? N/A 

Is it a problem with Flash or HTML5? HTML5

Does this work in other browsers? Yes

Chrome version: 67.0.3396.99  Channel: stable
OS Version: 10.0
Flash Version: 

Contents of chrome://gpu: 

Graphics Feature Status
Canvas: Hardware accelerated
CheckerImaging: Disabled
Flash: Hardware accelerated
Flash Stage3D: Hardware accelerated
Flash Stage3D Baseline profile: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Enabled
Native GpuMemoryBuffers: Software only. Hardware acceleration disabled
Rasterization: Hardware accelerated
Surface Synchronization: Enabled
Video Decode: Hardware accelerated
Viz Service Display Compositor: Disabled
WebGL: Hardware accelerated
WebGL2: Hardware accelerated
Driver Bug Workarounds
clear_uniforms_before_first_program_use
decode_encode_srgb_for_generatemipmap
disable_accelerated_vpx_decode
disable_discard_framebuffer
disable_framebuffer_cmaa
exit_on_context_lost
force_cube_complete
msaa_is_slow
scalarize_vec_and_mat_constructor_args
texsubimage_faster_than_teximage
Problems Detected
Some drivers are unable to reset the D3D device in the GPU process sandbox
Applied Workarounds: exit_on_context_lost
TexSubImage is faster for full uploads on ANGLE
Applied Workarounds: texsubimage_faster_than_teximage
Clear uniforms before first program use on all platforms: 124764, 349137
Applied Workarounds: clear_uniforms_before_first_program_use
Always rewrite vec/mat constructors to be consistent: 398694
Applied Workarounds: scalarize_vec_and_mat_constructor_args
ANGLE crash on glReadPixels from incomplete cube map texture: 518889
Applied Workarounds: force_cube_complete
On Intel GPUs MSAA performance is not acceptable for GPU rasterization: 527565
Applied Workarounds: msaa_is_slow
Framebuffer discarding can hurt performance on non-tilers: 570897
Applied Workarounds: disable_discard_framebuffer
Use GL_INTEL_framebuffer_CMAA on ChromeOS: 535198
Applied Workarounds: disable_framebuffer_cmaa
Disable KHR_blend_equation_advanced until cc shaders are updated: 661715
Applied Workarounds: disable(GL_KHR_blend_equation_advanced), disable(GL_KHR_blend_equation_advanced_coherent)
Decode and Encode before generateMipmap for srgb format textures on Windows: 634519
Applied Workarounds: decode_encode_srgb_for_generatemipmap
VPx decoding is too slow on Intel Broadwell, Skylake, and CherryView: 616318
Applied Workarounds: disable_accelerated_vpx_decode
Don't expose disjoint_timer_query extensions to WebGL: 808744
Native GpuMemoryBuffers have been disabled, either via about:flags or command line.
Disabled Features: native_gpu_memory_buffers
Viz service display compositor is not enabled by default.
Disabled Features: viz_display_compositor
Checker-imaging has been disabled via finch trial or the command line.
Disabled Features: checker_imaging
Version Information
Data exported	2018-07-05T07:36:52.821Z
Chrome version	Chrome/67.0.3396.99
Operating system	Windows NT 10.0.17134
Software rendering list URL	https://chromium.googlesource.com/chromium/src/+/a337fbf3c2ab8ebc6b64b0bfdce73a20e2e2252b/gpu/config/software_rendering_list.json
Driver bug list URL	https://chromium.googlesource.com/chromium/src/+/a337fbf3c2ab8ebc6b64b0bfdce73a20e2e2252b/gpu/config/gpu_driver_bug_list.json
ANGLE commit id	702006f4a07e
2D graphics backend	Skia/67 baf6686f92df805d3e25e80a0f3c79597cb3a6a5-
Command Line	"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --flag-switches-begin --flag-switches-end
Driver Information
Initialization time	173
In-process GPU	false
Passthrough Command Decoder	false
Direct Composition	true
Supports overlays	true
Sandboxed	true
GPU0	VENDOR = 0x8086, DEVICE= 0x191b *ACTIVE*
Optimus	false
AMD switchable	false
Desktop compositing	Aero Glass
Diagonal Monitor Size of \\.\DISPLAY1	24.0"
Diagonal Monitor Size of \\.\DISPLAY1	15.5"
Diagonal Monitor Size of \\.\DISPLAY2	15.5"
Diagonal Monitor Size of \\.\DISPLAY3	15.5"
DX12	true
Vulkan	true
Driver vendor	Intel Corporation
Driver version	22.20.16.4771
Driver date	8-13-2017
Pixel shader version	5.0
Vertex shader version	5.0
Max. MSAA samples	16
Machine model name	
Machine model version	
GL_VENDOR	Google Inc.
GL_RENDERER	ANGLE (Intel(R) HD Graphics 530 Direct3D11 vs_5_0 ps_5_0)
GL_VERSION	OpenGL ES 2.0 (ANGLE 2.1.0.702006f4a07e)
GL_EXTENSIONS	GL_ANGLE_client_arrays GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_program_cache_control GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_color_buffer_float_rgb GL_CHROMIUM_color_buffer_float_rgba GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc_srgb GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_surfaceless_context GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
Disabled Extensions	GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent
Disabled WebGL Extensions	EXT_disjoint_timer_query EXT_disjoint_timer_query_webgl2
Window system binding vendor	Google Inc. (adapter LUID: 0000000000015621)
Window system binding version	1.4 (ANGLE 2.1.0.702006f4a07e)
Window system binding extensions	EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_CHROMIUM_sync_control EGL_EXT_pixel_format_float EGL_KHR_surfaceless_context EGL_ANGLE_display_texture_share_group EGL_ANGLE_create_context_client_arrays EGL_ANGLE_program_cache_control EGL_ANGLE_robust_resource_initialization EGL_ANGLE_create_context_extensions_enabled
Direct rendering	Yes
Reset notification strategy	0x8252
GPU process crash count	0
Compositor Information
Tile Update Mode	One-copy
Partial Raster	Enabled
GpuMemoryBuffers Status
ATC	Software only
ATCIA	Software only
DXT1	Software only
DXT5	Software only
ETC1	Software only
R_8	Software only
R_16	Software only
RG_88	Software only
BGR_565	Software only
RGBA_4444	Software only
RGBX_8888	GPU_READ, SCANOUT
RGBA_8888	GPU_READ, SCANOUT
BGRX_8888	Software only
BGRX_1010102	Software only
RGBX_1010102	Software only
BGRA_8888	Software only
RGBA_F16	Software only
YVU_420	Software only
YUV_420_BIPLANAR	Software only
UYVY_422	Software only
Display(s) Information
Info	Display[2528732444] bounds=[0,0 1920x1200], workarea=[0,0 1920x1160], scale=1, external.
Color space information	{primaries:BT709, transfer:IEC61966_2_1, matrix:RGB, range:FULL}
Bits per color component	8
Bits per pixel	24
Video Acceleration Information
Decode h264 baseline	up to 4096x2304 pixels
Decode h264 baseline	up to 2304x4096 pixels
Decode h264 main	up to 4096x2304 pixels
Decode h264 main	up to 2304x4096 pixels
Decode h264 high	up to 4096x2304 pixels
Decode h264 high	up to 2304x4096 pixels
Encode h264 baseline	up to 3840x2176 pixels and/or 30.000 fps
Encode h264 main	up to 3840x2176 pixels and/or 30.000 fps
Encode h264 high	up to 3840x2176 pixels and/or 30.000 fps
Diagnostics
0
b3DAccelerationEnabled	true
b3DAccelerationExists	true
bAGPEnabled	true
bAGPExistenceValid	true
bAGPExists	true
bCanRenderWindow	true
bDDAccelerationEnabled	true
bDriverBeta	false
bDriverDebug	false
bDriverSigned	false
bDriverSignedValid	false
bNoHardware	false
dwBpp	32
dwDDIVersion	12
dwHeight	1200
dwRefreshRate	59
dwWHQLLevel	0
dwWidth	1920
iAdapter	0
lDriverSize	59869552
lMiniVddSize	0
szAGPStatusEnglish	Enabled
szAGPStatusLocalized	Enabled
szChipType	Intel(R) HD Graphics Family
szD3DStatusEnglish	Enabled
szD3DStatusLocalized	Enabled
szDACType	Internal
szDDIVersionEnglish	12
szDDIVersionLocalized	12
szDDStatusEnglish	Enabled
szDDStatusLocalized	Enabled
szDXVAHDEnglish	Supported
szDXVAModes	ModeMPEG2_A ModeMPEG2_C ModeWMV9_C ModeVC1_C
szDescription	Intel(R) HD Graphics 530
szDeviceId	0x191B
szDeviceIdentifier	{D7B78E66-5A5B-11CF-4B64-1945BDC2DB35}
szDeviceName	\\.\DISPLAY1
szDisplayMemoryEnglish	8262 MB
szDisplayMemoryLocalized	8262 MB
szDisplayModeEnglish	1920 x 1200 (32 bit) (59Hz)
szDisplayModeLocalized	1920 x 1200 (32 bit) (59Hz)
szDriverAssemblyVersion	22.20.16.4771
szDriverAttributes	Final Retail
szDriverDateEnglish	8/13/2017 2:00:00 AM
szDriverDateLocalized	8/13/2017 02:00:00
szDriverLanguageEnglish	English
szDriverLanguageLocalized	English
szDriverModelEnglish	WDDM 2.1
szDriverModelLocalized	WDDM 2.1
szDriverName	C:\WINDOWS\System32\DriverStore\FileRepository\igdlh64.inf_amd64_f10b0c0616a02072\igdumdim64.dll,C:\WINDOWS\System32\DriverStore\FileRepository\igdlh64.inf_amd64_f10b0c0616a02072\igd10iumd64.dll,C:\WINDOWS\System32\DriverStore\FileRepository\igdlh64.inf_amd64_f10b0c0616a02072\igd10iumd64.dll,C:\WINDOWS\System32\DriverStore\FileRepository\igdlh64.inf_amd64_f10b0c0616a02072\igd12umd64.dll
szDriverNodeStrongName	oem12.inf:5f63e5340a14bf6c:iSKLD_w10_DS:22.20.16.4771:pci\ven_8086&dev_191b
szDriverSignDate	Unknown
szDriverVersion	22.20.0016.4771
szKeyDeviceID	Enum\PCI\VEN_8086&DEV_191B&SUBSYS_65091558&REV_06
szKeyDeviceKey	\Registry\Machine\System\CurrentControlSet\Control\Video\{CCDEF62C-65BF-11E8-ADE0-80FA5B38C880}\0000
szManufacturer	Intel Corporation
szMiniVdd	unknown
szMiniVddDateEnglish	Unknown
szMiniVddDateLocalized	unknown
szMonitorMaxRes	Unknown
szMonitorName	Generic PnP Monitor
szNotesEnglish	No problems found.
szNotesLocalized	No problems found.
szOverlayEnglish	Supported
szRankOfInstalledDriver	00D12001
szRegHelpText	Unknown
szRevision	Unknown
szRevisionId	0x0006
szSubSysId	0x65091558
szTestResultD3D7English	Not run
szTestResultD3D7Localized	Not run
szTestResultD3D8English	Not run
szTestResultD3D8Localized	Not run
szTestResultD3D9English	Not run
szTestResultD3D9Localized	Not run
szTestResultDDEnglish	Not run
szTestResultDDLocalized	Not run
szVdd	unknown
szVendorId	0x8086
Log Messages
GpuProcessHostUIShim: The GPU process exited normally. Everything is okay.
GpuProcessHostUIShim: The GPU process exited normally. Everything is okay.
 
bugreport.zip
2.6 MB Download
Labels: Needs-Triage-M67
Cc: wolenetz@chromium.org
Components: -Internals>Media Internals>Media>Source
+mse

I haven't looked at the file yet and am no expert, but some thoughts:

Isn't a fragmented mp4 required to start with a keyframe? So the last two cases are not quite the same as manually cutting the first segment?

Assuming you do have proper keyframes, just skipping the segments without re-timecoding them isn't sufficient. So dropping just the first segment probably requires you to either seek to the right time code or set timestampOffset appropriately.


I agree that the last two cases are different from the first four, I added them for completeness. As for your remarks, here is some additional info:
- The segments were analyzed with a ISOBMFF analyzer which reported that they all start with a key frame.
- The provided code seeks to the start of buffer. This results in an error in the two failing cases (2 and 4) since the video element reports the buffer as empty.
Thanks for the details. I took a look and I'm not sure why it's not playing. There are no errors emitted which is a problem at least. Adjusting timestampOffset does not fix anything.

One thing that I did notice is that ffmpeg doesn't report the individual packets as being flagged as keyframes, even though cat init.dash + 1.dash > out.mp4 into Mp4box shows the first sample as marked as keyframe. I wonder if somehow we're dropping data thinking it's not a keyframe.
I'll take a look now that I'm back from vacation. Thanks for your patience.
MP4 has multiple ways of marking coded frames as keyframes. I'll check the files to see what Chrome thinks about them -- it's likely that no keyframes were ever detected by Chrome, hence nothing buffered. Seek in an unbuffered MSE media element itself shouldn't cause error -- if the error is coming from the seek operation itself in Chrome, that could also be a bug. Regardless, I'll look today.
Owner: wolenetz@chromium.org
I don't think there's any error thrown other than an unchecked access to .buffered.start(0) when buffered.length==0. So the hypothesis that it's just not detecting the key frame markings seems plausible.
It looks like the media only has precisely 1 keyframe: the very first frame in the first segment. MSE drops any non-keyframes parsed since the beginning of parsing (or since the last discontinuity detected by the coded frame processing algorithm, or after an explicit abort() operation).

So nothing buffered in cases 2 and 4 is working as intended, assuming Chrome is correctly identifying there is no keyframe in the media segments appended in those cases.

Further, no parse error or other error is expected per specification in such case.

Examining the frame sizes in cases 2 and 4, there are periodic spikes; possibly for frames incorrectly marked non-keyframes.

I've attached some log captures from a local build for each of cases 1, 2, and 4, filtered to show each coded frame parsed from the ISO-BMFF stream, in the sequence they are emitted to be processed by the MSE coded frame processing algorithm. Cases 2 and 4 logs do not differ. The latter part of case 1 matches case 2.
case_1.log
22.2 KB View Download
case_2.log
19.0 KB View Download
case_4.log
19.0 KB View Download
@#9 clarification of my wording: It is working as intended that nothing is buffered in cases 2 and 4 (since no keyframes were parsed).
@#9 more clarification: MSE drops any non-keyframes parsed since the beginning of parsing...until a keyframe is parsed.
It's not clear to me that they aren't keyframes though, since init + 1 plays:

http://xorax.sea.corp.google.com/mstest/segments/out-1.mp4

MP4Box also thinks that file has a keyframe at the beginning, but neither ffmpeg nor Chrome think so... so it's unclear how mp4box is determining that.
Cc: -wolenetz@chromium.org sande...@chromium.org
Status: Assigned (was: Unconfirmed)
+sandersd who might know why; see c#12.
Chrome MSE MP4 parser detected a keyframe, but only in the first segment.
It's logic has lots of comments due to the fairly complicated legacy around various media encodings not marking their ISOBMFF keyframes quite right:

https://chromium.googlesource.com/chromium/src/+/542807b25b6e7befcb71e69597fbc45dc3a52e6a/media/formats/mp4/track_run_iterator.cc#191

MP4Box might have some lenience different from Chrome MSE, and maybe just for the first frame in the media.

sandersd@, please take a closer look. Thanks!


One thing to note is that some other implementations of MSE might incorrectly be parsing and *not ignoring* sidx boxes, which can provide information like "starts_with_sap" that can inform whether or not the first indexed access unit of that sidx is perhaps a keyframe.

Using such sidx information in an MSE implementation would not be compliant with the MSE ISOBMFF byte stream format (https://www.w3.org/TR/mse-byte-stream-format-isobmff/).
In particular:
"Valid top-level boxes such as pdin, free, and sidx are allowed to appear before the moov box. These boxes must be accepted and ignored by the user agent and are not considered part of the initialization segment in this specification."

and

"Valid top-level boxes defined in ISO/IEC 14496-12 other than ftyp, moov, styp, moof, and mdat are allowed to appear between the end of an initialization segment or media segment and before the beginning of a new media segment. These boxes must be accepted and ignored by the user agent and are not considered part of the media segment in this specification."
Cc: -sande...@chromium.org wolenetz@chromium.org
Owner: sande...@chromium.org
One other bit of info I've now recalled is that other MSE implementations may base their determination of "keyframe-ness" for some codecs based on codec-specific information, disregarding or overriding any such indication in the container itself. For example, an IDR in H.264 in MP4 might be determined as a keyframe by such implementations even if the MP4 doesn't mark the frame as such. In Chrome MSE, we rely on correctly muxed streams. We even have a complementary case  bug 584384 .

Dan, perhaps, as a complement to fixing  bug 584384 , we could also MEDIA_LOG when a codec-bitstream keyframe is not marked keyframe in the container. WDYT?

Please also look into the media for this bug to see if that's what's really happening here.
I took a look; Dan please correct if I'm wrong:
In the sample media, there are multiple coded frames with nal_unit_type 5 (IDR slice); each corresponds to one of the longer periodic codec frames. But only the very first of those is determined to be a keyframe by our parsing of the MP4 container metadata.

This explains how other MSE implementations (and perhaps ffmpeg) can be more lenient: they cross layers and use information from the codec bitstream to determine keyframe-ness.

jeroen.tempels@theoplayer.com: To get this stream to work in Chrome MSE, please remux it with correct keyframe indications in the ISO-BMFF container, not just in the encoded video frames.
Summary: MEDIA_LOG when AVC IDR slice occurs in a frame, but the container didn't mark the frame as a keyframe (was: Video does not play based on non-init segment being added to source buffer or not)
Cc: -wolenetz@chromium.org sande...@chromium.org
Owner: wolenetz@chromium.org
From chat w/Dan, the route to MEDIA_LOG when kIDRSplice is seen in any NALU for a coded frame, but the container didn't mark the frame as a keyframe, appears to be the best way forward.

Note also that the MSE spec for the ISO-BMFF bytestream defines random access points in terms of the container, not the coded frames it contains.  Some other MSE implementations may be providing more lenience for handling incorrectly muxed media streams w.r.t. the spec; there are workarounds for handling such streams (in worst case, transmuxing in JS app before appending to MSE).

I'll see about adding a LIMITED_MEDIA_LOG soon.
Thanks for the all the information and investigation! I will remux the stream.
Project Member

Comment 21 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8005190febec38ff366befdab72cbd4b15e9a0c4

commit 8005190febec38ff366befdab72cbd4b15e9a0c4
Author: Matt Wolenetz <wolenetz@chromium.org>
Date: Fri Jul 20 00:32:07 2018

MSE: Remove old AVC and HEVC IsValidAnnexB vector wrapper

Chromium's allowed use of C++11's std::vector::data for quite a while
now.  In anticipation of some other refactoring coming soon to the
underlying code, this CL removes the workaround previously in place for
IsValidAnnexB(std::vector, ...) for each of MSE's AVC and HEVC bitstream
conversion utilities.

BUG= 860420 , 584384 

Change-Id: I15f1cedebaa37bbac91cb62456b8322dc70e34c4
Reviewed-on: https://chromium-review.googlesource.com/1144218
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Reviewed-by: Sergey Volk <servolk@chromium.org>
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#576720}
[modify] https://crrev.com/8005190febec38ff366befdab72cbd4b15e9a0c4/media/formats/mp4/avc.cc
[modify] https://crrev.com/8005190febec38ff366befdab72cbd4b15e9a0c4/media/formats/mp4/avc.h
[modify] https://crrev.com/8005190febec38ff366befdab72cbd4b15e9a0c4/media/formats/mp4/avc_unittest.cc
[modify] https://crrev.com/8005190febec38ff366befdab72cbd4b15e9a0c4/media/formats/mp4/hevc.cc
[modify] https://crrev.com/8005190febec38ff366befdab72cbd4b15e9a0c4/media/formats/mp4/hevc.h

Labels: M-70
Status: Started (was: Assigned)
Project Member

Comment 23 by bugdroid1@chromium.org, Jul 24

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae

commit 568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae
Author: Matt Wolenetz <wolenetz@chromium.org>
Date: Tue Jul 24 22:52:59 2018

MSE: refactor mp4 AnnexB validation to also report keyframe-ness

In preparation for logging when there appears to be a mismatch between
MSE MP4 keyframe metadata and the encoded video bitstream's keyframe
metadata, this change incorporates basic analysis of the latter as part
of the existing Annex-B bitstream validation for MSE MP4 video.

Note that mp4::BitstreamConverter::AnalysisResult contains
base::Optional<bool> for each of conformance and keyframe-ness fields.
If a field doesn't have a value, that portion of the result was not
analyzed.

Note that keyframe analysis is implemented only for AVC currently, but
even that is skipped if the frame is detected as non-conformant before
enough indications of whether or not it is a keyframe are detected.
Neither AVC-DV nor HEVC AnnexB analyses do any actual keyframe analysis,
since such was either skipped or not yet implemented previously,
respectively.

A subsequent CL will use the newly reported MSE MP4 video bitstream
converter's keyframe analysis results, if that analysis was done, in
reporting to chrome://media-internals when the bitstream keyframe-ness
mismatches that of the mp4 container for a coded frame.

BUG= 860420 , 584384 

Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I169c7774070ad232c86658bcd7803160323993ad
Reviewed-on: https://chromium-review.googlesource.com/1144456
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Reviewed-by: Sergey Volk <servolk@chromium.org>
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577717}
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/filters/ffmpeg_demuxer_unittest.cc
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/avc.cc
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/avc.h
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/avc_unittest.cc
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/bitstream_converter.cc
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/bitstream_converter.h
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/hevc.cc
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/hevc.h
[modify] https://crrev.com/568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae/media/formats/mp4/mp4_stream_parser.cc

Project Member

Comment 24 by bugdroid1@chromium.org, Jul 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/61697367404dc4c7033e389556f01a652f2b6516

commit 61697367404dc4c7033e389556f01a652f2b6516
Author: Matt Wolenetz <wolenetz@chromium.org>
Date: Wed Jul 25 00:42:07 2018

MSE: Log when MP4 keyframe-ness mismatches the compressed media

Logs to chrome://media-internals when the container's keyframe metadata
for a frame mismatches the contents of that frame (if the latter was
determined by the respective bitstream converter's Analyze()
implementation, of whom only AVC currently does any keyframe analysis).

BUG= 860420 , 584384 
TEST=Repro crbug 860420 shows logs, new MP4StreamParserTest.AVC_Key* and media

Change-Id: Ic87eab8726ad76b36b569df08ab2212b267008c2
Reviewed-on: https://chromium-review.googlesource.com/1149153
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577751}
[modify] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/formats/mp4/mp4_stream_parser.cc
[modify] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/formats/mp4/mp4_stream_parser.h
[modify] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/formats/mp4/mp4_stream_parser_unittest.cc
[modify] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/test/data/README
[add] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/test/data/bear-640x360-v-2frames-keyframe-is-non-sync-sample_frag.mp4
[add] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/test/data/bear-640x360-v-2frames-nonkeyframe-is-sync-sample_frag.mp4
[add] https://crrev.com/61697367404dc4c7033e389556f01a652f2b6516/media/test/data/bear-640x360-v-2frames_frag.mp4

Project Member

Comment 25 by bugdroid1@chromium.org, Jul 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9a4096bed9f7071637393d67a2d2fe9f554bfd45

commit 9a4096bed9f7071637393d67a2d2fe9f554bfd45
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed Jul 25 17:50:12 2018

Revert "MSE: Log when MP4 keyframe-ness mismatches the compressed media"

This reverts commit 61697367404dc4c7033e389556f01a652f2b6516.

Reason for revert:  crbug.com/867520 

Original change's description:
> MSE: Log when MP4 keyframe-ness mismatches the compressed media
> 
> Logs to chrome://media-internals when the container's keyframe metadata
> for a frame mismatches the contents of that frame (if the latter was
> determined by the respective bitstream converter's Analyze()
> implementation, of whom only AVC currently does any keyframe analysis).
> 
> BUG= 860420 , 584384 
> TEST=Repro crbug 860420 shows logs, new MP4StreamParserTest.AVC_Key* and media
> 
> Change-Id: Ic87eab8726ad76b36b569df08ab2212b267008c2
> Reviewed-on: https://chromium-review.googlesource.com/1149153
> Reviewed-by: Dan Sanders <sandersd@chromium.org>
> Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#577751}

TBR=wolenetz@chromium.org,sandersd@chromium.org

Change-Id: I8437af99f24bf04910ad56931a9e91f960a59469
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  860420 ,  584384 
Reviewed-on: https://chromium-review.googlesource.com/1150243
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577968}
[modify] https://crrev.com/9a4096bed9f7071637393d67a2d2fe9f554bfd45/media/formats/mp4/mp4_stream_parser.cc
[modify] https://crrev.com/9a4096bed9f7071637393d67a2d2fe9f554bfd45/media/formats/mp4/mp4_stream_parser.h
[modify] https://crrev.com/9a4096bed9f7071637393d67a2d2fe9f554bfd45/media/formats/mp4/mp4_stream_parser_unittest.cc
[modify] https://crrev.com/9a4096bed9f7071637393d67a2d2fe9f554bfd45/media/test/data/README
[delete] https://crrev.com/352492fdd7794a1c7672bee0e081764a667ebd9b/media/test/data/bear-640x360-v-2frames-keyframe-is-non-sync-sample_frag.mp4
[delete] https://crrev.com/352492fdd7794a1c7672bee0e081764a667ebd9b/media/test/data/bear-640x360-v-2frames-nonkeyframe-is-sync-sample_frag.mp4
[delete] https://crrev.com/352492fdd7794a1c7672bee0e081764a667ebd9b/media/test/data/bear-640x360-v-2frames_frag.mp4

Project Member

Comment 26 by bugdroid1@chromium.org, Jul 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8fba4f3d294510d3e8317f950c01138d956fd554

commit 8fba4f3d294510d3e8317f950c01138d956fd554
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed Jul 25 17:50:57 2018

Revert "MSE: refactor mp4 AnnexB validation to also report keyframe-ness"

This reverts commit 568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae.

Reason for revert: <INSERT REASONING HERE>

Original change's description:
> MSE: refactor mp4 AnnexB validation to also report keyframe-ness
> 
> In preparation for logging when there appears to be a mismatch between
> MSE MP4 keyframe metadata and the encoded video bitstream's keyframe
> metadata, this change incorporates basic analysis of the latter as part
> of the existing Annex-B bitstream validation for MSE MP4 video.
> 
> Note that mp4::BitstreamConverter::AnalysisResult contains
> base::Optional<bool> for each of conformance and keyframe-ness fields.
> If a field doesn't have a value, that portion of the result was not
> analyzed.
> 
> Note that keyframe analysis is implemented only for AVC currently, but
> even that is skipped if the frame is detected as non-conformant before
> enough indications of whether or not it is a keyframe are detected.
> Neither AVC-DV nor HEVC AnnexB analyses do any actual keyframe analysis,
> since such was either skipped or not yet implemented previously,
> respectively.
> 
> A subsequent CL will use the newly reported MSE MP4 video bitstream
> converter's keyframe analysis results, if that analysis was done, in
> reporting to chrome://media-internals when the bitstream keyframe-ness
> mismatches that of the mp4 container for a coded frame.
> 
> BUG= 860420 , 584384 
> 
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
> Change-Id: I169c7774070ad232c86658bcd7803160323993ad
> Reviewed-on: https://chromium-review.googlesource.com/1144456
> Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
> Reviewed-by: Sergey Volk <servolk@chromium.org>
> Reviewed-by: Dan Sanders <sandersd@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#577717}

TBR=wolenetz@chromium.org,sandersd@chromium.org,servolk@chromium.org

Change-Id: If4d14f0cb6fb0779255b9d7eb3d5b2b8c20e5be1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  860420 ,  584384 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/1150244
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577969}
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/filters/ffmpeg_demuxer_unittest.cc
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/avc.cc
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/avc.h
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/avc_unittest.cc
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/bitstream_converter.cc
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/bitstream_converter.h
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/hevc.cc
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/hevc.h
[modify] https://crrev.com/8fba4f3d294510d3e8317f950c01138d956fd554/media/formats/mp4/mp4_stream_parser.cc

Blockedon: 867520
Project Member

Comment 28 by bugdroid1@chromium.org, Jul 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a9b7a993e82e84c44dd16f332e55da569ab016c9

commit a9b7a993e82e84c44dd16f332e55da569ab016c9
Author: Matt Wolenetz <wolenetz@chromium.org>
Date: Wed Jul 25 23:12:39 2018

Reland "MSE: refactor mp4 AnnexB validation to also report keyframe-ness"

This is a reland of 568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae, which had
been reverted in 8fba4f3d294510d3e8317f950c01138d956fd554 ( bug 867520 )
due to failing link on windows builders with proprietary_codecs in their
build configuration ( bug 867536 ).

Versus the original change, this one includes an additional MEDIA_EXPORT
on the inner type "media::mp4::BitstreamConverter::AnalysisResult".

Original change's description:
> MSE: refactor mp4 AnnexB validation to also report keyframe-ness
>
> In preparation for logging when there appears to be a mismatch between
> MSE MP4 keyframe metadata and the encoded video bitstream's keyframe
> metadata, this change incorporates basic analysis of the latter as part
> of the existing Annex-B bitstream validation for MSE MP4 video.
>
> Note that mp4::BitstreamConverter::AnalysisResult contains
> base::Optional<bool> for each of conformance and keyframe-ness fields.
> If a field doesn't have a value, that portion of the result was not
> analyzed.
>
> Note that keyframe analysis is implemented only for AVC currently, but
> even that is skipped if the frame is detected as non-conformant before
> enough indications of whether or not it is a keyframe are detected.
> Neither AVC-DV nor HEVC AnnexB analyses do any actual keyframe analysis,
> since such was either skipped or not yet implemented previously,
> respectively.
>
> A subsequent CL will use the newly reported MSE MP4 video bitstream
> converter's keyframe analysis results, if that analysis was done, in
> reporting to chrome://media-internals when the bitstream keyframe-ness
> mismatches that of the mp4 container for a coded frame.
>
> BUG= 860420 , 584384 
>
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
> Change-Id: I169c7774070ad232c86658bcd7803160323993ad
> Reviewed-on: https://chromium-review.googlesource.com/1144456
> Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
> Reviewed-by: Sergey Volk <servolk@chromium.org>
> Reviewed-by: Dan Sanders <sandersd@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#577717}

Bug:  860420 ,  584384 
Change-Id: I8bbb3abc2079b53a14db8babbf04d5bad9e94767
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/1150247
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578116}
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/filters/ffmpeg_demuxer_unittest.cc
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/avc.cc
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/avc.h
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/avc_unittest.cc
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/bitstream_converter.cc
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/bitstream_converter.h
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/hevc.cc
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/hevc.h
[modify] https://crrev.com/a9b7a993e82e84c44dd16f332e55da569ab016c9/media/formats/mp4/mp4_stream_parser.cc

Project Member

Comment 29 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ae47fd35e9ca08850dc9a3585644616358d49364

commit ae47fd35e9ca08850dc9a3585644616358d49364
Author: Matt Wolenetz <wolenetz@chromium.org>
Date: Thu Jul 26 02:16:45 2018

Reland "MSE: Log when MP4 keyframe-ness mismatches the compressed media"

This is a reland of 61697367404dc4c7033e389556f01a652f2b6516, which had
been reverted in 9a4096bed9f7071637393d67a2d2fe9f554bfd45 as part of
reverting 568cc6a9efbcc2cbfbc33d5442dbaa7eaf5ae6ae which had broken some
builders. That CL has now been fixed and relanded
(a9b7a993e82e84c44dd16f332e55da569ab016c9).

This reland is the same as the original.

Original change's description:
> MSE: Log when MP4 keyframe-ness mismatches the compressed media
>
> Logs to chrome://media-internals when the container's keyframe metadata
> for a frame mismatches the contents of that frame (if the latter was
> determined by the respective bitstream converter's Analyze()
> implementation, of whom only AVC currently does any keyframe analysis).
>
> BUG= 860420 , 584384 
> TEST=Repro crbug 860420 shows logs, new MP4StreamParserTest.AVC_Key* and media
>
> Change-Id: Ic87eab8726ad76b36b569df08ab2212b267008c2
> Reviewed-on: https://chromium-review.googlesource.com/1149153
> Reviewed-by: Dan Sanders <sandersd@chromium.org>
> Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#577751}

TBR=sandersd@chromium.org

Bug:  860420 ,  584384 
Change-Id: I5a00819883e71ca9fd8c1a211956bdca66047b54
Reviewed-on: https://chromium-review.googlesource.com/1150541
Reviewed-by: Matthew Wolenetz <wolenetz@chromium.org>
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578171}
[modify] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/formats/mp4/mp4_stream_parser.cc
[modify] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/formats/mp4/mp4_stream_parser.h
[modify] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/formats/mp4/mp4_stream_parser_unittest.cc
[modify] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/test/data/README
[add] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/test/data/bear-640x360-v-2frames-keyframe-is-non-sync-sample_frag.mp4
[add] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/test/data/bear-640x360-v-2frames-nonkeyframe-is-sync-sample_frag.mp4
[add] https://crrev.com/ae47fd35e9ca08850dc9a3585644616358d49364/media/test/data/bear-640x360-v-2frames_frag.mp4

Status: Fixed (was: Started)

Sign in to add a comment