99% sure of memory leak in chromium GPU process on Windows 7
Reported by
kennyhom...@gmail.com,
Dec 15 2016
|
||||
Issue description
UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
Example URL:
Steps to reproduce the problem:
1. Windows 7 install on Asus E210
2. electron v1.4.3
3. loop trough batch of 1080p 16:9 h264 mp4 video's 24/7
What is the expected behavior?
Video to keep playing as smoothly after more than 8 hours as it does the first few hours.
What went wrong?
After more than 8 hours video start to stutter and eventually the render process crashes to a white screen. Sometimes Windows will give a "system out of memory" error.
Did this work before? No
Is it a problem with Flash or HTML5? HTML5
Does this work in other browsers? N/A
Chrome version: 53.0.2785.113 Channel: stable
OS Version: Windows 7
Flash Version: none
Contents of chrome://gpu:
Graphics Feature Status
Canvas: Hardware accelerated
Flash: Hardware accelerated
Flash Stage3D: Hardware accelerated
Flash Stage3D Baseline profile: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Disabled
Native GpuMemoryBuffers: Software only. Hardware acceleration disabled
Rasterization: Software only. Hardware acceleration disabled
Video Decode: Hardware accelerated
Video Encode: Hardware accelerated
WebGL: Hardware accelerated
Driver Bug Workarounds
clear_uniforms_before_first_program_use
disable_direct_composition
disable_discard_framebuffer
disable_dxgi_zero_copy_video
disable_framebuffer_cmaa
exit_on_context_lost
force_cube_complete
msaa_is_slow
scalarize_vec_and_mat_constructor_args
texsubimage_faster_than_teximage
Problems Detected
Some drivers are unable to reset the D3D device in the GPU process sandbox
Applied Workarounds: exit_on_context_lost
TexSubImage is faster for full uploads on ANGLE
Applied Workarounds: texsubimage_faster_than_teximage
Clear uniforms before first program use on all platforms: 124764, 349137
Applied Workarounds: clear_uniforms_before_first_program_use
Always rewrite vec/mat constructors to be consistent: 398694
Applied Workarounds: scalarize_vec_and_mat_constructor_args
ANGLE crash on glReadPixels from incomplete cube map texture: 518889
Applied Workarounds: force_cube_complete
On Intel GPUs MSAA performance is not acceptable for GPU rasterization: 527565
Applied Workarounds: msaa_is_slow
Framebuffer discarding can hurt performance on non-tilers: 570897
Applied Workarounds: disable_discard_framebuffer
Direct composition flashes black initially on Win <10: 588588
Applied Workarounds: disable_direct_composition
Zero copy DXGI video hangs on shutdown on Win < 8.1: 621190
Applied Workarounds: disable_dxgi_zero_copy_video
Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198
Applied Workarounds: disable_framebuffer_cmaa
Accelerated rasterization has been disabled, either via blacklist, about:flags or the command line.
Disabled Features: rasterization
Raster is using a single thread.
Disabled Features: multiple_raster_threads
Native GpuMemoryBuffers have been disabled, either via about:flags or command line.
Disabled Features: native_gpu_memory_buffers
Version Information
Data exported 12/15/2016, 11:33:51 AM
Chrome version Chrome/53.0.2785.113
Operating system Windows NT 6.1.7601 SP1
Software rendering list version 11.7
Driver bug list version 8.93
ANGLE commit id unknown hash
2D graphics backend Skia
Command Line Args --no-sandbox --allow-file-access-from-files
Driver Information
Initialization time 86
In-process GPU false
Sandboxed false
GPU0 VENDOR = 0x8086, DEVICE= 0x0f31
Optimus false
AMD switchable false
Desktop compositing Aero Glass
Diagonal Monitor Size of \\.\DISPLAY1 21.4"
Driver vendor Intel Corporation
Driver version 10.18.10.3740
Driver date 7-4-2014
Pixel shader version 5.0
Vertex shader version 5.0
Max. MSAA samples 8
Machine model name
Machine model version
GL_VENDOR Google Inc.
GL_RENDERER ANGLE (Intel(R) HD Graphics Direct3D11 vs_5_0 ps_5_0)
GL_VERSION OpenGL ES 2.0 (ANGLE 2.1.0.unknown hash)
GL_EXTENSIONS GL_OES_element_index_uint GL_OES_packed_depth_stencil GL_OES_get_program_binary GL_OES_rgb8_rgba8 GL_EXT_texture_format_BGRA8888 GL_EXT_read_format_bgra GL_NV_pixel_buffer_object GL_OES_mapbuffer GL_EXT_map_buffer_range GL_EXT_color_buffer_half_float GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_float GL_OES_texture_float_linear GL_EXT_texture_rg GL_EXT_texture_compression_dxt1 GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_OES_compressed_ETC1_RGB8_texture GL_EXT_sRGB GL_ANGLE_depth_texture GL_OES_depth32 GL_EXT_texture_storage GL_OES_texture_npot GL_EXT_draw_buffers GL_EXT_texture_filter_anisotropic GL_EXT_occlusion_query_boolean GL_NV_fence GL_EXT_disjoint_timer_query GL_EXT_robustness GL_EXT_blend_minmax GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_pack_reverse_row_order GL_OES_standard_derivatives GL_EXT_shader_texture_lod GL_EXT_frag_depth GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_EXT_discard_framebuffer GL_EXT_debug_marker GL_OES_EGL_image GL_OES_EGL_image_external GL_NV_EGL_stream_consumer_external GL_EXT_unpack_subimage GL_NV_pack_subimage GL_OES_vertex_array_object GL_KHR_debug GL_ANGLE_lossy_etc_decode GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_sync_query
Disabled Extensions
Window system binding vendor Google Inc. (adapter LUID: 000000000000761e)
Window system binding version 1.4 (ANGLE 2.1.0.unknown hash)
Window system binding extensions EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility
Direct rendering Yes
Reset notification strategy 0x8252
GPU process crash count 0
Compositor Information
Tile Update Mode One-copy
Partial Raster Enabled
GpuMemoryBuffers Status
ATC Software only
ATCIA Software only
DXT1 Software only
DXT5 Software only
ETC1 Software only
R_8 Software only
BGR_565 Software only
RGBA_4444 Software only
RGBX_8888 Software only
RGBA_8888 Software only
BGRX_8888 Software only
BGRA_8888 Software only
YVU_420 Software only
YUV_420_BIPLANAR Software only
UYVY_422 Software only
Diagnostics
0
b3DAccelerationEnabled true
b3DAccelerationExists true
bAGPEnabled true
bAGPExistenceValid true
bAGPExists true
bCanRenderWindow true
bDDAccelerationEnabled true
bDriverBeta false
bDriverDebug false
bDriverSigned false
bDriverSignedValid false
bNoHardware false
dwBpp 32
dwDDIVersion 11
dwHeight 1080
dwRefreshRate 59
dwWHQLLevel 0
dwWidth 1920
iAdapter 0
lDriverSize 10647064
lMiniVddSize 0
szAGPStatusEnglish Enabled
szAGPStatusLocalized Enabled
szChipType Intel(R) HD Graphics
szD3DStatusEnglish Enabled
szD3DStatusLocalized Enabled
szDACType Internal
szDDIVersionEnglish 11
szDDIVersionLocalized 11
szDDStatusEnglish Enabled
szDDStatusLocalized Enabled
szDXVAHDEnglish Supported
szDXVAModes ModeMPEG2_A ModeMPEG2_C ModeWMV9_C ModeVC1_C
szDescription Intel(R) HD Graphics
szDeviceId 0x0F31
szDeviceIdentifier {D7B78E66-4C71-11CF-6F7D-3EA5B3C2C735}
szDeviceName \\.\DISPLAY1
szDisplayMemoryEnglish 1696 MB
szDisplayMemoryLocalized 1696 MB
szDisplayModeEnglish 1920 x 1080 (32 bit) (59Hz)
szDisplayModeLocalized 1920 x 1080 (32 bit) (59Hz)
szDriverAssemblyVersion 10.18.10.3740
szDriverAttributes Final Retail
szDriverDateEnglish 7/11/2014 07:25:52
szDriverDateLocalized 11/07/2014 7:25:52
szDriverLanguageEnglish English
szDriverLanguageLocalized English
szDriverModelEnglish WDDM 1.1
szDriverModelLocalized WDDM 1.1
szDriverName igdumdim64.dll,igd10iumd64.dll,igd10iumd64.dll,igdumdim32,igd10iumd32,igd10iumd32
szDriverNodeStrongName oem143.inf:IntelGfx.NTamd64.6.1:iVLV2M_w7:10.18.10.3740:pci\ven_8086&dev_0f31
szDriverSignDate
szDriverVersion 10.18.0010.3740
szKeyDeviceID Enum\PCI\VEN_8086&DEV_0F31&SUBSYS_85341043&REV_0E
szKeyDeviceKey \Registry\Machine\System\CurrentControlSet\Control\Video\{EF286C39-BFBA-406C-8A77-EEF79D8879B7}\0000
szManufacturer Intel Corporation
szMiniVdd n/a
szMiniVddDateEnglish n/a
szMiniVddDateLocalized n/a
szMonitorMaxRes
szMonitorName Generic PnP Monitor
szNotesEnglish No problems found.
szNotesLocalized No problems found.
szOverlayEnglish Supported
szRankOfInstalledDriver 00E02001
szRegHelpText
szRevision
szRevisionId 0x000E
szSubSysId 0x85341043
szTestResultD3D7English Not run
szTestResultD3D7Localized Not run
szTestResultD3D8English Not run
szTestResultD3D8Localized Not run
szTestResultD3D9English Not run
szTestResultD3D9Localized Not run
szTestResultDDEnglish Not run
szTestResultDDLocalized Not run
szVdd n/a
szVendorId 0x8086
Log Messages
GpuProcessHostUIShim: The GPU process exited normally. Everything is okay.
(I'm sorry for the long text, I'm trying to give you every piece of information I have gathered from 2 months of testing):
I am a web developer using electron (node with chromium v53) to create a digital signage platform.
The signage app will loop trough a playlist of video's/images/html apps according to predefined schedule.
Problems arise when a playlist consisting of ONLY 1080p videos is used.
The videos are all encoded to h264 in a mp4 container and are being played through an HTML5 video element.
The hardware on which we have noticed the choppy video playback is an Asus E210.
I have been running tests for weeks trying to isolate the issue. I am 99% sure the problem lies within the GPU process (and the interaction with the hardware).
WHEN DO WE SEE THE PROBLEM?
When a pure video pllaylist has been running for a few hours.
WHAT ARE THE PROBLEMS WE CAN SEE?
After a few hours the video starts to get choppy, a while later a video can hang on the beginning frame for the duration of that video. Sometimes the video will hang on a frame for a number of seconds and then fast-forwards to catch up with real time.
If held even longer the render process crashes and a white screen is seen. The electron app as a whole has not crashed since my main process is still logging to files as it should. When I look at the task manager only the render process has died. The main process and GPU process are still present, the screen remain white.
Often times this crash comes together with an error dialog of "system is running out of memory".
WHEN DO WE NOT SEE ANY ISSUES?
When gpu acceleration is turned of OR I add the command line switch "--in-process-gpu". Video playback is now choppy all the time as no gpu is being used, only CPU. BUT the big difference is that even after 48 hours, the video is still equally choppy. In other words the video performance does not drop over time!!
We have this same test running on another ASUS device which does not have GPU acceleration, this device does have a stronger CPU and thus has no problem decoding on the fly resulting in smooth playback.
IS IT THE DEVICE ITSELF?
I would think not because the previous installed player worked with either cute framework or used vlc to playback 1080p h264 video content, all without any issues and no performance drop over time.
Second argument: Because the video runs perfectly smooth for a few hours, this shows that the device is strong enough to decode the video using GPU. When video gets choppy we don't have to restart the device itself, simply closing the chromium instance and reopening it is enough to get that smooth playing back.
IS THE LEAK IN THE APP ITSELF OR IN JAVASCRIPT?
No, would had these issues in the beginning, we could see the performance drop on all devices and the memory spike. I adjusted the code to force garbage collection of unused DOM nodes and source files and the leak disappeared.
As a stronger test I took out one of the videos in the playlist and created a separate app containing no css or js, just a page with an HTML5 video element containing that video in loop with autoplay. After more than 8 hours of playing the exact same thing happens, the video gets choppy and later on totally hangs and eventually crashes.
IS IT THE CODEC?
No, as this is chromium I decided to make an identical playlist containing webm videos, one with the VP8 codec and another with the VP9 codec. Same results.
(These were videos without audio and the HTML5 video object had the tag muted turned on)
ADDITIONAL HELPFUL INFORMATION:
-When the muted tag is turned off (sound enabled) problems appeared to happen sooner.
-Adding some command line switches to limit ram/cache usage seemed to help a little but didn't solve the problem. List of used tags below.
-The behaviour is quite random, some devices show a crash after only 30min of playing while others can have no crash in 48 hours. Though on average most problems arise after about 8 hours of continuous video playback.
-Lower resolution video's and display help but also didn't solve the problem.
-When running a test where my chromium browser shut down en restarted by electron every 2 hours, video playback remained perfect all time.
LIST of tested command line switches (not all were GPU related, we also tried shutting down caching):
flag: 'disable-drive-search-in-app-launcher',
flag: 'noerrdialogs', //Suppresses all error dialogs when present.
flag: 'touch-events', //Tells Chrome to do additional touch noise filtering. Should only be used if the driver level filtering is insufficient
flag: 'enabled', //enabled: touch events always enabled.
flag: 'material-hybrid', //Material design hybrid mode for the |kTopChromeMD| switch. Targeted for mouse/touch hybrid devices.
flag: 'ignore-gpu-blacklist', //Ignores GPU blacklist.
flag: 'enable-low-end-device-mode', //Force disabling of low-end device mode when set.
flag: 'disable-infobars', //Prevent infobars from appearing.
flag: 'disable-notifications', //Disables the Web Notification and the Push APIs.
flag: 'skip-gpu-data-loading', //Skip gpu info collection, blacklist loading, and blacklist auto-update scheduling at browser startup time. Therefore, all GPU features are available, and about:gpu page shows empty content. The switch is intended only for layout tests. TODO(gab): Get rid of
flag: 'disable-boot-animation', //Disables wallpaper boot animation (except of OOBE case)
flag: 'disable-breakpad', //Disables the crash reporting.
flag: 'disable-sync', //Disables syncing browser data to a Google Account.
flag: 'kiosk',
flag: 'noerrdialogs', //Suppresses all error dialogs when present.
flag: 'enable-nacl', //Runs the Native Client inside the renderer process and enables GPU plugin (internally adds lEnableGpuPlugin to the command line).
flag: 'disable-zero-copy-dxgi-video', //Disable the video decoder from drawing directly to a texture.
flag: 'disable-avfoundation-overlays', //Disable use of AVFoundation to draw video content.
flag: 'video-threads=16', //Set number of threads to use for video decoding.
flag: 'aggressive-cache-discard',
flag: 'disable-gpu-program-cache', //Turn off gpu program caching
flag: 'disable-gpu-shader-disk-cache', //Disables the GPU shader on disk cache
flag: 'gpu-program-cache-size-kb=0', //Sets the maximum size of the in-memory gpu program cache, in kb
flag: 'in-process-gpu', //Run the GPU process as a thread in the browser process.
flag: 'mse-video-buffer-size-limit',
flag: 'enable-native-gpu-memory-buffers',
flag: 'enable-zero-copy',
flag: 'disable-gpu',
flag: 'disable-software-rasterizer',
flag: 'disk-cache-dir=null',
flag: 'media-cache-dir=null',
flag: 'disable-gpu-shader-disk-cache',
flag: 'disk-cache-size=0',
flag: 'media-cache-size=0',
IN CONCLUSION:
I have tried EVERYTHING I can, I keep hoping to find the solution. The fact that everything runs smoothly in the first few hours is proof that this device has sufficient power to handle our application. We can no longer find any proof that the problem would be in the code. Every test brings us to the same conclusion: When GPU process is running with full hardware acceleration on this device, somewhere, somehow a memory leak is created.
We are trying to move forward with chrome os devices in the future to avoid problems like these, however we still have several thousand players of this type in the field and this is turning out to be a major showstopper for us.
I wouldn't bother the specialist in the core team without trying everything I can first. Our fate is in your expertise now. I really hope somewhen can shed some light on this.
Thanks in advance.
,
Dec 20 2016
Are you able to reproduce this with a normal webapp in Chrome Stable? Sorry we can't assist with electron issues unless they also reproduce in Chrome.
,
Dec 29 2016
,
Jan 24 2017
no response from bug opener. as per c#2, close this bug
,
Jan 25 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by ajha@chromium.org
, Dec 16 2016