High CPU Utilization in VMware VDI environment
Reported by
steve.bu...@whrsd.org,
Mar 7 2017
|
||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Example URL: http://www.abcya.com/spanish_vocabulary.htm Steps to reproduce the problem: 1. Review your test VM CPU baseline on the ESXi host using esxtop 2. In the Guest VM launch chrome and navigate to a website with multimedia demand. In our example we use www.abcya.com/spanish_vocabulary.htm 3. Review your test VM CPU utilization in esxtop. What is the expected behavior? Following the steps you will notice the VM CPU utilization on the ESXi host maxes out. In our testing our baseline CPU was 3-5% and when we navigate to certain websites the CPU utilization jumps to 200%+ What went wrong? CPU utilization shouldn't max out when accessing websites. Did this work before? N/A Is it a problem with Flash or HTML5? N/A Does this work in other browsers? N/A Chrome version: 56.0.2924.87 Channel: stable OS Version: 6.1 (Windows 7, Windows Server 2008 R2) Flash Version: Shockwave Flash 24.0 r0 Contents of chrome://gpu: Graphics Feature Status Canvas: Hardware accelerated Flash: Hardware accelerated Flash Stage3D: Hardware accelerated Flash Stage3D Baseline profile: Hardware accelerated Compositing: Hardware accelerated Multiple Raster Threads: Disabled Native GpuMemoryBuffers: Software only. Hardware acceleration disabled Rasterization: Software only, hardware acceleration unavailable Video Decode: Hardware accelerated Video Encode: Hardware accelerated VPx Video Decode: Software only, hardware acceleration unavailable WebGL: Hardware accelerated WebGL2: Hardware accelerated Driver Bug Workarounds clear_uniforms_before_first_program_use disable_direct_composition disable_discard_framebuffer disable_dxgi_zero_copy_video disable_framebuffer_cmaa exit_on_context_lost force_cube_complete scalarize_vec_and_mat_constructor_args texsubimage_faster_than_teximage Problems Detected VPx decoding isn't supported before Windows 10 anniversary update.: 616318 Disabled Features: accelerated_vpx_decode GPU rasterization should only be enabled on NVIDIA Pascal and Maxwell, Intel Broadwell+, and AMD RX-R5 GPUs for now.: 643850 Disabled Features: gpu_rasterization Some drivers are unable to reset the D3D device in the GPU process sandbox Applied Workarounds: exit_on_context_lost TexSubImage is faster for full uploads on ANGLE Applied Workarounds: texsubimage_faster_than_teximage Clear uniforms before first program use on all platforms: 124764, 349137 Applied Workarounds: clear_uniforms_before_first_program_use Always rewrite vec/mat constructors to be consistent: 398694 Applied Workarounds: scalarize_vec_and_mat_constructor_args ANGLE crash on glReadPixels from incomplete cube map texture: 518889 Applied Workarounds: force_cube_complete Framebuffer discarding can hurt performance on non-tilers: 570897 Applied Workarounds: disable_discard_framebuffer Direct composition flashes black initially on Win <10: 588588 Applied Workarounds: disable_direct_composition Zero copy DXGI video hangs on shutdown on Win < 8.1: 621190 Applied Workarounds: disable_dxgi_zero_copy_video Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198 Applied Workarounds: disable_framebuffer_cmaa Raster is using a single thread. Disabled Features: multiple_raster_threads Native GpuMemoryBuffers have been disabled, either via about:flags or command line. Disabled Features: native_gpu_memory_buffers Version Information Data exported 3/7/2017, 4:39:01 PM Chrome version Chrome/56.0.2924.87 Operating system Windows NT 6.1.7601 SP1 Software rendering list version 12.06 Driver bug list version 9.24 ANGLE commit id a4aaa2de57dc 2D graphics backend Skia/56 bf2d9e02d58ea01f1c239f7e2fc024cba140ccb1 Command Line Args Files\Google\Chrome\Application\chrome.exe" --flag-switches-begin --flag-switches-end Driver Information Initialization time 33 In-process GPU false Sandboxed false GPU0 VENDOR = 0x15ad, DEVICE= 0x0405 Optimus false AMD switchable false Desktop compositing none Driver vendor VMware, Inc. Driver version 8.15.1.46 Driver date 4-14-2016 Pixel shader version 3.0 Vertex shader version 3.0 Max. MSAA samples 1 Machine model name Machine model version GL_VENDOR Google Inc. GL_RENDERER ANGLE (VMware SVGA 3D Direct3D9Ex vs_3_0 ps_3_0) GL_VERSION OpenGL ES 2.0 (ANGLE 2.1.0.a4aaa2de57dc) GL_EXTENSIONS GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_pack_reverse_row_order GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_frag_depth GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_fence GL_NV_pack_subimage GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object Disabled Extensions Window system binding vendor Google Inc. (adapter LUID: 000000000000a4cc) Window system binding version 1.4 (ANGLE 2.1.0.a4aaa2de57dc) Window system binding extensions EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource Direct rendering Yes Reset notification strategy 0x8252 GPU process crash count 0 Compositor Information Tile Update Mode One-copy Partial Raster Enabled GpuMemoryBuffers Status ATC Software only ATCIA Software only DXT1 Software only DXT5 Software only ETC1 Software only R_8 Software only RG_88 Software only BGR_565 Software only RGBA_4444 Software only RGBX_8888 Software only RGBA_8888 Software only BGRX_8888 Software only BGRA_8888 Software only YVU_420 Software only YUV_420_BIPLANAR Software only UYVY_422 Software only Diagnostics 0 b3DAccelerationEnabled true b3DAccelerationExists true bAGPEnabled true bAGPExistenceValid true bAGPExists true bCanRenderWindow true bDDAccelerationEnabled true bDriverBeta false bDriverDebug false bDriverSigned false bDriverSignedValid false bNoHardware false dwBpp 32 dwDDIVersion 9 dwHeight 846 dwRefreshRate 60 dwWHQLLevel 0 dwWidth 1432 iAdapter 0 lDriverSize 326208 lMiniVddSize 0 szAGPStatusEnglish Enabled szAGPStatusLocalized Enabled szChipType VMware Virtual SVGA 3D Graphics Adapter szD3DStatusEnglish Enabled szD3DStatusLocalized Enabled szDACType n/a szDDIVersionEnglish 9Ex szDDIVersionLocalized 9Ex szDDStatusEnglish Enabled szDDStatusLocalized Enabled szDXVAHDEnglish Not Supported szDXVAModes szDescription VMware SVGA 3D szDeviceId 0x0405 szDeviceIdentifier {D7B71B4D-4745-11CF-3376-0424A0C2C535} szDeviceName \\.\DISPLAY1 szDisplayMemoryEnglish 800 MB szDisplayMemoryLocalized 800 MB szDisplayModeEnglish 1432 x 846 (32 bit) (60Hz) szDisplayModeLocalized 1432 x 846 (32 bit) (60Hz) szDriverAssemblyVersion 8.15.1.46 szDriverAttributes Final Retail szDriverDateEnglish 5/22/2016 19:12:56 szDriverDateLocalized 5/22/2016 7:12:56 PM szDriverLanguageEnglish English szDriverLanguageLocalized English szDriverModelEnglish WDDM 1.1 szDriverModelLocalized WDDM 1.1 szDriverName vm3dum.dll,vm3dum_10.dll szDriverNodeStrongName oem8.inf:VMware.NTx86.6.0:VM3D_X86:8.15.1.46:pci\ven_15ad&dev_0405&subsys_040515ad&rev_00 szDriverSignDate szDriverVersion 8.15.0001.0046 szKeyDeviceID Enum\PCI\VEN_15AD&DEV_0405&SUBSYS_040515AD&REV_00 szKeyDeviceKey \Registry\Machine\System\CurrentControlSet\Control\Video\{50F68C61-D37E-46EE-B60B-AB9BB9F7FA24}\0000 szManufacturer VMware, Inc. szMiniVdd n/a szMiniVddDateEnglish n/a szMiniVddDateLocalized n/a szMonitorMaxRes szMonitorName Generic Non-PnP Monitor szNotesEnglish No problems found. szNotesLocalized No problems found. szOverlayEnglish Not Supported szRankOfInstalledDriver 00F60000 szRegHelpText szRevision szRevisionId 0x0000 szSubSysId 0x040515AD szTestResultD3D7English Not run szTestResultD3D7Localized Not run szTestResultD3D8English Not run szTestResultD3D8Localized Not run szTestResultD3D9English Not run szTestResultD3D9Localized Not run szTestResultDDEnglish Not run szTestResultDDLocalized Not run szVdd n/a szVendorId 0x15AD Log Messages [4356:5568:0307/163648.510:ERROR:angle_platform_impl.cc(33)] : ANGLE Display::initialize error 4: Could not create D3D11 device. [4356:5568:0307/163648.510:ERROR:gl_surface_egl.cc(646)] : eglInitialize D3D11 failed with error EGL_NOT_INITIALIZED, trying next display type [4628:2244:0307/163900.472:ERROR:angle_platform_impl.cc(33)] : ANGLE Display::initialize error 4: Could not create D3D11 device. [4628:2244:0307/163900.472:ERROR:gl_surface_egl.cc(646)] : eglInitialize D3D11 failed with error EGL_NOT_INITIALIZED, trying next display type GpuProcessHostUIShim: I am looking to see if Google can offer any insight into tweaks that can be made to the Chrome browser to optimize it for a VDI environment. I posted this on the Chrome Support forum and Julian Pastarmov asked that I open this as a bug on crbug.com instead. When monitoring our VDI sessions via esxtop we are noticing increased CPU utilization. We contacted VMware support and their engineering team is attributing the spikes to video rendering demand of websites. While I was aware that opening full screen 4k videos on youtube would trigger this behavior I wasn't expecting simple interactive games such as www.abcya.com/spanish_vocabulary.htm to cause the threads to spike to 200+%. This site has been used as an example since it is known to trigger the issue. Attached I have included screenshots showing VM CPU baseline around 4% sitting at desktop but when we launch chrome and navigate to the aforementioned site the VM CPU spikes to 200%+ . VMware engineering was able to replicate this behavior across different ESXi builds and hardware and is reporting that the increased utilization is a result of increased demand. During research I found sites that mention advanced settings in chrome://settings, chrome://flags, and chrome://gpu however none specific to VDI or how each would be beneficial. One guide http://superuser.com/questions/697567/how-to-reduce-google-chromes-cpu-usage mentions reducing CPU utilization by adjusting certain settings but the thread is a few years old and I know flags and settings are changed with each update. Attachments: 1. Baseline.jpg – esxtop excerpt showing baseline VM CPU 2. Chrome_open.jpg – esxtop excerpt showing 200%+ CPU with only one website open 3. Taskmgr.jpg – Windows Task Manager showing utilization inside the guest VM 4. Chrome_debug.log – Browser debug log while navigating to the website during testing 5. Msinfo32.txt – System information from the VM session used during test 6. Chrome_system.pdf – printout of chrome://system 7. Chrome_gpu_settings.pdf – printout of chrome://gpu
,
Mar 7 2017
When we expand the GID in esxtop it shows the majority of cpu utilization is being performed by the vmx-svga and not vcpu. When we check guest VM taskmgr the CPU utilization is not spiked. This is the reason it was flagged as media related since the graphics threads are spiking. Additional Attachments: 1. sitting_at_start_page.jpg - cpu utilization in esxtop sitting at page without clicking to start the game. 2. game_started.jpg - cpu utilization in esxtop once game is started 3. taskmgr_guestVM.jpg - guest vm task managed showing minimal cpu utilization 4. trace_Tue_Mar_07_2017_5.30.35_PM.json.gz - trace log captured while navigating to the site and playing the game. captured according to https://www.chromium.org/developers/how-tos/submitting-a-performance-bug
,
Mar 7 2017
,
Mar 8 2017
,
Mar 8 2017
One thing to try here is to run chrome with the --disable-gpu flag. This will force software rasterization for everything instead of the virtual GPU driver provided by the VM. It will be useful to know if the built in rasterization does a better job sparing the CPU.
,
Mar 8 2017
I launched chrome enabling logging and disabling gpu. I checked the esxtop baseline for the VM and saw around 4% CPU utilization. I confirmed gpu disabled via chrome://gpu and then navigated to the test site. I checked esxtop again for the VM and saw 200%+. I expanded the GID and this time the vCPU threads are doing all the processing and the vmx-svga threads aren't being used. New attachments: 1. chrome_gpu_whendisabled.pdf - printout of chrome://gpu confirming disabled 2. 2ndtest_baseline_esxtop.jpg - esxtop baseline with chrome open 3. 2ndtest_baseline_taskmgr.jpg - guest vm taskmgr baseline with chrome open 4. 2ndtest_websiteopen_esxtop.jpg - esxtop with website open 5. 2ndtest_websiteopen_esxtop_expanded.jpg - esxtop expanded with website open 6. 2ndtest_taskmgr_websiteopen.JPG - guest vm taskmgr with website open 7. 2ndtest_chrome_debug.log - chrome debug log while testing
,
Mar 9 2017
,
Mar 13 2017
If I read the results right you mean overall load is the same only in one case it is the virtual CPU doing the work in the other case it is the virtual GPU but in either cases load is too high for your expectations?
,
Mar 13 2017
That is correct - our users may encounter websites that are too demanding and we were wondering if there is a way to throttle the demand instead of allowing it to exhaust all resources. In a traditional PC model this isn't much of an issue because the browser utilizes the resources it has available and if it needs any more the quality/performance may be degraded until the site is closed. In a VDI environment where resources are pooled together in an oversubscribed model this issue is more concerning because instead of impacting one user's session you now have the potential to impact an entire host containing up to 80 user sessions. VMware's approach is to have the VM comply with the demand at the expense of the host. The only recommendation is to purchase additional hardware to satisfy the demand. I was hoping there way a way to limit the demand instead of throwing more hardware at it. They recommended purchased GPU accelerator cards like a Nvidia Tesla M6 to offload the graphics to free up the CPU or purchase additional hosts. Unfortunately we were sold BL460c blade servers and HPE won't support the accelerator cards - they want us to have WS460c blade servers instead. We can't add additional servers without purchasing another C7000 because it will be fully populated with no available server bays. We are already committed to the BL460 servers and were hoping for another alternative than purchasing all new WS460C servers to replace the existing since they are only a year old.
,
Mar 13 2017
,
Mar 14 2017
Thanks for the complete justification Steve. I am totally on board with that and there are some ongoing efforts for getting Chrome to use less resources in constrained environments. I am currently investigating how this translates to shared environments like VMWare or TS. Once we have something to share on that I will update this bug.
,
May 24 2017
Hello, pastarmovj@chromium.org Any updates on optimizing Chrome to use less resources. I also manaage a virtual desktop enviroment and find Chrome v58 to be heavy on CPU. Thanks
,
May 26 2017
There are a few efforts that are targeted at reducing battery usage on laptops by reducing GPU usage but we are hopeful to apply the same tricks to get GPU usage on shared servers as well. I don't have much more concrete news to share for now though but certainly not dropping the ball here! :)
,
Dec 20 2017
HI, I found this because I have the same problem with clients who have Chrome in a XenApp Enviroment. Have you looked into this? Citrix Workspace Environment Manager? https://www.citrix.com/blogs/2017/04/11/user-experience-on-steroids-citrix-workspace-environment-management/ It's not a fix for Chrome, but it does put Memory Constraits on an app you want, including Chrome.
,
Jan 9 2018
Just checking in to see if there are any update on CPU utilization within Chrome specific to VDI environments. We purchased 6 GPU cards to offload rendering from CPU to GPU in attempts to increase performance however Chrome has now blacklisted all VMware video drivers on issue 754435 . I'm concerned since CPU utilization is high and now we are no longer able to use GPU.
,
Jan 9 2018
I discovered issue 766068 discussing the concept of throttling. It has more traction than this case as they are relating it to security instead of performance. Curious to see how the outcome of that case impacts this case.
,
Feb 22 2018
Steve, I believe we talked last year when we put in our VDI environment here at MISD. After a lot of testing Cisco sent us four new servers to add to our environment to help with this exact issue and we are going to purchase the Nvidia Tesla M6 cards for the servers that don't currently have them. I did find software that helps out. I purchased "Process Lasso", I think we paid $1500 for lifetime site license, but they have a free version also. This software allows you to limit the amount of CPU a program can use, but in testing I found that if you did this Chrome would just freeze when going to these problem sites. The software also allows you to limit the number of processors a program can use or assign Chrome to a specific processor. So, I set it up on our Horizon images, allowed Chrome access to one of the two CPUs and it dropped the CPU utilization around 1000Mhz per machine. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by dalecur...@chromium.org
, Mar 7 2017Components: -Internals>Media Speed