New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 699252 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug



Sign in to add a comment

High CPU Utilization in VMware VDI environment

Reported by steve.bu...@whrsd.org, Mar 7 2017

Issue description

UserAgent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36

Example URL:
http://www.abcya.com/spanish_vocabulary.htm

Steps to reproduce the problem:
1. Review your test VM CPU baseline on the ESXi host using esxtop
2. In the Guest VM launch chrome and navigate to a website with multimedia demand. In our example we use www.abcya.com/spanish_vocabulary.htm
3. Review your test VM CPU utilization in esxtop. 

What is the expected behavior?
Following the steps you will notice the VM CPU utilization on the ESXi host maxes out. In our testing our baseline CPU was 3-5% and when we navigate to certain websites the CPU utilization jumps to 200%+

What went wrong?
CPU utilization shouldn't max out when accessing websites.

Did this work before? N/A 

Is it a problem with Flash or HTML5? N/A

Does this work in other browsers? N/A

Chrome version: 56.0.2924.87  Channel: stable
OS Version: 6.1 (Windows 7, Windows Server 2008 R2)
Flash Version: Shockwave Flash 24.0 r0

Contents of chrome://gpu: 
Graphics Feature Status
Canvas: Hardware accelerated
Flash: Hardware accelerated
Flash Stage3D: Hardware accelerated
Flash Stage3D Baseline profile: Hardware accelerated
Compositing: Hardware accelerated
Multiple Raster Threads: Disabled
Native GpuMemoryBuffers: Software only. Hardware acceleration disabled
Rasterization: Software only, hardware acceleration unavailable
Video Decode: Hardware accelerated
Video Encode: Hardware accelerated
VPx Video Decode: Software only, hardware acceleration unavailable
WebGL: Hardware accelerated
WebGL2: Hardware accelerated
Driver Bug Workarounds
clear_uniforms_before_first_program_use
disable_direct_composition
disable_discard_framebuffer
disable_dxgi_zero_copy_video
disable_framebuffer_cmaa
exit_on_context_lost
force_cube_complete
scalarize_vec_and_mat_constructor_args
texsubimage_faster_than_teximage
Problems Detected
VPx decoding isn't supported before Windows 10 anniversary update.: 616318
Disabled Features: accelerated_vpx_decode
GPU rasterization should only be enabled on NVIDIA Pascal and Maxwell, Intel Broadwell+, and AMD RX-R5 GPUs for now.: 643850
Disabled Features: gpu_rasterization
Some drivers are unable to reset the D3D device in the GPU process sandbox
Applied Workarounds: exit_on_context_lost
TexSubImage is faster for full uploads on ANGLE
Applied Workarounds: texsubimage_faster_than_teximage
Clear uniforms before first program use on all platforms: 124764, 349137
Applied Workarounds: clear_uniforms_before_first_program_use
Always rewrite vec/mat constructors to be consistent: 398694
Applied Workarounds: scalarize_vec_and_mat_constructor_args
ANGLE crash on glReadPixels from incomplete cube map texture: 518889
Applied Workarounds: force_cube_complete
Framebuffer discarding can hurt performance on non-tilers: 570897
Applied Workarounds: disable_discard_framebuffer
Direct composition flashes black initially on Win <10: 588588
Applied Workarounds: disable_direct_composition
Zero copy DXGI video hangs on shutdown on Win < 8.1: 621190
Applied Workarounds: disable_dxgi_zero_copy_video
Limited enabling of Chromium GL_INTEL_framebuffer_CMAA: 535198
Applied Workarounds: disable_framebuffer_cmaa
Raster is using a single thread.
Disabled Features: multiple_raster_threads
Native GpuMemoryBuffers have been disabled, either via about:flags or command line.
Disabled Features: native_gpu_memory_buffers
Version Information
Data exported	3/7/2017, 4:39:01 PM
Chrome version	Chrome/56.0.2924.87
Operating system	Windows NT 6.1.7601 SP1
Software rendering list version	12.06
Driver bug list version	9.24
ANGLE commit id	a4aaa2de57dc
2D graphics backend	Skia/56 bf2d9e02d58ea01f1c239f7e2fc024cba140ccb1
Command Line Args	Files\Google\Chrome\Application\chrome.exe" --flag-switches-begin --flag-switches-end
Driver Information
Initialization time	33
In-process GPU	false
Sandboxed	false
GPU0	VENDOR = 0x15ad, DEVICE= 0x0405
Optimus	false
AMD switchable	false
Desktop compositing	none
Driver vendor	VMware, Inc.
Driver version	8.15.1.46
Driver date	4-14-2016
Pixel shader version	3.0
Vertex shader version	3.0
Max. MSAA samples	1
Machine model name	
Machine model version	
GL_VENDOR	Google Inc.
GL_RENDERER	ANGLE (VMware SVGA 3D Direct3D9Ex vs_3_0 ps_3_0)
GL_VERSION	OpenGL ES 2.0 (ANGLE 2.1.0.a4aaa2de57dc)
GL_EXTENSIONS	GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_pack_reverse_row_order GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_frag_depth GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_fence GL_NV_pack_subimage GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
Disabled Extensions	
Window system binding vendor	Google Inc. (adapter LUID: 000000000000a4cc)
Window system binding version	1.4 (ANGLE 2.1.0.a4aaa2de57dc)
Window system binding extensions	EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource
Direct rendering	Yes
Reset notification strategy	0x8252
GPU process crash count	0
Compositor Information
Tile Update Mode	One-copy
Partial Raster	Enabled
GpuMemoryBuffers Status
ATC	Software only
ATCIA	Software only
DXT1	Software only
DXT5	Software only
ETC1	Software only
R_8	Software only
RG_88	Software only
BGR_565	Software only
RGBA_4444	Software only
RGBX_8888	Software only
RGBA_8888	Software only
BGRX_8888	Software only
BGRA_8888	Software only
YVU_420	Software only
YUV_420_BIPLANAR	Software only
UYVY_422	Software only
Diagnostics
0
b3DAccelerationEnabled	true
b3DAccelerationExists	true
bAGPEnabled	true
bAGPExistenceValid	true
bAGPExists	true
bCanRenderWindow	true
bDDAccelerationEnabled	true
bDriverBeta	false
bDriverDebug	false
bDriverSigned	false
bDriverSignedValid	false
bNoHardware	false
dwBpp	32
dwDDIVersion	9
dwHeight	846
dwRefreshRate	60
dwWHQLLevel	0
dwWidth	1432
iAdapter	0
lDriverSize	326208
lMiniVddSize	0
szAGPStatusEnglish	Enabled
szAGPStatusLocalized	Enabled
szChipType	VMware Virtual SVGA 3D Graphics Adapter
szD3DStatusEnglish	Enabled
szD3DStatusLocalized	Enabled
szDACType	n/a
szDDIVersionEnglish	9Ex
szDDIVersionLocalized	9Ex
szDDStatusEnglish	Enabled
szDDStatusLocalized	Enabled
szDXVAHDEnglish	Not Supported
szDXVAModes	
szDescription	VMware SVGA 3D
szDeviceId	0x0405
szDeviceIdentifier	{D7B71B4D-4745-11CF-3376-0424A0C2C535}
szDeviceName	\\.\DISPLAY1
szDisplayMemoryEnglish	800 MB
szDisplayMemoryLocalized	800 MB
szDisplayModeEnglish	1432 x 846 (32 bit) (60Hz)
szDisplayModeLocalized	1432 x 846 (32 bit) (60Hz)
szDriverAssemblyVersion	8.15.1.46
szDriverAttributes	Final Retail
szDriverDateEnglish	5/22/2016 19:12:56
szDriverDateLocalized	5/22/2016 7:12:56 PM
szDriverLanguageEnglish	English
szDriverLanguageLocalized	English
szDriverModelEnglish	WDDM 1.1
szDriverModelLocalized	WDDM 1.1
szDriverName	vm3dum.dll,vm3dum_10.dll
szDriverNodeStrongName	oem8.inf:VMware.NTx86.6.0:VM3D_X86:8.15.1.46:pci\ven_15ad&dev_0405&subsys_040515ad&rev_00
szDriverSignDate	
szDriverVersion	8.15.0001.0046
szKeyDeviceID	Enum\PCI\VEN_15AD&DEV_0405&SUBSYS_040515AD&REV_00
szKeyDeviceKey	\Registry\Machine\System\CurrentControlSet\Control\Video\{50F68C61-D37E-46EE-B60B-AB9BB9F7FA24}\0000
szManufacturer	VMware, Inc.
szMiniVdd	n/a
szMiniVddDateEnglish	n/a
szMiniVddDateLocalized	n/a
szMonitorMaxRes	
szMonitorName	Generic Non-PnP Monitor
szNotesEnglish	No problems found.
szNotesLocalized	No problems found.
szOverlayEnglish	Not Supported
szRankOfInstalledDriver	00F60000
szRegHelpText	
szRevision	
szRevisionId	0x0000
szSubSysId	0x040515AD
szTestResultD3D7English	Not run
szTestResultD3D7Localized	Not run
szTestResultD3D8English	Not run
szTestResultD3D8Localized	Not run
szTestResultD3D9English	Not run
szTestResultD3D9Localized	Not run
szTestResultDDEnglish	Not run
szTestResultDDLocalized	Not run
szVdd	n/a
szVendorId	0x15AD
Log Messages
[4356:5568:0307/163648.510:ERROR:angle_platform_impl.cc(33)] : ANGLE Display::initialize error 4: Could not create D3D11 device.
[4356:5568:0307/163648.510:ERROR:gl_surface_egl.cc(646)] : eglInitialize D3D11 failed with error EGL_NOT_INITIALIZED, trying next display type
[4628:2244:0307/163900.472:ERROR:angle_platform_impl.cc(33)] : ANGLE Display::initialize error 4: Could not create D3D11 device.
[4628:2244:0307/163900.472:ERROR:gl_surface_egl.cc(646)] : eglInitialize D3D11 failed with error EGL_NOT_INITIALIZED, trying next display type
GpuProcessHostUIShim: 

I am looking to see if Google can offer any insight into tweaks that can be made to the Chrome browser to optimize it for a VDI environment. I posted this on the Chrome Support forum and Julian Pastarmov asked that I open this as a bug on crbug.com instead.

When monitoring our VDI sessions via esxtop we are noticing increased CPU utilization. We contacted VMware support and their engineering team is attributing the spikes to video rendering demand of websites. While I was aware that opening full screen 4k videos on youtube would trigger this behavior I wasn't expecting simple interactive games such as www.abcya.com/spanish_vocabulary.htm to cause the threads to spike to 200+%. This site has been used as an example since it is known to trigger the issue. 

Attached I have included screenshots showing VM CPU baseline around 4% sitting at desktop but when we launch chrome and navigate to the aforementioned site the VM CPU spikes to 200%+ . VMware engineering was able to replicate this behavior across different ESXi builds and hardware and is reporting that the increased utilization is a result of increased demand. 

During research I found sites that mention advanced settings in chrome://settings, chrome://flags, and chrome://gpu however none specific to VDI or how each would be beneficial. One guide http://superuser.com/questions/697567/how-to-reduce-google-chromes-cpu-usage mentions reducing CPU utilization by adjusting certain settings but the thread is a few years old and I know flags and settings are changed with each update.

Attachments:
1.	Baseline.jpg – esxtop excerpt showing baseline VM CPU

2.	Chrome_open.jpg – esxtop excerpt showing 200%+ CPU with only one website open

3.	Taskmgr.jpg – Windows Task Manager showing utilization inside the guest VM

4.	Chrome_debug.log – Browser debug log while navigating to the website during testing

5.	Msinfo32.txt – System information from the VM session used during test

6.	Chrome_system.pdf – printout of chrome://system

7.	Chrome_gpu_settings.pdf – printout of chrome://gpu

 
Cc: dalecur...@chromium.org
Components: -Internals>Media Speed
Don't think this is media related. The page you link uses 30% CPU usage just idling and there are no active media players at that time. There are plenty of ways a bad page can hose the cpu usage unfortunately.

+speed team in case there's anything actionable here.
When we expand the GID in esxtop it shows the majority of cpu utilization is being performed by the vmx-svga and not vcpu. When we check guest VM taskmgr the CPU utilization is not spiked. This is the reason it was flagged as media related since the graphics threads are spiking. 

Additional Attachments:

1. sitting_at_start_page.jpg - cpu utilization in esxtop sitting at page without clicking to start the game.

2. game_started.jpg - cpu utilization in esxtop once game is started

3. taskmgr_guestVM.jpg - guest vm task managed showing minimal cpu utilization

4. trace_Tue_Mar_07_2017_5.30.35_PM.json.gz - trace log captured while navigating to the site and playing the game. captured according to https://www.chromium.org/developers/how-tos/submitting-a-performance-bug


sitting_at_start_page.jpg
82.6 KB View Download
game_started.jpg
84.3 KB View Download
taskmgr_guestVM.JPG
73.7 KB View Download
trace_Tue_Mar_07_2017_5.30.35_PM.json.gz
11.9 MB Download
Labels: Needs-Triage-M56
Cc: georgesak@chromium.org blumberg@chromium.org pastarmovj@chromium.org
Components: Enterprise
One thing to try here is to run chrome with the --disable-gpu flag. This will force software rasterization for everything instead of the virtual GPU driver provided by the VM. It will be useful to know if the built in rasterization does a better job sparing the CPU.
I launched chrome enabling logging and disabling gpu. I checked the esxtop baseline for the VM and saw around 4% CPU utilization. I confirmed gpu disabled via chrome://gpu and then navigated to the test site. I checked esxtop again for the VM and saw 200%+. I expanded the GID and this time the vCPU threads are doing all the processing and the vmx-svga threads aren't being used. 

New attachments:

1. chrome_gpu_whendisabled.pdf - printout of chrome://gpu confirming disabled

2. 2ndtest_baseline_esxtop.jpg - esxtop baseline with chrome open

3. 2ndtest_baseline_taskmgr.jpg - guest vm taskmgr baseline with chrome open

4. 2ndtest_websiteopen_esxtop.jpg - esxtop with website open

5. 2ndtest_websiteopen_esxtop_expanded.jpg - esxtop expanded with website open

6. 2ndtest_taskmgr_websiteopen.JPG - guest vm taskmgr with website open

7. 2ndtest_chrome_debug.log - chrome debug log while testing
chrome_gpu_whendisabled.pdf
97.7 KB Download
2ndtest_websiteopen_esxtop_expanded.jpg
97.1 KB View Download
2ndtest_websiteopen_esxtop.jpg
49.1 KB View Download
2ndtest_taskmgr_websiteopen.JPG
61.4 KB View Download
2ndtest_chrome_debug.log
525 KB View Download
2ndtest_baseline_taskmgr.jpg
55.9 KB View Download
2ndtest_baseline_esxtop.jpg
42.7 KB View Download
Labels: TE-NeedsTriageHelp
If I read the results right you mean overall load is the same only in one case it is the virtual CPU doing the work in the other case it is the virtual GPU but in either cases load is too high for your expectations?
That is correct - our users may encounter websites that are too demanding and we were wondering if there is a way to throttle the demand instead of allowing it to exhaust all resources. In a traditional PC model this isn't much of an issue because the browser utilizes the resources it has available and if it needs any more the quality/performance may be degraded until the site is closed. In a VDI environment where resources are pooled together in an oversubscribed model this issue is more concerning because instead of impacting one user's session you now have the potential to impact an entire host containing up to 80 user sessions. VMware's approach is to have the VM comply with the demand at the expense of the host. The only recommendation is to purchase additional hardware to satisfy the demand. I was hoping there way a way to limit the demand instead of throwing more hardware at it. They recommended purchased GPU accelerator cards like a Nvidia Tesla M6 to offload the graphics to free up the CPU or purchase additional hosts. Unfortunately we were sold BL460c blade servers and HPE won't support the accelerator cards - they want us to have WS460c blade servers instead. We can't add additional servers without purchasing another C7000 because it will be fully populated with no available server bays. We are already committed to the BL460 servers and were hoping for another alternative than purchasing all new WS460C servers to replace the existing since they are only a year old.  
Cc: dskaram@chromium.org
Labels: Enterprise-Triaged
Owner: pastarmovj@chromium.org
Status: Assigned (was: Unconfirmed)
Thanks for the complete justification Steve. 

I am totally on board with that and there are some ongoing efforts for getting Chrome to use less resources in constrained environments. I am currently investigating how this translates to shared environments like VMWare or TS.

Once we have something to share on that I will update this bug.
Hello, pastarmovj@chromium.org

Any updates on optimizing Chrome to use less resources.  I also manaage a virtual desktop enviroment and find Chrome v58 to be heavy on CPU.  Thanks
There are a few efforts that are targeted at reducing battery usage on laptops by reducing GPU usage but we are hopeful to apply the same tricks to get GPU usage on shared servers as well. 

I don't have much more concrete news to share for now though but certainly not dropping the ball here! :)
HI, 

I found this because I have the same problem with clients who have Chrome in a XenApp Enviroment. 

Have you looked into this?  Citrix Workspace Environment Manager?

https://www.citrix.com/blogs/2017/04/11/user-experience-on-steroids-citrix-workspace-environment-management/

It's not a fix for Chrome, but it does put Memory Constraits on an app you want, including Chrome.





Just checking in to see if there are any update on CPU utilization within Chrome specific to VDI environments. We purchased 6 GPU cards to offload rendering from CPU to GPU in attempts to increase performance however Chrome has now blacklisted all VMware video drivers on  issue 754435 . I'm concerned since CPU utilization is high and now we are no longer able to use GPU.
I discovered issue 766068 discussing the concept of throttling. It has more traction than this case as they are relating it to security instead of performance. Curious to see how the outcome of that case impacts this case.
Steve, I believe we talked last year when we put in our VDI environment here at MISD. After a lot of testing Cisco sent us four new servers to add to our environment to help with this exact issue and we are going to purchase the Nvidia Tesla M6 cards for the servers that don't currently have them. I did find software that helps out. I purchased "Process Lasso", I think we paid  $1500 for lifetime site license, but they have a free version also. This software allows you to limit the amount of CPU a program can use, but in testing I found that if you did this Chrome would just freeze when going to these problem sites. The software also allows you to limit the number of processors a program can use or assign Chrome to a specific processor. So, I set it up on our Horizon images, allowed Chrome access to one of the two CPUs and it dropped the CPU utilization around 1000Mhz per machine. 

Sign in to add a comment