Memory leak in svchost.exe on build machines |
||
Issue description[ADD relevant info inline] Link to build or pfq page. build # for that buildbot. Snippet of log that contains the failure. https://build.chromium.org/p/chromium.fyi/builders/CrWin7Goma%28clbr%29/builds/22069/steps/compile/logs/stdio FAILED: obj/third_party/WebKit/Source/web/web/FrameLoaderClientImpl.obj ninja -t msvc -e environment.x86 -- E:\b\c\cipd\goma/gomacc.exe "E:\b\depot_tools\win_toolchain\vs_files\d5dc33b15d1b2c086f2f6632e2fd15882f80dbd3\VC\bin\amd64_x86/cl.exe" /nologo /showIncludes /FC @obj/third_party/WebKit/Source/web/web/FrameLoaderClientImpl.obj.rsp /c ../../third_party/WebKit/Source/web/FrameLoaderClientImpl.cpp /Foobj/third_party/WebKit/Source/web/web/FrameLoaderClientImpl.obj /Fd"obj/third_party/WebKit/Source/web/web_cc.pdb" e:\b\c\b\crwin7goma_clbr_\src\third_party\webkit\source\platform\text\textrun.h(85): fatal error C1060: compiler is out of heap space
,
Sep 27 2016
The builder started leaking memory about 48 hours ago. The system memory pressure is now bad enough that builds are failing. Roughly 12 of the machines 16 GB of RAM is consumed before builds even start - probably more, actually, given that much data has probably been paged out. The machine has not been rebooted for four days. Before it is rebooted I will try to log on and figure out what is going on because this issue has been seen on multiple build machines. See attached image showing builder memory consumption and up-time over the last five days.
,
Sep 27 2016
I logged on to the machine and, while some of the information made no sense, I did see a svchost.exe process that had a commit size of 31 GB. The machine had zero bytes free, 330 MB available, and couldn't even launch git.exe. That service host process was running: - WinRM, Windows Remote Management (WS-Management) - TermService, Remote Desktop Services - NlaSvc, Network Location Awareness - LanmanWorkstation, Workstation - Dnscache, DNS Client - CryptSvc, Cryptographic Services I don't know which service was the problem. Oddly enough task manager claimed that all of the in-use memory was 'Cached' and actually listed very little memory as being part of the svchost.exe working set. However its peak working set was listed as 9 GB, and both that and the 31 GB commit size are rather too large for a 16 GB machine. This is ultimately a duplicate of bug 645170. I'd be surprised if they didn't have the same root cause. |
||
►
Sign in to add a comment |
||
Comment 1 by thakis@chromium.org
, Sep 27 2016