Link parallelism needs adjustment to avoid OOM while maximizing parallelism |
|||||||
Issue descriptionBy default Chromium allows a maximum of GB-Of-RAM/5 links to run in parallel, so twelve parallel links on a 64-GB machine. This is reasonable for most links, but there are a number of large binaries that can consume ~13 GB of RAM when linking, which means that if the timing is right then even a 64 GB machine may end up swapping. This actually happens when doing a full build of the 'all' target. This then causes the machine to be sluggish afterwards because private data from VS and other processes gets pushed out to pagefile.sys. Currently I avoid this problem by setting GYP_LINK_CONCURRENCY=5, but this means that my normal builds run more slowly (artificially constrained link parallelism) and this doesn't help other developers or build machines. Ninja supports a way of tagging certain build steps as going to a pool with a maximum amount of parallelism (https://ninja-build.org/manual.html#ref_pool) which could be used something like this: # No more than 5 heavy objects at a time. pool heavy_object_pool depth = 5 # A build statement can specify a pool directly. # Only one of these builds will run at a time. build heavy_object1.obj: cc heavy_obj1.cc pool = heavy_object_pool
,
Jun 17 2016
I have some stats for linker memory usage on all targets across three flavors of release builds. For 64-bit official builds (LTCG) there are 5-6 targets where the linker consumes 20-29 GB of RAM, which implies that we can have a maximum linker parallelism of two (on a 64 GB machine). However for the majority of link targets we can support 5-10 times more parallelism. Given how slow 64-bit official-build links are it is crucial that we maximize linker parallelism while avoiding paging. More details are available, and a summary is on this VC++ bug that I filed to request improvements in 64-bit LTCG linking performance: https://connect.microsoft.com/VisualStudio/feedback/details/2828366
,
Jun 18 2016
This is actually in line with my observations for LLVM LTO builds. They take even more RAM (up to ~40 GB), but that was mitigated by adding more RAM into the slaves. When running on a machine with >200 GB RAM, we can actually take all large targets linked concurrently, so we could set up enough parallelism. I agree that this is problematic for builds on the dev desktops, and that link pools will address this. Given my schedule (I am on vacation now, and then I'll have to prepare CFI on Linux for a launch), I don't expect to have time for another attempt to push the link pools until late August.
,
Jun 20 2016
If you have an in-progress CL that you can share then I might be able to take a look.
,
Jun 24 2016
,
Jun 24 2016
@krasin - can you give me access to that doc?
,
Jun 24 2016
,
Jul 1 2016
,
Nov 22 2017
,
Nov 22 2017
,
Nov 22 2017
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by krasin@chromium.org
, May 24 2016