New issue
Advanced search Search tips

Issue 796021 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Task

Blocking:
issue 787983



Sign in to add a comment

Support non-using rsp file build on windows

Project Member Reported by tikuta@chromium.org, Dec 19 2017

Issue description

ninja takes around 25% cpu time to create/delete rsp file in windows.
It becomes one of build speed limiting factor when goma backend has sufficiently warm cache.

Let me support the build not using rsp files.

CreateProcess supports 32,768 characters command line, it should be sufficient for current chrome building.
https://msdn.microsoft.com/ja-jp/library/windows/desktop/ms682425(v=vs.85).aspx
 
Blocking: 787983
Cc: tikuta@chromium.org
 Issue 796269  has been merged into this issue.
Pasting in information from the duplicate bug:

When doing a NOP build of Chrome (ninja -n chrome, make sure you have explicitly run "gn gen" first) the StartEdge and FinishCommand stages consume most of the time (use -d stats to see this). StartEdge takes ~20 s and FinishCommand takes ~8 s. The vast majority of this (on my workstation) is spent in creating and deleting .rsp files. This was confirmed when I did ETW profiling of the NOP build which showed that the vast majority of this time was in the OS - in kernel functions or perhaps in anti-virus.

This overhead means that a NOP build of chrome (with jumbo and goma enabled so there are "only" 25,470 build steps) takes ~36 s. Experiments with disabling of .rsp creation and deletion shows that it reduces the NOP build time to ~8 s.

The effect on 'real' builds is unknown but it is possible that the absolute savings may be similar, or at least the .rsp file creation savings may apply.



In particular note that on -n builds of Chrome the RSP files account for more than 75% of elapsed time, or about 28 s. On real builds of Chrome this change seems to help ninja ramp up to full build speed slightly faster, although that is hard to tell for sure.

Project Member

Comment 4 by bugdroid1@chromium.org, Jan 5 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/goma/client/+/ff39c26fa5c23c70b88179a3d0a5e499250dc90f

commit ff39c26fa5c23c70b88179a3d0a5e499250dc90f
Author: Takuto Ikuta <tikuta@google.com>
Date: Fri Jan 05 09:12:48 2018

Project Member

Comment 5 by bugdroid1@chromium.org, Jan 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d2164444bcfdf6e4e5eadc55bcfb77fd74f9dba8

commit d2164444bcfdf6e4e5eadc55bcfb77fd74f9dba8
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Sat Jan 13 01:33:45 2018

Do not use rsp files for windows compile

Creating and deleting rsp files is time consuming part when building on windows with warmed goma backend cache.
To reduce such bottleneck, we can use all compiler flags directory in command line instead of using rsp file.

This CL keeps using rsp file for linking because linking flags can be longer than compiles and the number of linking is relatively small compared to compile.

I took 3 time build stats for 'chrome' on 48 thread Z840 windows 10 with following args.gn and -j 500.
```
goma_dir = "C:\\src\\goma_client\\client\\out\\Release"
is_component_build = true
is_debug = false
strip_absolute_paths_from_debug_symbols = true
symbol_level = 0
target_cpu = "x86"
use_goma = true

is_clang = false
enable_nacl = false
use_lld = true
```

* With this CL
TotalSeconds: 279.9604737
TotalSeconds: 279.4099064
TotalSeconds: 273.8253059
Avg: 277.731895333333

* Without this CL
TotalSeconds: 325.0155664
TotalSeconds: 319.4094433
TotalSeconds: 299.9709
Avg: 314.798636566667


Bug:  796021 
Change-Id: Ice959c196b6879b39962a3b628cdcf531884ec36
Reviewed-on: https://chromium-review.googlesource.com/832593
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529137}
[modify] https://crrev.com/d2164444bcfdf6e4e5eadc55bcfb77fd74f9dba8/build/toolchain/win/BUILD.gn

Comment 6 by thakis@chromium.org, Jan 14 2018

Status: Fixed (was: Assigned)
I just tested "ninja.exe -n -C out\default chrome" on a freshly created out\default directory (no args.gn file). This used to be painfully slow, mostly from the creation and destruction of .rsp files (~36 s). It now takes ~9 s, or ~20 s if VsChromium is running (creating file notification objects takes a while). I don't remember if the ~36 s was with VsChromium running or not.

The new 9/20 s times are with a non-jumbo build. A jumbo build would be faster.

So, yeah. It's faster now.

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 5 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/74612b2ba5df3042eca38da71223f2a22114ba40

commit 74612b2ba5df3042eca38da71223f2a22114ba40
Author: Nico Weber <thakis@chromium.org>
Date: Thu Apr 05 17:47:33 2018

Stop passing --rsp-quoting to clang-cl.

We no longer use rsp files for compiles as of
https://chromium-review.googlesource.com/832593

Bug:  796021 
Change-Id: I2ffb54f5bf2a596025af9a43d9e06fd992234f48
Reviewed-on: https://chromium-review.googlesource.com/997994
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#548475}
[modify] https://crrev.com/74612b2ba5df3042eca38da71223f2a22114ba40/build/toolchain/win/BUILD.gn

Sign in to add a comment