New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 635308 link

Starred by 9 users

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 3
Type: Bug

Blocking:
issue 695864
issue 735361



Sign in to add a comment

GN: action targets should support explicit pool

Project Member Reported by sdefresne@chromium.org, Aug 7 2016

Issue description

When building for iOS fat binary, with dSYM generation, the generation of the dSYM and the linking are performed by "action" targets. Since those have no pool speficied, they are executed with maximum parallelism specified by the build, which is bad as those are expensive (they are roughly as expensive as "solink"/"solink_module"/"executable" actions).

If "action" supported explicit pool, then the same pool as the "link" tool could be used to limit the parallelism.
 
Have you had any time to look at this?

Another use-case for this has come up regarding executing proguard actions on android internal builders. Each one uses about 1.5G, and builds that run too many in parallel on a bot tend to fail.
No, I did not have any time to look into this.
Cc: brettw@chromium.org sdefresne@chromium.org agrieve@chromium.org
Components: Build
Labels: -OS-iOS Build-Tools-GN OS-All
Owner: ----
Status: Available (was: Assigned)
This isn't an ios-specific thing ...

Comment 4 by tikuta@chromium.org, Feb 28 2017

Blocking: 695864

Comment 5 by brettw@chromium.org, Feb 28 2017

I suspect this should just be a pool for all actions. I doubt this is specific to the bindings generator.
One could imagine supporting pools per action if some were cpu-intensive and others were slow and i/o-intensive or something, but I agree that just one pool would be a good start.
Owner: phosek@chromium.org
We need this in Fuchsia as well so I'm going to try and implement this.
Had a chat with Petr and he had a good use-case for per-action pools so I think this is OK to add.
Per-action, or for all actions? I think in the usual goma or equivalent case everyone would be better off limiting non-distributed actions to something like ninja's 2*cores default.
Maybe we need both?

For a global limit we would define a "action" tool on the toolchain since there is already a facility to have pools for tools.

If we do a per-action pool, we should replace the current "console" variable on the action with it so we don't have two ways to specify a pool. I'm guessing the best thing would be to have a "console = true" variable on the pool definition.
Per action, so:

pool("custom_pool") {
  depth = 1
}

action("action") {
  ...
  pool = ":custom_pool"
}
phosek@: that'll solve the "some actions are as heavy as linking and need to use a restricted pool issue" but won't really help the "system grinds to a halt when 200 random python actions all start grinding". I'd really like both like brettw@ suggests.
I think we need both.
OK, I'll try to address both.
Cc: tikuta@chromium.org
Blocking: 735361
FYI: change is in progress here: https://codereview.chromium.org/2926013002/
Project Member

Comment 18 by bugdroid1@chromium.org, Jun 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/43ee2b1163bdee1094aac234d25f52aab9decb62

commit 43ee2b1163bdee1094aac234d25f52aab9decb62
Author: phosek <phosek@chromium.org>
Date: Thu Jun 29 01:56:01 2017

Support explicit pools in actions

This change allows explicitly specifying pools for actions.
Furthermore, it is also possible to specify a default pool for
actions as part of the toolchain. Pools can be also defined as
console ones to emulate the original console attribute behavior.

BUG= 635308 

Review-Url: https://codereview.chromium.org/2926013002
Cr-Commit-Position: refs/heads/master@{#483252}

[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/action_target_generator.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/action_target_generator.h
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/action_values.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/action_values.h
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/builder.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/builder.h
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/command_help.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/docs/reference.md
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/function_toolchain.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/ninja_action_target_writer.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/ninja_action_target_writer_unittest.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/ninja_build_writer.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/ninja_toolchain_writer.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/pool.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/pool.h
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/toolchain.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/toolchain.h
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/variables.cc
[modify] https://crrev.com/43ee2b1163bdee1094aac234d25f52aab9decb62/tools/gn/variables.h

This issue needs more action?
I don't think so, I think this could be closed.
Status: Fixed (was: Available)
\o/

Sign in to add a comment