I tried to build chrome with jumbo configuration using goma with following configs, and become bit negative to use jumbo build with goma now.

Machine: 48hyperthread/24core Z840 Linux

args.gn:
"""
is_debug = false
use_goma = true
is_component_build = true
enable_nacl = false
strip_absolute_paths_from_debug_symbols = true
use_lld = true
use_jumbo_build = true
"""

command:
$ GOMA_STORE_ONLY=true time ~/git/ninja/ninja -C out/Release/ -j 1000 chrome # set env var not to use goma's backend cache

build time:
8:09.13  # 4:03.51 when same setting other than use_jumbo_build = false

ninja_log trace:
http://chromium-build-stats.appspot.com/ninja_log/upload/ninja_log.7YolbQqi2rVgUE7jo395fWi4cdiwv2yGSKbTTkKedXM=.gz

From ninja log, some v8 related objects took too long time (more than 2 minutes for 6 tasks) and there are many tasks waiting completion of v8 related tasks.
It depends on when and how we use jumbo build, but before enabling jumbo build, I think we need to fix issues like
https://bugs.chromium.org/p/chromium/issues/detail?id=725639
for faster build in highly parallelized build environment.

Jumbo builds are a tradeoff between the number of tasks/amount of parallelism and the compilation time per source code file, of course.  GOMA is at the extremely parallel end of this scale, and will probably not see as much benefit (if any at all) that someone building locally might.

At the moment, we have a single jumbo_file_merge_limit gn arg that specifies for all jumbo targets the maximum number of source files to merge into a single jumbo compilation unit, but some targets probably benefit from different values.  The question is then what would be a good way to specify this?

Brainstorming some possibilities:
* Add a new gn arg for each target that is possibly a bottleneck, and set appropriate values in args.gn.  Yuck.
* Do some sort of arithmetic in bottleneck targets and pass that to the internal_jumbo_target template, eg "local_jumbo_file_merge_limit = jumbo_file_merge_limit / 2", though I suspect that GN does not support arithmetic and if so this would require running a python script for each of these targets.
* Use an environment variables, eg if JUMBO_<target name> is set, use that value for jumbo_file_merge_limit in that target.  Might be good for developers, who want jumbo everywhere except the target they're working on.
* Simply break large targets into a parent and multiple source sets.  This would require manually keeping the sizes/build times of the siblings roughly equal over time.

Anyone have other suggestions?

Thank you for further explanation.

Jumbo build for v8 looks slower compared to other jumbo target from the ninja log.
Other targets do not let many targets wait long time.
If few components of jumbo build causes long wait time of other build tasks, we can keep size of tasks in such components manually.

@tikuta: I wonder if you could try making some more GOMA builds with different values for jumbo_file_merge_limit and see if you can find a sweet spot for a typical -j value?  The default value for jumbo_file_merge_limit is 100, so I would be interested to see how long it takes v8 to compile when this is set to say 50, 25 and 10, for GOMA "ninja -j 1000 v8" builds.  To remove v8's deps from the equation you could build v8 with the gn args being tested (ignore time), then touch all v8 source files and build again and measure the time taken.

Re specifying target-specific args, I have poked around in gn a little more:
* Re: "local_jumbo_file_merge_limit = jumbo_file_merge_limit / 2" doesn't work because gn currently only supports integer addition and subtraction.  Possible workaround: 'local_jumbo_file_merge_limit = exec_script("expr.py", [jumbo_file_merge_limit, "/", "2"], "value")' (could also implement max(), min() etc as required).
* New idea: add a "target_jumbo_file_merge_limit" that takes values that are a list of 2-element lists of the form [ <target name string>, <integer value for jumbo_file_merge_limit> ].  Then the internal_jumbo_target template can do a foreach on top-level list, if there's a target name match then use the second value for that item as the file merge limit.  Possible downside: it might be difficult to specify this arg in gn wrapper scripts (need to worry about escaping etc).

We could adjust jumbo_file_merge_limit to a much lower number (10?) when goma is enabled, although that doesn't handle the many-core case. But, it's a start. And we could come up with some recommendations for values based on core counts and document them, as a first start. Then we can figure out how to make it more automatic.

I tried to take stats with 70, 50, 25, 10 for local_jumbo_file_merge_limit.
But all configuration caused build failure on v8 component.

Is there known buildable parameter other than 100?

Not yet :(

Looks like there's a symbol clash between the Register structs in src/<arch>/assembler-<arch>.* and the Register struct in src/interpreter/bytecode-register.*, which happened to be ok with the default jumbo_file_merge_limit == 100.  This might require a moderate amount of renaming, or namespace tweaking to fix.  I will try a few options (renaming one or both of the symbols, adding a new namespace to distinguish the two) to see if one turns out obviously nicer than the others, but it will probably take me a few days to get back to you.

The hope was that any exact divisor of 100 would work. So, the failures on 70 are not surprising but the failures on 50, 25, and 10 are. Supporting arbitrary values is probably untenable but it would be good to support, say, 10 and 100 (for high-core and low-core counts).

v8 had a couple of files that were very slow to compile before jumbo and if those now end up in the same translation unit, then I can see how that translation unit can take an extreme time to get through the compiler.

From one of my test builds (times in seconds):
49.7 v8_base/objects.o
44.0 v8_base/code-stub-assembler.o
32.9 v8_base/api.o
30.5 v8_base/elements.o
25.9 v8_builtins_generators/builtins-regexp-gen.o
22.8 v8_base/parser.o
21.2 v8_base/heap.o

All of these are in the slowest 0.1% ninja jobs so they are extreme in some way. I think I would just exclude them all (or at least the 30s+ ones) completely from jumbo.

https://chromium-review.googlesource.com/c/v8/v8/+/637971

(I'm still working on a couple of alternatives for fixing jumbo builds for jumbo_file_merge_limit values other than 100.)

> I think I would just exclude them all (or at least the 30s+ ones) completely from jumbo.

That makes sense.

The CL above slows down jumbo builds by 10-11s (10m39.733s with, 10m28.253s without).

If I only exclude files that take longer than 30s to compile in Daniel's reference times, then I hit one of the issues that appears with other jumbo_file_merge_limit values (the Register struct/class clash).  So I will focus my attention on that.  Not sure if it's worth someone trying the CL above on GOMA in the meantime.

CLs posted to add builders (finally!):

https://chromium-review.googlesource.com/c/chromium/src/+/647705
https://chromium-review.googlesource.com/c/chromium/tools/build/+/647320

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c83f955c49a8770bfa9e6e776284903ad49ff71

commit 6c83f955c49a8770bfa9e6e776284903ad49ff71
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Sep 06 00:51:11 2017

Add configurations for the jumbo builders.

The "jumbo" build is a GN configuration where we compile
multiple C++ (or Obj-C++) files in a single translation unit;
this should lead to significant speedups in compile times.

However, we actually want to run benchmarks to validate this.
The "Jumbo {Linux,Mac,Win}" bots on the chromium.fyi waterfall
will build the same things as the {"Linux x64", "Mac", "Win x64"}
bots on the chromium waterfall, which will hopefully give
us something like an apples-to-apples comparison of compile times.

R=brucedawson@chromium.org, bratell@opera.com, jbudorick@chromium.org
BUG=745862
NOTRY=true
No-Equivalent-Builders: true


Change-Id: I8b637cff65a02cd4e1add349b636f14e8ce261ac
Reviewed-on: https://chromium-review.googlesource.com/647705
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#499821}
[modify] https://crrev.com/6c83f955c49a8770bfa9e6e776284903ad49ff71/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/6c83f955c49a8770bfa9e6e776284903ad49ff71/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/905214522b55a346a2e9d3b5eb08b3a1f7e21944

commit 905214522b55a346a2e9d3b5eb08b3a1f7e21944
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Sep 06 00:58:46 2017

Add jumbo builders to chromium.fyi

The "jumbo" build is a GN configuration where we compile
multiple C++ (or Obj-C++) files in a single translation unit;
this should lead to significant speedups in compile times.

However, we actually want to run benchmarks to validate this.
The "Jumbo {Linux,Mac,Win}" bots on the chromium.fyi waterfall
will build the same things as the {"Linux x64", "Mac", "Win x64"}
bots on the chromium waterfall, which will hopefully give
us something like an apples-to-apples comparison of compile times.

R=jbudorick@chromium.org, brucedawson@chromium.org, bratell@opera.com
BUG=745862

Change-Id: If750fdcc67e87283211a66ec19add21047af398f
Reviewed-on: https://chromium-review.googlesource.com/647320
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/905214522b55a346a2e9d3b5eb08b3a1f7e21944/masters/master.chromium.fyi/master.cfg
[modify] https://crrev.com/905214522b55a346a2e9d3b5eb08b3a1f7e21944/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
[modify] https://crrev.com/905214522b55a346a2e9d3b5eb08b3a1f7e21944/masters/master.chromium.fyi/slaves.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/42156d87e03605abcc64f41227805065dab5f646

commit 42156d87e03605abcc64f41227805065dab5f646
Author: Dirk Pranke <dpranke@google.com>
Date: Wed Sep 06 03:25:35 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/64d0d058cde01d466eaffacca5ecb220067edc69

commit 64d0d058cde01d466eaffacca5ecb220067edc69
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Sep 06 04:04:56 2017

Fix typo in chromium_fyi SPEC.

Apparently when I get fancy I should actually test what I'm
doing.

TBR=jbudorick@chromium.org
BUG=745862

Change-Id: Ic1bf6be16ad7adba56add7978ba048dd7f6e7964
Reviewed-on: https://chromium-review.googlesource.com/651141
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/64d0d058cde01d466eaffacca5ecb220067edc69/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/80440d5ab88ac0b9c8864f723bc19f619d96d208

commit 80440d5ab88ac0b9c8864f723bc19f619d96d208
Author: Dirk Pranke <dpranke@chromium.org>
Date: Thu Sep 07 01:48:08 2017

Switch the Jumbo builders to actually use jumbo builds.

This affects the "Jumbo Linux x64", "Jumbo Mac", and "Jumbo Win x64"
builders on the chromium.fyi waterfall.

R=bratell@opera.com, brucedawson@chromium.org
BUG=745862

Change-Id: Ie5f608d81f2a5e61e171de2c7fc1fa49855457ee
Reviewed-on: https://chromium-review.googlesource.com/654005
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#500177}
[modify] https://crrev.com/80440d5ab88ac0b9c8864f723bc19f619d96d208/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/7fed104d2a9cd6e214739bc29bb8edce7d35cf86

commit 7fed104d2a9cd6e214739bc29bb8edce7d35cf86
Author: Daniel Bratell <bratell@opera.com>
Date: Mon Oct 30 02:44:37 2017

Mail bratell, brucedawson and dpranke if jumbo builds break.

Bug: 745862
Change-Id: Id2c2b22f6cbcf4077a7a11481ebc957bac0ad2ed
Reviewed-on: https://chromium-review.googlesource.com/738036
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/7fed104d2a9cd6e214739bc29bb8edce7d35cf86/scripts/slave/gatekeeper.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c4dc5012c3861a2976a959a37680e57759e13ee1

commit c4dc5012c3861a2976a959a37680e57759e13ee1
Author: Mostyn Bramley-Moore <mostynb@vewd.com>
Date: Mon Oct 30 16:11:47 2017

mail Mostyn too, if linux jumbo builds fail

Bug: 745862
Change-Id: If4c0305d4129279a62c68f28b5da6aa2652309d9
Reviewed-on: https://chromium-review.googlesource.com/743017
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/c4dc5012c3861a2976a959a37680e57759e13ee1/scripts/slave/gatekeeper.json

I don't know when I'm going to have much time to work on this.

tikuta, martiniss - maybe this is something one of you'd be interested in?