The arguments for 8 and 50 are obvious, but I'm not convinced we need a 29 as well; I understand that it might catch more issues, but we don't usually test configurations we don't expect anyone to actively be using.

The rational for an intermediate number is that it would catch issues earlier - issues that would eventually break the supported jumbo build configurations.

But, such a configuration is certainly less important, and could even be confusing I suppose.

Since merge_for_jumbo.py tries to balance jumbo units in a given target, small changes like the addition or removal of source files can alter the combinations significantly.  I think Daniel had an experiment where he set jumbo_file_merge_limit to the highest value possible with the amount of ram in the test machine, in order to increase the change of symbol collision so we could find and fix them before they happen in real configurations.

This might be even better than testing predefined jumbo_file_merge_limit values on goma, but it requires an unusual configuration: lots of ram and low number of ninja jobs (1?) to compensate, and might need to only make incremental builds to be fast enough.  I'm not sure how far he got with this experiment, though.

I don't know when I'm going to have the time to work on this. 

tikuta, martiniss@ - maybe this is something one of you can take over as part of your CQ cycle time work? 

We have 2 with goma default (5?) now and one, Linux, with hard coded 50. That helped us yesterday.

Having one with limit 9999 would be good but that one will be slower (though consume fewer electrons in total). I have tried keeping that configuration alive so it might just be a case of adding another few lines to the config scripts.

I think jumbo has large potential for faster CQ cycle time.
But for build speed on CQ, jumbo needs more tuning.
When using goma, few compile unit consumes too much time and resource is not utilized.

e.g.
https://ci.chromium.org/buildbot/chromium.fyi/Jumbo%20Win%20x64/4680
trace: https://chromium-build-stats.appspot.com/ninja_log/2018/02/02/vm140-m1/ninja_log.vm140-m1.chrome-bot.20180202-013138.2260.gz/trace.html

Sometimes even normal compile of small file takes much longer time than single jumbo compile task merged with several files.
Before changing/searching good merge limit value with/without goma, it is better to find time consuming single compile task and treat it as jumbo exception.
e.g. 
src/ui/gl/gl_mock.h
src/third_party/WebKit/Source/core/dom/Document.cpp
(Or we can ask clang team to improve compile speed for such slow compile?)

I see obj/third_party/WebKit/Source/bindings/core/v8/bindings_core_impl/bindings_core_impl_jumbo_19.obj taking 277 seconds in that trace which is of course way too long and it kind of surprises me since *all* of bindings (not just a part, all of it) build in 228 seconds on a single core here locally. 

But maybe this one was a fluke.

There are files that take too long in v8 which have been excluded for that reason, but we have seen as the main problem here that compilation get blocked on irrelevant earlier steps. It seems to be a side effect of gn being conservative in dependencies and we're not sure if we have any good solution.

I don't know if anyone in the clang project has looked at performance for these 100+ second compilations. Do you know anyone that might be interested in taking a look?

> since *all* of bindings ... build in 228 seconds on a single core here locally.

If that file was being compiled locally by itself then it would be easy for the compile to take almost twice as long in other circumstances. It is the nature of hyper-threading that it typically gives a very modest increase in total throughput, meaning that individual tasks often run at ~60% of normal speed (that corresponds to an overall 20% increase in throughput). Thus, 228 s to build when the only task running could easily become ~380 s when sharing a core with other tasks.

Or, maybe goma is slowing it down. It does some local work and then ships the file off for compilation and that can give different performance characteristics.

Either way, the best way to bring anomalously slow compilations to the attention of the clang team is to file bugs at https://bugs.llvm.org/. It's worth getting an account in order to do that.