Forking from internal bug b/30007023, idea for possibly making build faster. Filing a bug so we don't keep track of this, might be a nice 20% proj.

By looking at some ninja log traces from the bots (e.g., [1,2]) looks like the last ~minute of the C++ build phase (i.e. the pre-linking edges) are not that parallel. The reason seems to be the following by eye-balling the traces:
some, fortunately few, compilation units take a huge amount of time (minutes instead of seconds). when they are scheduled too late in a -j pool, all the other compilation tasks are done and the only thing ninja can do is sit there waiting for the long tasks to be run.
If we knew in advance (here's the tricky part) the duration of a task, ninja could do some fancier Least slack time scheduling. Or more simply just scheduling those jobs first could be a good enough approximation.


The idea here is that we could have a list of known long compilation units checked in the repo and feed that into ninja to bias its scheduling decisions. Might be worth a try, by eyeballing the traces looks like it could save approx 1min of build time.

Here's a graphic explanation of what I'm saying (I am in the ascii art mood today)

Current schedule
 1  +---------------------+
    |T1|T2|T3|     T4     |
    +---------------------+
                          |
 2  +----------------+    |
    |T5|T6|T7|   T8  |    |
    +----------------+    |
                          |
 3  +--+                  |
    |T9|                  |
    +--+                  |
                          |

Possibly smarter schedule
1  +---------------+      |
   |     T4     |T1|      |
   +---------------+      |
                   |      |
2  +-------------+ |      |
   |   T8  |T5|T6| |      |
   +-------------+ |      |
                   |      |
3  +-----------+   |      |
   |T9|T2|T3|T7|   |      |
   +-----------+   +      +

Here's the win:    <------>

(This assumes that all T1-T9 tasks are not dependent onto each other, which should be sufficiently true for most of our compilation tasks)

[1] https://chromium-build-stats.appspot.com/ninja_log/2016/07/12/build11-m1/ninja_log.build11-m1.chrome-bot.20160712-035857.4228.gz/trace.html
[2] http://chromium-build-stats.appspot.com/ninja_log/2016/07/12/slave136-c1/ninja_log.slave136-c1.chrome-bot.20160712-044917.12688.gz/trace.html