TL;DR - 95% of the time is spent on linking. use_lld=true might help, otherwise more memory or an SSD.

Such consistently long times for non-clobber builds does seem weird.

I looked at one build that only had eleven changes and the compile stage still took 2 hours. There were only 2654 build steps, so 0.37 build steps per second with -j 80. Huh?

https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.fyi%2FJumbo_Win_x64%2F4579%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

So, I looked at the .ninja_log output. The post-build summary (it's based on my post-build script) shows that 95% of the weighted time is spent in linking:

https://chromium-build-stats.appspot.com/ninja_log/2018/01/29/vm140-m1/ninja_log.vm140-m1.chrome-bot.20180128-205521.4208.gz/table?dedup=true

Weighted time is what matters, so that tells us where the issue is. The trace-viewer summary of the build confirms this:

https://chromium-build-stats.appspot.com/ninja_log/2018/01/29/vm140-m1/ninja_log.vm140-m1.chrome-bot.20180128-205521.4208.gz/trace.html

Okay, so linking is the problem, and goma is irrelevant.
We could add is_win_fastlink but with goma the default is symbol_level=1 so fastlink won't help much (it might actually hurt).
We should (Dirk should :-) make sure the machine has an SSD since that makes linking way slower
We could/should try use_lld = true
chengx@ got a new machine a few weeks ago that didn't have an SSD and his link times were atrocious. For compilation it doesn't matter much, but for linking it is crucial. lld can paper over that to some extent and is easy to try, but for best results we need an SSD.

I don't know why the other builders aren't affected. Less disk I/O? Clean living? Maybe they actually have SSDs?

Thank you for taking a look Bruce.
As Bruce said, link time is very slow.

In addition to disk throughput, it looks symbol_level unintendedly becomes 2 in the builder because clang-cl supports pdb.
https://chromium.googlesource.com/chromium/src/+/070a9b22c39b0407ee0221dd765460f450a333cf/build/config/compiler/compiler.gni#190

Let me specify symbol_level=1 explicitly.


#2, I hear that win7 and win10 CQ tryserver using SSD now, but not sure on the builder.

> symbol_level unintendedly becomes 2

D'oh! Good catch. I hadn't realized that clang's defaults were different for goma/symbol_level. That will make a huge difference. Using Microsoft's linker with symbol_leve=2 and without is_win_fastlink definitely makes for painfully slow links.

Thanks for noticing this bratell@

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ad666b3d6b5c2e0a7149884e1ee1b2a1cd47ea75

commit ad666b3d6b5c2e0a7149884e1ee1b2a1cd47ea75
Author: Takuto Ikuta <tikuta@google.com>
Date: Thu Feb 01 03:51:02 2018

Specify minimal_symbols explicitly for Jumbo builder

This makes link time faster on the bots.

Bug:  807660 
Change-Id: Id4ae4a533971e53d5b095cec9bc73a5c273129af
Reviewed-on: https://chromium-review.googlesource.com/895426
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Daniel Bratell <bratell@opera.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Cr-Commit-Position: refs/heads/master@{#533546}
[modify] https://crrev.com/ad666b3d6b5c2e0a7149884e1ee1b2a1cd47ea75/tools/mb/mb_config.pyl

Decreasing symbol level fixed this issue.
https://ci.chromium.org/buildbot/chromium.fyi/Jumbo%20Win%20x64/4612
trace: https://chromium-build-stats.appspot.com/ninja_log/2018/02/01/vm140-m1/ninja_log.vm140-m1.chrome-bot.20180201-002602.4624.gz/trace.html

Link is dominant part yet on incremental build on the bots.
I'm looking forward use_lld=true

> Decreasing symbol level fixed this issue.

It really did! Thanks for looking into this and fixing it!