The ToT bots all don't use goma and do clobber builds, so they have fairly stable build performance.

All bots I just looked at regressed build perf by about 10% a week ago. Not clear if this is due to a chrome or a clang change. For one bot, I looked through all chrome and clang commits, but nothing jumped out.

https://build.chromium.org/p/chromium.fyi/stats/ClangToTLinux -> view source, look for "stepTimescompile" -> looks like this started in build 6978 -> https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux/builds/6978 That has got_clang_revision 287040, the previous build has clang 286993

Same exercise for ClangToTWin(dbg) -> https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%28dbg%29/builds/7106 clang range 287012:287071 

Those two ranges intersect, which I guess is nice. `svn log -r287012:287040 https://nico@llvm.org/svn/llvm-project` doesn't print anything very interesting.

Comparing https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%28dbg%29/builds/7106/steps/compile/logs/stdio and https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%28dbg%29/builds/7105/steps/compile/logs/stdio , the latter build does build a few more steps, but not 10% more. Diffing the output (after some massaging), the main difference is more irt_x86 and irt_x64 steps.

Repeating the dance for https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux/builds/6978/steps/compile/logs/stdio and https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux/builds/6977/steps/compile/logs/stdio also shows a couple hundred more irt_ commands, so that's probably the cause.

Not sure if this is a clang or a chrome thing, or if it's even actionable. Probably something started depending on some targets under the irt toolchain more, so this is probably a chromium-side regression. I don't see what introduced it. Dirk, do you see what might've caused this? (it might not even be a problem, but if it was unintentional...)

(inglorion: just fyi, the analysis approach might be interesting to you since you've probably not seen it yet)


(FWIW I looked at the windows build graph to check if https://codereview.chromium.org/2520863002/ helped with clang-cl build times at all. To my surprise, it apparently has no effect at all (?) sigbjorn rnk fyi)
 

Deleted: Screen Shot 2016-11-23 at 2.00.24 PM.png 201 KB

	Screen Shot 2016-11-23 at 2.00.24 PM.png 201 KB View Download