Clobber build speed regressed a lot with recent clang |
|||
Issue descriptionhttps://build.chromium.org/p/chromium.fyi/stats/ClangToTLinux uses clang trunk and hence can't use goma and does local builds. Look at the compile graph -- it's not looking pretty :-( Not clear if bot config change or clang or what.
,
Mar 8 2016
Last good according to that table: https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux/builds/4471 clang r 262315 First bad: https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux/builds/4495 clang r 262447 The builds in between had failing compiles due to clang crashing.
,
Mar 8 2016
,
Mar 8 2016
I failed to reproduce this locally when building both base_unittests and content. Hence I'm guessing something happened that made linking take longer, and it requires linking many binaries to repro this.
,
Mar 8 2016
Traces don't look very enlightening and don't obviously back up the "many links" theory: http://chromium-build-stats.appspot.com/ninja_log/2016/03/01/slave3-c1/ninja_log.slave3-c1.chrome-bot.20160301-040157.5773.gz/trace.html http://chromium-build-stats.appspot.com/ninja_log/2016/03/02/slave3-c1/ninja_log.slave3-c1.chrome-bot.20160301-202324.31278.gz/trace.html
,
Mar 8 2016
Aha, with linux_dump_symbols=1 I can repro building just base_unittests. That takes about 3:30 with the old clang and about 3:58 with the new clang.
,
Mar 8 2016
(...so it's some debug info change)
,
Mar 8 2016
debug info changes from looking through `svn log -r 262315:262447 https://nico@llvm.org/svn/llvm-project/`: r262339 r262361 The first feels more likely.
,
Mar 8 2016
Hm, I retract comment 6 (and 7), I had accidentally enabled goma for these builds, which makes the build timings useless. With goma disabled, I'm back at not being able to repro this locally with base_unittests, even with linux_dump_symbols=1 set.
,
Mar 8 2016
I tried building the same way that bot does: GYP_DEFINES="clang_use_chrome_plugins=0 component=shared_library linux_dump_symbols=1 target_arch=x64 test_isolation_mode=prepare" build/gyp_chromium time ninja -C out/Release/ all I tried this with clang 262817 and clang 261368-1. In both cases, both builds needed almost exactly the same amount of time (26.5 min) :-/
,
Mar 8 2016
Also happens on ClangToTLinux (dbg) which is on a different slave. So it's not limited to a single build slave. https://build.chromium.org/p/chromium.fyi/stats/ClangToTLinux%20(dbg) var stepTimescompile = [ [0, 7019.88221908], // build 2758 [1, 6955.29116392], // build 2759 [2, 6963.3603971], // build 2760 [3, 6985.12014985], // build 2761 [4, 7006.17214179], // build 2762 [5, 6916.46946311], // build 2763 [6, 6987.17480206], // build 2764 [7, 6894.80592895], // build 2765 [8, 6954.08649015], // build 2767 [9, 7037.20642996], // build 2768 [10, 7038.11938691], // build 2769 [11, 7047.41698813], // build 2770 [12, 6976.42197204], // build 2771 [13, 7061.43442202], // build 2772 [14, 7035.59835005], // build 2773 [15, 7220.91753983], // build 2776 [16, 7688.89696813], // build 2777 [17, 7689.65450191], // build 2778 [18, 7771.90366888], // build 2779 [19, 7714.40111303], // build 2780 [20, 7715.62023902], // build 2781 [21, 7711.17391706], // build 2785 [22, 7659.6538341], // build 2786 [23, 7697.86096096], // build 2787 [24, 7734.859653], // build 2788 [25, 7694.2813921], // build 2789 [26, 7688.10086799], // build 2790 [27, 7640.59005904], // build 2791 [28, 7633.88227987], // build 2792 [29, 7778.11030006], // build 2794 [30, 7694.09511518], // build 2811 [31, 7719.041852], // build 2812 [32, 7676.098737], // build 2813 [33, 7692.2761209], // build 2814 [34, 7704.67007399], // build 2815 [35, 7661.58972001], // build 2816 [36, 7755.731529], // build 2817 [37, 7711.70651507], // build 2819 [38, 7856.61518097], // build 2820 [39, 7958.18020701], // build 2821 [40, 7882.10634494], // build 2822 [41, 7906.71503687], // build 2823 [42, 7709.49205399], // build 2824 [43, 7828.96392989], // build 2825 [44, 7934.8862021], // build 2826 [45, 7915.80251122], // build 2827 [46, 7849.76700687], // build 2828 [47, 7928.032305], // build 2829 [48, 7919.15391898], // build 2830 [49, 7935.35645103], // build 2831 [50, 7877.25000501], // build 2833 [51, 8198.34286809], // build 2834 [52, 8174.70465088], // build 2835 [53, 8130.18780208], // build 2836 [54, 8187.98074293], // build 2837 [55, 8187.65501785], // build 2839 [56, 8237.04644394], // build 2840 [57, 8224.696527], // build 2841 [58, 8217.88988709], // build 2842 [59, 8226.53422499], // build 2843 [60, 8220.89647698], // build 2844 [61, 8253.43387794], // build 2845 [62, 8237.25331116], // build 2846 [63, 8153.87939], // build 2847 [64, 8097.17342091], // build 2848 [65, 3674.87555385], // build 2849 [66, 8164.92522001], // build 2850 [67, 8217.275208], // build 2851 [68, 8185.19732809] // build 2853 ]; Almost looks like two jumps here…
,
Mar 8 2016
Last good: https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux%20%28dbg%29/builds/2776 clang r 261672 First bad: https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux%20%28dbg%29/builds/2777 clang r 261707 Hm, that's well before the revision range on the release bot. Maybe the debug bot takes longer to cycle, and hence we only see this larger jump on its history? Let's see where the debug bot's second jump is at: Good https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux%20%28dbg%29/builds/2833 clang r 262645 Bad https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinux%20%28dbg%29/builds/2834 clang r 262679 ...nope, that's _after_ the release regression.
,
Mar 10 2016
It looks like bots started timing out during compile at the same time as today's clang roll, which is somewhat suspicious (https://bugs.chromium.org/p/chromium/issues/detail?id=593891#c18). So maybe this will be a hard blocker for the next roll.
,
Mar 11 2016
Theory: Maybe something either made make clang need much more memory, or made it produce .o files that make ld use much more memory, and maybe the memory/processor ratio on my linux desktop is high enough that this isn't a problem here, but it causes heavy swapping on machines where that ratio is lower. I'll try to find a machine with less memory per core and try to repro there.
,
Mar 11 2016
The second jump might be due to https://llvm.org/bugs/show_bug.cgi?id=26918
,
Nov 10 2017
|
|||
►
Sign in to add a comment |
|||
Comment 1 by thakis@chromium.org
, Mar 8 2016From "view source" on that stats page: var stepTimescompile = [ [0, 7526.97706294], // build 4453 [1, 7539.64770889], // build 4454 [2, 7523.12252092], // build 4455 [3, 1978.10185885], // build 4456 [4, 7528.57257295], // build 4457 [5, 3533.57877421], // build 4458 [6, 393.565494061], // build 4459 [7, 7533.19857502], // build 4460 [8, 7477.60910416], // build 4461 [9, 7467.35311604], // build 4462 [10, 7519.86322308], // build 4463 [11, 7483.17713809], // build 4464 [12, 7516.35947299], // build 4465 [13, 7540.96134186], // build 4466 [14, 7570.04225087], // build 4467 [15, 7536.02793097], // build 4468 [16, 7547.31357789], // build 4470 [17, 7598.09055686], // build 4471 [18, 8513.4341011], // build 4495 [19, 8496.07906294], // build 4496 [20, 8392.09541297], // build 4497 [21, 8483.39231491], // build 4499 [22, 8490.66596389], // build 4500 [23, 8479.60659885], // build 4501 [24, 8608.66878104], // build 4502 [25, 8524.58581805], // build 4503 [26, 8448.9735558], // build 4504 [27, 8471.57602596], // build 4505 [28, 8475.62165594], // build 4506 [29, 8564.89161992], // build 4507 [30, 8595.37368989], // build 4508 [31, 8566.90202999], // build 4510 [32, 8807.66656685], // build 4511 [33, 8691.61180902], // build 4512 [34, 8679.60070896], // build 4513 [35, 8705.66281199], // build 4514 [36, 8720.15801382], // build 4515 [37, 8764.67968392], // build 4516 [38, 8791.89431715], // build 4517 [39, 8816.69473195], // build 4518 [40, 8719.99218702], // build 4519 [41, 8668.33809114], // build 4520 [42, 8777.07674408], // build 4521 [43, 8778.38750696], // build 4522 [44, 205.987664938], // build 4523 [45, 355.459419966], // build 4524 [46, 8564.44531012], // build 4525 [47, 8781.21405005], // build 4526 [48, 8762.37868285], // build 4527 [49, 8765.90583897], // build 4528 [50, 8732.04628015], // build 4529 [51, 8714.12603998], // build 4530 [52, 8727.68310308], // build 4531 [53, 8732.93338013], // build 4532 [54, 8797.2023499], // build 4533 [55, 8834.01507211], // build 4534 [56, 8798.15282321], // build 4535 [57, 8679.40933609], // build 4536 [58, 2220.83204699], // build 4537 [59, 2324.68723822], // build 4538 [60, 8670.97405791], // build 4539 [61, 8729.11721802], // build 4540 [62, 8727.37540817], // build 4541 [63, 8846.60946107], // build 4542 [64, 8835.4372592], // build 4543 [65, 8703.82588196], // build 4546 [66, 8776.21905994], // build 4547 [67, 8697.42341089] // build 4548