precise64 bots build more targets that needed |
|||||
Issue descriptionThere are multiple complaints from the test team that long build times on precise64 desktop release builders harm their productivity. See, for example, https://crbug.com/641343, https://crbug.com/641536 . One legitimate reason for the Linux builds being slow is that they are doing builds with LTO (aka LinkTimeOptimization). This is planned to be addressed by deploying ThinLTO: https://crbug.com/660216 . While it's definitely coming, ThinLTO needs to be stabilized before being rolled into the official builders. After looking into the list of targets being built on the precise64 bots (they call ninja -C <..> All), the following targets are built but not used: 1. Fuzzers: v8_simple_json_fuzzer - 220 sec v8_simple_parser_fuzzer - 250 sec v8_simple_regexp_fuzzer - 250 sec v8_simple_wasm_asmjs_fuzzer - 220 sec v8_simple_wasm_fuzzer - 250 sec cast_remoting_connector_fuzzer - 5 sec Most of the fuzzers are disabled in the official builds, because use_libfuzzer=false on these bots. For some reason these 7 targets are still enabled. 2. Examples and shells: examples_with_content_main_exe -- 1500 sec views_examples_exe -- 250 sec v8_hello_world -- 200 sec v8_sample_process -- 200 sec aura_demo -- 200 sec content_shell -- 1500 sec blimp_shell -- 850 sec blimp_engine_app -- 1400 sec app_shell -- 1400 sec ash_shell_with_content -- 1500 sec gin_shell -- 200 sec sync_client -- 250 sec v8_shell -- 200 sec v8_parser_shell - 230 sec <and others> At the first glance, the outputs of those targets are not used on the precise64 bot: https://uberchromegw.corp.google.com/i/official.desktop/builders/precise64. They are not even in the list of files to be uploaded to the cloud storage. Would it be reasonable instead of building All targets, build only those we use? In total, we could speed up the release build by at least 30 minutes, and may be closer to 1 hour. Nico, do you have any context in your head about this?
,
Dec 6 2016
For fuzzers, it feels that there is no legitimate purpose for them being built at all in this environment. What's about the examples and shells? They are taking much more time in total and to me, it's a pure waste of cycles, both machine and human.
,
Dec 6 2016
examples and hello world programs are unimportant for sure. blimp is dead now, so we can delete those targets. We use some v8 shell to compile the js tests, but that might go through d8 instead of v8_shell. But that one might be needed. I don't think we ship content_shell or ash_shell, so I think these can probably go too.
,
Dec 6 2016
Max, can you please take a look why do these fuzzers being built? As far as I can tell, use_libfuzzer gn variable (which is false by default) is supposed to guard us against that, and it mostly works, but not always, as you can see, and it has a non zero cost.
,
Dec 6 2016
>examples and hello world programs are unimportant for sure. good. > blimp is dead now, so we can delete those targets. do you mean delete from the repo? I will be happy to do that, can you please file an issue with more details about what needs to be deleted? > We use some v8 shell to compile the js tests, but that might go through d8 instead of v8_shell. But that one might be needed. It's relatively cheap (200 seconds and the linker takes just one slot out of 14). Let's keep it. > I don't think we ship content_shell or ash_shell, so I think these can probably go too. good It seems that we should start with fuzzers and blimp first, and then go after the rest of the unneeded stuff.
,
Dec 9 2016
Sorry for the delayed response (I've been OOO). Yes, I'll take a look into those fuzzers.
,
Dec 9 2016
Thank you, Max!
,
Dec 9 2016
fuzzer_test GN template has a bit more complicated condition than 'if use_libfuzzer': https://cs.chromium.org/chromium/src/testing/libfuzzer/fuzzer_test.gni?q=fuzzer_test&sq=package:chromium&l=21 so it's always possible to build fuzzer_test targets on Linux (unless 'disable_libfuzzer=true' set explicitly) v8_simple.*_fuzzer targets come from here: https://cs.chromium.org/chromium/src/v8/BUILD.gn?q=v8_simple_json_fuzzer&sq=package:chromium&l=2482 CC'ing machenbach@ here to ask if that's really needed for V8. And cast_remoting_connector_fuzzer probably comes from https://cs.chromium.org/chromium/src/BUILD.gn?q=cast_remoting_connector_fuzzer&sq=package:chromium&l=849 CC'ing miu@ to check if the time for TODO has come :)
,
Dec 9 2016
Oops, forgot to CC folks. Please take a look at c#8, would appreciate your comments on that :)
,
Dec 12 2016
Most of the extra V8 targets sneaked in when switching to gn. I think in gyp we needed to add those targets explicitly in order to have them built in chromium. In gn, the "All" target seems to just take everything it finds. We do not need many of these v8 targets compiled in a chromium context, but we'd still want the targets to exist (in gn) so that we could compile them on-demand. What we need: v8_shell - is used on the host during compilation What we'd like to have: d8, cctest, unittests The rest is not required. E.g. no simple_fuzzer targets.
,
Dec 12 2016
I agree that most of these targets should exist in GN with all flags, but for fuzzers specifically it makes sense to follow the way it's in in src/: only add fuzzers into the tree, if use_libfuzzer=true.
,
Dec 12 2016
>> only add fuzzers into the tree, if use_libfuzzer=true. Today it makes sense but in (near?) future I'd like to build all fuzz targets in regular build modes as regression tests. https://bugs.chromium.org/p/chromium/issues/detail?id=653139 It can be made in yet another special build mode though.
,
Dec 12 2016
In this case, it does worth the effort here. That leaves us with the only viable way to not build unnecessary targets: instead of All, introduce another GN target specifically for the desktop builders.
,
Dec 12 2016
The simple fuzzer targets in V8 aren't actual libfuzzer targets. They're dummy stand-alone versions of the respective libfuzzer targets without libfuzzer for smoke testing in V8 stand-alone. We could also introduce another switch that differentiates between V8 stand-alone and V8 in-Chromium, controlled through the build_overrides/v8.gni file. Then we could simply guard/hide a bunch of targets in V8 stand-alone.
,
Dec 12 2016
If that's the right way to go, please assign the task to me. CC dpranke to reason about comment 14 as well.
,
Dec 12 2016
Per kcc@ (#12), excluding fuzzers from GN tree is not the right way to go in the longer term, no action is needed. Thank you for your readiness to help.
,
Dec 12 2016
As mentioned it would only be an exclusion of V8's dummy targets from the GN tree in Chromium. The GN tree in V8 would still allow to build those targets and run our smoke tests. In Chromium, they have no real meaning.
,
Jan 9 2017
re: comment #14 - there is already a flag in //build_overrides/build.gni called `build_with_chromium` that you can use.
,
Jan 9 2017
CL for excluding most of the unneeded v8 targets: https://codereview.chromium.org/2619743002/
,
Jan 9 2017
developers need those targets, and the bots should make sure that they build in all configurations. If the official builders don't need all targets, they shouldn't build them (i.e. don't build "all" but introduce a special target for the official builder that depends on whatever the builder needs). Also, we should just increase goma capacity so the bots are faster again.
,
Jan 9 2017
(This is about linking with LTO, so it's independent of goma capacity. I don't disagree with the rest though.)
,
Aug 1
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by thakis@chromium.org
, Dec 6 2016