Optimize AP firmware build time |
||||||||
Issue descriptionBuilding a complete firmware for the AP (coreboot (+ arm-trusted-firmware on some platforms) + libpayload + depthcharge + chromeos-bootimage) takes >4 minutes, even with a pair of 12x2 E5-2690 v3 CPUs and an SSD. This should not be. Are there any steps we can do to optimize the development flow here? For instance, it seems like there are several steps that get run multiple times, one for each of the types of image we want to build. I know that some firmware folks tend to avoid emerge entirely, and use some shortcut incremental build approach to combat this. But this seems like it's easy to cause problems and is more of a workaround than anything.
,
Jan 6 2017
,
Jan 6 2017
,
Jan 6 2017
,
Jan 6 2017
,
Jan 6 2017
Random smallish thing I noticed: >>> Preparing source in /build/kevin/tmp/portage/sys-boot/coreboot-0.0.1-r1898/work/coreboot-0.0.1 ... make -j48 clean Why do we have to clean the source tree, even for non-worked-on builds (which get git-cloned in; they shouldn't copy any local developer artifacts)? This adds several seconds to both depthcharge and coreboot.
,
Jan 6 2017
Whoa, so just 'emerge-kevin chromeos-bootimage' still takes over 3 minutes on ToT. Samus is more like 1 minute. $ time emerge-kevin chromeos-bootimage !!! CONFIG_PROTECT is empty for '/build/kevin/' Calculating dependencies... done! >>> Emerging (1 of 1) sys-boot/chromeos-bootimage-0.0.2-r1346::chromiumos for /build/kevin/ * Running stacked hooks for pre_pkg_setup * sysroot_build_bin_dir ... [ ok ] * Running stacked hooks for pre_src_unpack * python_multilib_setup ... [ ok ] >>> Unpacking source... Cloning into '/build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/work/chromeos-bootimage-0.0.2'... done. >>> Source unpacked in /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/work * Running stacked hooks for post_src_unpack * asan_init ... [ ok ] * Running stacked hooks for pre_src_prepare * use_gcc ... [ ok ] >>> Preparing source in /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/work/chromeos-bootimage-0.0.2 ... >>> Source prepared. >>> Configuring source in /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/work/chromeos-bootimage-0.0.2 ... >>> Source configured. >>> Compiling source in /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/work/chromeos-bootimage-0.0.2 ... * Add static assets to images Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... Performing operation on 'COREBOOT' region... * No legacy boot payloads specified. * Building production image. Performing operation on 'COREBOOT' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'VBLOCK_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'VBLOCK_B' region... * Building serial image. Performing operation on 'COREBOOT' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'VBLOCK_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'VBLOCK_B' region... * Building developer image. Performing operation on 'COREBOOT' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'VBLOCK_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'VBLOCK_B' region... * Building netboot image. Performing operation on 'COREBOOT' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'FW_MAIN_A' region... Performing operation on 'VBLOCK_A' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'FW_MAIN_B' region... Performing operation on 'VBLOCK_B' region... * Netboot configured to boot briannorris/kevin/vmlinuz, fetch kernel command line from briannorris/kevin/cmdline, and use the DHCP-provided TFTP server IP. >>> Source compiled. >>> Test phase [not enabled]: sys-boot/chromeos-bootimage-0.0.2-r1346 >>> Install chromeos-bootimage-0.0.2-r1346 into /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/image/ category sys-boot >>> Completed installing chromeos-bootimage-0.0.2-r1346 into /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346/image/ * Generating license for sys-boot/chromeos-bootimage-0.0.2-r1346 in /build/kevin/tmp/portage/sys-boot/chromeos-bootimage-0.0.2-r1346 11:26:56: INFO: Read licenses for sys-boot/chromeos-bootimage-0.0.2-r1346: GPL-2 11:26:56: INFO: sys-boot/chromeos-bootimage-0.0.2-r1346: using stock|cust license(s) GPL-2 * Removing /usr/lib*/*.la * Removing /etc/init.d * Removing /etc/conf.d * Removing /etc/logrotate.d * Removing /etc/sandbox.d * Removing /usr/share/bash-completion tar: Write checkpoint 1000 tar: Write checkpoint 2000 tar: Write checkpoint 3000 >>> Done. >>> Installing (1 of 1) sys-boot/chromeos-bootimage-0.0.2-r1346::chromiumos to /build/kevin/ * Removing /usr/lib*/*.la * Removing /etc/init.d * Removing /etc/conf.d * Removing /etc/logrotate.d * Removing /etc/sandbox.d * Removing /usr/share/bash-completion * Removing /usr/share/man * Removing /usr/share/info * Removing /usr/share/doc * Running stacked hooks for pre_pkg_preinst * wrap_old_config_scripts ... [ ok ] >>> Auto-cleaning packages... >>> Using system located in ROOT tree /build/kevin/ >>> No outdated packages were found on your system. real 3m11.866s user 3m7.863s sys 0m8.518s
,
Jan 6 2017
#6 Brian, I believe this was explicitly added because people had issues with local artifacts being copied in. Is there a better way to prevent this?
,
Jan 6 2017
#6 Brian: I guess it might be worth to look into optimizing the "clean" targets. Right now, I think they collect a list of object files to build and then delete them. We should be able to do better (and remove dangling files from deleted/renamed source files, too) Will check if I can find something to do here. #7 Brian: I'll see if I can reproduce this (and check where exactly it's slow - I have some ideas) all in all, keep them coming :-)
,
Jan 6 2017
#8: I was essentially suggesting we could avoid doing this for the non-cros-workon build. e.g., for people building locally, but not running "cros_workon-${BOARD} start coreboot depthcharge".
i.e., this:
https://chromium-review.googlesource.com/425451
I've only sort of tested this. I'm pretty sure you don't need to 'clean' a non-9999 build.
,
Jan 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/954596e0e0450a370c133b5b2f8cb0f51ad8302e commit 954596e0e0450a370c133b5b2f8cb0f51ad8302e Author: Brian Norris <briannorris@chromium.org> Date: Fri Jan 06 21:54:56 2017 coreboot, depthcharge: don't 'clean' for non-worked-on builds The source tree will not be dirty when building from a git checkout. BUG= chromium:630451 TEST=emerge with and without `cros_workon` Change-Id: I62cda35b60981e0da9fbb820320343e897a05edb Signed-off-by: Brian Norris <briannorris@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/425451 Reviewed-by: Patrick Georgi <pgeorgi@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/954596e0e0450a370c133b5b2f8cb0f51ad8302e/sys-boot/coreboot/coreboot-9999.ebuild [modify] https://crrev.com/954596e0e0450a370c133b5b2f8cb0f51ad8302e/sys-boot/depthcharge/depthcharge-9999.ebuild
,
Jan 7 2017
The difference between kevin and samus is likely a result of the size of the recovery screen bitmaps (using eve instead of samus because that's what I had set up): 16M /build/eve/firmware/rocbfs/ 62M /build/kevin/firmware/rocbfs/ Those are pushed through an lzma compressor with no parallelization whatsoever, which makes for noticably slower addition to CBFS. Just watch the output of emerge-$board chromeos-bootimage after " * Add static assets to images": The following lines "Performing operation on 'COREBOOT' region..." (each covers compression and addition of a single file) show up slower for kevin than for eve. There's room for improvement, as "xz *" (which should be roughly comparable in terms of complexity, but an incompatible bitstream) on those files takes 15 and 3 seconds respectively. We have to double those numbers because we add the files to two images, but even then that's way less than what's currently going on. The compressor in cbfstool is just _very_ slow. I'll see how to improve that. After that, it could be parallelized (at a nearly linear speedup by CPU count) and deduplicated (compress once, add twice) to improve matters even more. There's also some effort to rework the recovery screen handling which might make all of this moot, which is why I postponed it - but the performance on kevin builds really is hideous.
,
Jan 13 2017
https://chromium-review.googlesource.com/#/q/status:open+owner:pgeorgi%2540chromium.org+topic:speedup deals with the chromeos-bootimage step for me, down from 3:10 to 0:25 on kevin, samus down from 1:40 to 0:20. The speedup on build_image-style runs might be smaller (because there can be as many emerge jobs in parallel as there are cores), but for local builds on powerful machines, this avoids approx. nproc-1 CPU threads from dying from boredom. (and it avoids compressing everything _twice_) Review appreciated!
,
Jan 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/coreboot/+/bc3460aa659cf4ae736ca3e4b7b838701e22b174 commit bc3460aa659cf4ae736ca3e4b7b838701e22b174 Author: Patrick Georgi <pgeorgi@chromium.org> Date: Wed Jan 11 16:17:18 2017 UPSTREAM: util/cbfstool: compile with -O2 by default This speeds up the lzma encoder approximately four-fold. BUG= chromium:630451 BRANCH=none TEST=emerge-$board chromeos-bootimage's set of adding static assets it noticeably faster Change-Id: Ie8cc9b6106ac72c0b0e96bcd76bb7d13d48b2025 Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Original-Change-Id: Ibf896098799693ddd0f8a6c74bda2e518ecea869 Original-Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Original-Reviewed-on: https://review.coreboot.org/18098 Original-Tested-by: build bot (Jenkins) Original-Reviewed-by: Stefan Reinauer <stefan.reinauer@coreboot.org> Original-Reviewed-by: Paul Menzel <paulepanter@users.sourceforge.net> Reviewed-on: https://chromium-review.googlesource.com/427701 Reviewed-by: Stefan Reinauer <reinauer@chromium.org> [modify] https://crrev.com/bc3460aa659cf4ae736ca3e4b7b838701e22b174/util/cbfstool/Makefile.inc
,
Jan 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/coreboot/+/f905d67b49547e385c671caf8aea117dc394668e commit f905d67b49547e385c671caf8aea117dc394668e Author: Patrick Georgi <pgeorgi@chromium.org> Date: Wed Jan 11 14:26:58 2017 UPSTREAM: util/cbfstool: Add cbfs-compression-tool cbfs-compression-tool provides a way to benchmark the compression algorithms as used by cbfstool (and coreboot) and allows to pre-compress data for later consumption by cbfstool (once it supports the format). For an impression, the benchmark's results on my machine: measuring 'none' compressing 10485760 bytes to 10485760 took 0 seconds measuring 'LZMA' compressing 10485760 bytes to 1736 took 2 seconds measuring 'LZ4' compressing 10485760 bytes to 41880 took 0 seconds And a possible use for external compression, parallel and non-parallel (60MB in 53 files compressed to 650KB on a machine with 40 threads): $ time (ls -1 *.* |xargs -n 1 -P $(nproc) -I '{}' cbfs-compression-tool compress '{}' out/'{}' LZMA) real 0m0.786s user 0m11.440s sys 0m0.044s $ time (ls -1 *.* |xargs -n 1 -P 1 -I '{}' cbfs-compression-tool compress '{}' out/'{}' LZMA) real 0m10.444s user 0m10.280s sys 0m0.064s BUG= chromium:630451 BRANCH=none TEST=manual execution of the tool works Change-Id: If2ac452dae4180b5df516a99808008ce41922621 Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Original-Change-Id: I40be087e85d09a895b1ed277270350ab65a4d6d4 Original-Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Original-Reviewed-on: https://review.coreboot.org/18099 Original-Tested-by: build bot (Jenkins) Original-Reviewed-by: Martin Roth <martinroth@google.com> Reviewed-on: https://chromium-review.googlesource.com/427702 Reviewed-by: Stefan Reinauer <reinauer@chromium.org> [modify] https://crrev.com/f905d67b49547e385c671caf8aea117dc394668e/util/cbfstool/cbfs_image.c [modify] https://crrev.com/f905d67b49547e385c671caf8aea117dc394668e/util/cbfstool/cbfs.h [add] https://crrev.com/f905d67b49547e385c671caf8aea117dc394668e/util/cbfstool/cbfscomptool.c [modify] https://crrev.com/f905d67b49547e385c671caf8aea117dc394668e/util/cbfstool/common.h [modify] https://crrev.com/f905d67b49547e385c671caf8aea117dc394668e/util/cbfstool/Makefile.inc [modify] https://crrev.com/f905d67b49547e385c671caf8aea117dc394668e/util/cbfstool/Makefile
,
Jan 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/coreboot/+/59e999ed718868c5fac08296c91ca571f50a12d1 commit 59e999ed718868c5fac08296c91ca571f50a12d1 Author: Patrick Georgi <pgeorgi@chromium.org> Date: Wed Jan 11 17:38:11 2017 UPSTREAM: util/cbfstool: Enable adding precompressed files to cbfs cbfstool ... add ... -c precompression assumes the input file to be created by cbfs-compression-tool's compress command and uses that to add the file with correct metadata. When adding the locale_*.bin files to Chrome OS images, this provides a nice speedup (since we can parallelize the precompression and avoid compressing everything twice) while creating a bit-identical file. BUG= chromium:630451 BRANCH=none TEST=with the necessary tweaks to the build system, emerge-kevin chromeos-bootimage takes 0:25 instead of 3:10 before on the z620 I work on, about a magnitude faster. Change-Id: Ib99b8c4960e174ea5b9a5077ca49992a93d7bd41 Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Original-Change-Id: Iadd106672c505909528b55e2cd43c914b95b6c6d Original-Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Original-Reviewed-on: https://review.coreboot.org/18102 Original-Tested-by: build bot (Jenkins) Original-Reviewed-by: Nico Huber <nico.h@gmx.de> Reviewed-on: https://chromium-review.googlesource.com/427703 Reviewed-by: Stefan Reinauer <reinauer@chromium.org> [modify] https://crrev.com/59e999ed718868c5fac08296c91ca571f50a12d1/util/cbfstool/cbfstool.c
,
Jan 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/0a25f1410ac6670a08509881e2945bd230eff9b9 commit 0a25f1410ac6670a08509881e2945bd230eff9b9 Author: Patrick Georgi <pgeorgi@chromium.org> Date: Fri Jan 13 16:16:42 2017 coreboot-utils, chromeos-bootimage: parallelize adding graphic assets Install cbfs-compression-tool as part of coreboot-utils and use it in chromeos-bootimage to parallelize lzma compressing the recovery screen bitmap assets, for a nice speed-up. BUG= chromium:630451 BRANCH=none CQ-DEPEND=CL:427703 TEST=emerge-kevin chromeos-bootimage takes 0:25 on my z620 instead of 3:10 before, the result is bit-identical. Change-Id: Ia93e4fe2673a5685306bc13ee6a153dc66e1b9eb Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/427762 Reviewed-by: Stefan Reinauer <reinauer@chromium.org> [modify] https://crrev.com/0a25f1410ac6670a08509881e2945bd230eff9b9/sys-boot/chromeos-bootimage/chromeos-bootimage-9999.ebuild [modify] https://crrev.com/0a25f1410ac6670a08509881e2945bd230eff9b9/sys-apps/coreboot-utils/coreboot-utils-9999.ebuild
,
Jan 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a73c33ccf39aad4b7e186de2133ab27d430d20c0 commit a73c33ccf39aad4b7e186de2133ab27d430d20c0 Author: Patrick Georgi <pgeorgi@chromium.org> Date: Thu Jan 19 11:52:23 2017 coreboot, libpayload: remove build left overs manually "make clean" takes while to clean up the tree completely (in case of coreboot, incl the payload directories) due to more complicated discovery of what needs cleaning. Instead, just remove the things that get in the way of _our_ builds, ie. the content of the build directory. In case of coreboot, also move objutil in its own directory so kconfig and various other parts aren't built twice. This shaves off 1 minute from $ emerge-kevin libpayload depthcharge chromeos-bmpblk chromeos-ec coreboot chromeos-bootimage on my system (with a hot filesystem cache), from 4:23 to 3:20. BUG= chromium:630451 TEST=run with and without change, measure speed improvement Change-Id: I86276956cb2cf3cc945c2e923ea8813ea9c2e61c Signed-off-by: Patrick Georgi <pgeorgi@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/430710 Reviewed-by: Julius Werner <jwerner@chromium.org> [modify] https://crrev.com/a73c33ccf39aad4b7e186de2133ab27d430d20c0/sys-boot/libpayload/libpayload-9999.ebuild [modify] https://crrev.com/a73c33ccf39aad4b7e186de2133ab27d430d20c0/sys-boot/coreboot/coreboot-9999.ebuild
,
Aug 22 2017
Just like with every software, every time it's optimized to be smaller or more efficient (or running on a bigger/faster machine), some new feature comes along and eats into that new improvement. This means: we can either consider this concrete instance done (with the 3:00 -> 0:25 improvement), encouraging folks to open new issues when there are new bottlenecks or we'll never close this issue because of that effect.
,
Jan 22 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by pgeorgi@chromium.org
, Jan 6 2017