Optimize factory branch builders' build time and output size. |
|||||||||
Issue description
What steps will reproduce the problem?
(1) Kick a build on factory branches
What is the expected output?
The image folder should contain only files we need: The factory.zip and signed factory shim.
What do you see instead?
Lots of files that factory branch does not need.
If we take cyan for example, a latest build has:
au-generator.zip 5.97 MB
chromeos_7458.218.0_cyan_factory_canary-channel_mp.bin 82.18 MB
chromeos_7458.218.0_cyan_factory_canary-channel_mp.bin.json 1.33 KB
chromeos_7458.218.0_cyan_factory_canary-channel_mp.bin.md5 89 B
ChromeOS-factory-R47-7458.218.0-cyan.instructions 217 B
ChromeOS-factory-R47-7458.218.0-cyan.instructions.json 1.33 KB
ChromeOS-factory-R47-7458.218.0-cyan.zip 257.8 MB
ChromeOS-R47-7458.218.0-cyan.zip 2.71 GB application/zip 2016/10/17 下午2:00
ChromeOS-recovery-R47-7458.218.0-cyan.tar.xz 494.09 MB
ChromeOS-test-R47-7458.218.0-cyan.tar.xz 692.09 MB
debug-cyan.tgz
And the files we really need are:
chromeos_7458.218.0_cyan_factory_canary-channel_mp.bin 82.18 MB
ChromeOS-factory-R47-7458.218.0-cyan.zip 257.8 MB
ChromeOS-test-R47-7458.218.0-cyan.tar.xz 692.09 MB
These files are not needed:
au-generator.zip 5.97 MB
ChromeOS-R47-7458.218.0-cyan.zip 2.71 GB application/zip 2016/10/17 下午2:00
ChromeOS-recovery-R47-7458.218.0-cyan.tar.xz 494.09 MB
debug-cyan.tgz
In other words, we can save 3G+ for space.
The proposal would be to
- change cbuildbot/chromeos_config.py to reduce factory build options. for example,
cbuildbot/chromeos_config.py site_config.AddTemplate(
'factory',
site_config.templates.release,
- upload_hw_test_artifacts=False,
- upload_symbols=False,
- hw_tests=[],
+ site_config.templates.no_hwtest_builder,
+ site_config.templates.no_unittest_builder,
+ site_config.templates.no_vmtest_builder,
+ afdo_use=False,
chrome_sdk=False,
description='Factory Builds',
+ factory_toolkit=True,
+ hwqual=False,
+ images=['test', 'factory_install'],
+ image_test=False,
paygen=False,
- afdo_use=False,
+ push_image=True,
+ signer_tests=False,
sign_types=['factory'],
+ upload_hw_test_artifacts=False,
+ upload_symbols=False,
)
Meanwhile, we want to see if we can create "factory bundles" directly.
A "factory bundle" is a repacked versionof factory.zip that added
- test image
- signed release image
- signed factory install shim
Currently creation of factory bundles were done manually by downloading factory.zip then run finalize_bundle.py (in factory.zip/factory_setup folder).
However this has few problems:
1. Partners can't do this on their own easily since they can't download images by simple scripting (via CPFE).
2. Partners does not have an easy way to download "full, complete, official release of all files they need from Google" - if it's them who execute the script.
A proposal is to have the repack script called when chromite is creating factory.zip, that it'll check manifest and see if the expected bundle was already created or not.
That brings a question: Can we upload the generated bundle to somewhere that is dedicated for bundles in buildbot stage, for example
gs://chromeos-releases/canary-channel/x86-zgb/factory_bundles/factory_bundle-zgb-pvt-201610109.zip
Will creating a 'factory_bundles' folder that is not version number (like 8913.0.0) cause problems?
,
Oct 19 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a533bbc6af8b1504405648af84f3a7f16c1ca56a commit a533bbc6af8b1504405648af84f3a7f16c1ca56a Author: Hung-Te Lin <hungte@chromium.org> Date: Wed Oct 19 09:53:57 2016 cbuildbot: Optimize factory branch build time and space. Currently factory branch builders are generating many files we don't need, and running tests that are not required for factory. This change tries to minimize the artifacts in output, and to reduce build time. BUG=chromium:657295 TEST=./cbuildbot x86-mario-factory --local -g CL:400302 Succeed on chromiumos.tryserver/builders/factory/builds/4 Original-Reviewed-URL: https://chromium-review.googlesource.com/#/400302 (Cherry-picked from I8e2d4f6b1faa62bb945bb7237e2f609a39013d23) Change-Id: I51e2cb5fba460b567a557c1e769e481b2f8cbd69 Reviewed-on: https://chromium-review.googlesource.com/400978 Tested-by: Hung-Te Lin <hungte@chromium.org> Reviewed-by: Wei-Han Chen <stimim@chromium.org> Commit-Queue: Wei-Han Chen <stimim@chromium.org> [modify] https://crrev.com/a533bbc6af8b1504405648af84f3a7f16c1ca56a/cbuildbot/config_dump.json [modify] https://crrev.com/a533bbc6af8b1504405648af84f3a7f16c1ca56a/cbuildbot/chromeos_config.py
,
Oct 19 2016
I'll try the change on gale and strago factory branches first, and see if we need to do change more flags.
,
Oct 20 2016
Experiment on strago. - Build output looks good. storage/browser/chromeos-releases/canary-channel/strago/7458.221.0/ storage/browser/chromeos-releases/canary-channel/wizpig/7458.221.0/ However, chromite unittest failed. I'll check later tonight.
,
Oct 20 2016
for strago, disk space saved 2.3G. We may save more if we discard debug-*.zip (which is usually not important for 99% of the case - but i haven't decided if we should just remove it), that's 2.8G. If we do more aggressive changes (allow not uploading image.zip) then we can get more 800MBs back.
,
Oct 21 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/e53f056bad9b6a62c74145792954a3ecd53e090c commit e53f056bad9b6a62c74145792954a3ecd53e090c Author: Hung-Te Lin <hungte@chromium.org> Date: Wed Oct 19 09:53:57 2016 cbuildbot: Optimize factory branch build time and space. Currently factory branch builders are generating many files we don't need, and running tests that are not required for factory. This change tries to minimize the artifacts in output, and to reduce build time. BUG=chromium:657295 TEST=./cbuildbot x86-mario-factory --local -g CL:400302 Succeed on chromiumos.tryserver/builders/factory/builds/4 Original-Reviewed-URL: https://chromium-review.googlesource.com/#/c/400978/ (cherry picked from commit a533bbc6af8b1504405648af84f3a7f16c1ca56a) Change-Id: I6fff91a626bf93aab30847deaaa7fb489b1a0656 Reviewed-on: https://chromium-review.googlesource.com/401059 Tested-by: Hung-Te Lin <hungte@chromium.org> Reviewed-by: Shun-Hsing Ou <shunhsingou@chromium.org> Reviewed-by: YH Lin <yueherngl@chromium.org> Commit-Queue: Hung-Te Lin <hungte@chromium.org> Trybot-Ready: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/e53f056bad9b6a62c74145792954a3ecd53e090c/cbuildbot/config_dump.json [modify] https://crrev.com/e53f056bad9b6a62c74145792954a3ecd53e090c/cbuildbot/chromeos_config.py
,
Oct 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a2dbe6437d364c3f36eee1f9182a7d426b1af735 commit a2dbe6437d364c3f36eee1f9182a7d426b1af735 Author: Hung-Te Lin <hungte@chromium.org> Date: Wed Oct 19 09:53:57 2016 cbuildbot: Optimize factory branch build time and space. Currently factory branch builders are generating many files we don't need, and running tests that are not required for factory. This change tries to minimize the artifacts in output, and to reduce build time. BUG=chromium:657295 TEST=./cbuildbot x86-mario-factory --local -g CL:400302 Succeed on chromiumos.tryserver/builders/factory/builds/4 Change-Id: I8e2d4f6b1faa62bb945bb7237e2f609a39013d23 Reviewed-on: https://chromium-review.googlesource.com/400302 Commit-Ready: Hung-Te Lin <hungte@chromium.org> Tested-by: Hung-Te Lin <hungte@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/a2dbe6437d364c3f36eee1f9182a7d426b1af735/cbuildbot/config_dump.json [modify] https://crrev.com/a2dbe6437d364c3f36eee1f9182a7d426b1af735/cbuildbot/chromeos_config.py
,
Feb 1 2017
,
Feb 10 2017
,
Nov 14 2017
,
Mar 30 2018
,
Mar 30 2018
,
Aug 3
This bug has an owner, thus, it's been triaged. Changing status to "assigned". |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by bugdroid1@chromium.org
, Oct 19 2016