move binpkg publishing to new -postsubmit builders |
||||||
Issue descriptionCurrently, each paladin uploads binpkgs after it passes its local build phases. These get published if the whole master-paladin passes. Since we often go multiple days between green CQ runs lately, this means we only get new binpkgs a couple of times per week. This forces every CQ run and every developer to rebuild dozens to hundreds of extra packages repeatedly. What if we move binpkg publishing to the release builders instead? Every CL picked up by the release builders has already gone through CQ or been chumped, so there shouldn't be any more risk of picking up an untested CL than there is now. release builders happen a couple of times per day instead of every 3-4 hours, but this is still a lot shorter than the current situation.
,
Oct 24
To expound on the above edit: we are working on a design that covers this request. It might make sense to sequence this feature earlier rather than later.
,
Oct 24
That's reasonable, and possibly as easy as making a number of build config changes. This has the nice effect of not requiring more build machines. Also considering a new class of postsubmit builder, which is more work, but not a huge amount of work.
,
Oct 25
,
Oct 25
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/f74d12074e1988ffc5095a6310a82cd0549256c7 commit f74d12074e1988ffc5095a6310a82cd0549256c7 Author: Don Garrett <dgarrett@google.com> Date: Thu Oct 25 22:09:35 2018 postsubmit: Create a new 'postsubmit' class of builder. They don't do much yet, but create the framework for our postsubmit builders. BUG= chromium:898541 TEST=None Change-Id: Ibc0432d5491d06ceaeff86db9368ad82ef9b0641 Reviewed-on: https://chromium-review.googlesource.com/c/1299459 Tested-by: Don Garrett <dgarrett@chromium.org> Trybot-Ready: Don Garrett <dgarrett@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/config/chromeos_config.py [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/config/waterfall_layout_dump.txt [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/lib/config_lib.py [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/config/config_dump.json [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/config/chromeos_config_unittest.py [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/config/luci-scheduler.cfg [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/lib/constants.py [modify] https://crrev.com/f74d12074e1988ffc5095a6310a82cd0549256c7/cbuildbot/lkgm_manager.py
,
Nov 2
Mike, you said that you might be able to produce the list of boards that should have binary prebuilts generated for them? Here is test config today: https://cs.corp.google.com/chromeos_public/chromite/config/chromeos_config.py?rcl=5dbb07d706256e7607fa71f9e2e72784dd8c0e6b&l=2157 It would be easy to either use "all boards" or "all boards with (important?) CQ builders", if either of those lists seems correct. We intend to alert on postsubmit failures in time.
,
Nov 12
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/64388846458a6a1664b176d0b673342af168b3af commit 64388846458a6a1664b176d0b673342af168b3af Author: Don Garrett <dgarrett@google.com> Date: Mon Nov 12 19:29:51 2018 chromeos_config: Adjust postsubmit configs. Configure postsubmit builds to run the expected list of stages, and no more. BUG= chromium:898541 TEST=None Change-Id: I611f3df00a287f83a768a6c606058d772d6d87b6 Reviewed-on: https://chromium-review.googlesource.com/c/1302893 Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/64388846458a6a1664b176d0b673342af168b3af/config/chromeos_config.py [modify] https://crrev.com/64388846458a6a1664b176d0b673342af168b3af/cbuildbot/prebuilts.py [modify] https://crrev.com/64388846458a6a1664b176d0b673342af168b3af/config/config_dump.json
,
Nov 12
The new builders are building and uploading presubmits as expected, but not updating BINHOST, which means the prebuilts won't be used. I haven't yet been able to figure out what's different between these builds and full builders (which do update BINHOST).
,
Nov 27
This is blocked on shutting down the chromeos.release waterfall and migrating it's GCE builders to swarming bots, so that we'll have enough build capacity.
,
Dec 4
Nice progress!
,
Dec 5
Not really. I had it that far along in a couple of days (+ CQ time), but haven't done much since then.
,
Dec 5
Different relevant command lines: Presubmit upload command: 18:15:12: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/bin/upload_prebuilts --build-path /b/swarming/w/ir/cache/cbuild/repository --board samus --prepend-version postsubmit --upload gs://chromeos-prebuilt --private --binhost-conf-dir src/private-overlays/chromeos-partner-overlay/chromeos/binhost --key FULL_BINHOST in /b/swarming/w/ir/cache/cbuild/repository Full builder upload command: 09:30:51: INFO: RunCommand: /b/swarming/wMmHSxg/ir/cache/cbuild/repository/chromite/bin/upload_prebuilts --build-path /b/swarming/wMmHSxg/ir/cache/cbuild/repository --board tael --set-version R73-11342.0.0-rc2 --git-sync --prepend-version full --upload gs://chromeos-prebuilt --binhost-conf-dir src/third_party/chromiumos-overlay/chromeos/binhost --key FULL_BINHOST in /b/swarming/wMmHSxg/ir/cache/cbuild/repository The paladins upload CLs on the child builds, but only commit them from the master. Paladin builder upload command: 12:41:27: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/chromite/bin/upload_prebuilts --build-path /b/swarming/w/ir/cache/cbuild/repository --board coral --set-version R73-11344.0.0-rc1 --previous-binhost-url gs://chromeos-prebuilt/board/amd64-generic/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/board/coral/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/board/caroline/chrome-R73-11333.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/host/amd64/tatl/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/host/amd64/tael/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/host/amd64/moblab-generic-vm/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R73-11342.0.0-rc3/packages/ --previous-binhost-url gs://chromeos-prebuilt/host/amd64/amd64-host/chroot-2018.12.03.164623/packages/ --prepend-version paladin --upload gs://chromeos-prebuilt --private --binhost-conf-dir src/private-overlays/chromeos-partner-overlay/chromeos/binhost --key PREFLIGHT_BINHOST in /b/swarming/w/ir/cache/cbuild/repository master-paladin: 11:11:14: INFO: RunCommand: /b/swarming/wDqtHiK/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable commit --all '--boards=' '--drop_file=/b/swarming/wDqtHiK/ir/cache/cbuild/repository/src/scripts/cbuildbot_package.list' --buildroot /b/swarming/wDqtHiK/ir/cache/cbuild/repository --overlay-type both in /b/swarming/wDqtHiK/ir/cache/cbuild/repository
,
Dec 5
The two options to "upload_prebuilts" that are missing on the postsubmit builders are: --set-version R73-11342.0.0-rc2 --git-sync I believe that setting both of those will allow postsubmit builders to commit prebuilts. I'm intentionally avoiding having the master part of the process. This is intended to ensure we publish more quickly, and that we can do so, even if some boards fail to compile.
,
Dec 5
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/fa74b5c9e6051c181070b539fbfe8fe1987bd928 commit fa74b5c9e6051c181070b539fbfe8fe1987bd928 Author: Don Garrett <dgarrett@google.com> Date: Wed Dec 05 02:40:37 2018 chromeos_config: Add git_sync to postsubmit builders. Enable git_sync to the postsubmit build configs, and use a manifest versioned sync, so that we'll have a version for the postsubmits. BUG= chromium:898541 TEST=run_tests Change-Id: I2bd372681d045575c67b443332fe08193656a1b4 Reviewed-on: https://chromium-review.googlesource.com/1362513 Commit-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/fa74b5c9e6051c181070b539fbfe8fe1987bd928/config/chromeos_config.py [modify] https://crrev.com/fa74b5c9e6051c181070b539fbfe8fe1987bd928/config/config_dump.json
,
Dec 5
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/9ac1758ffa0c2189f2346cc89a84673dc4bdd2d4 commit 9ac1758ffa0c2189f2346cc89a84673dc4bdd2d4 Author: Don Garrett <dgarrett@google.com> Date: Wed Dec 05 22:00:56 2018 chromeos_config: Create postsubmit for every release board. Define a full set of postsubmit builders, based on the boards represented in a release builder. BUG= chromium:898541 TEST=run_tests Change-Id: I946cba28ddf74ea070559c883035f0cdcbf6030d Reviewed-on: https://chromium-review.googlesource.com/1362517 Commit-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> [modify] https://crrev.com/9ac1758ffa0c2189f2346cc89a84673dc4bdd2d4/config/chromeos_config.py [modify] https://crrev.com/9ac1758ffa0c2189f2346cc89a84673dc4bdd2d4/config/waterfall_layout_dump.txt [modify] https://crrev.com/9ac1758ffa0c2189f2346cc89a84673dc4bdd2d4/config/config_dump.json
,
Dec 6
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/d138913917ce0c16c2932a3e02711477b123627c commit d138913917ce0c16c2932a3e02711477b123627c Author: Don Garrett <dgarrett@google.com> Date: Thu Dec 06 01:12:25 2018 master-postsubmit: Use MasterSlaveLKGMSyncStage. Postsubmit builders were using ManifestVersionedSyncStage, which meant that the ChromeOS version number was incremented for each build. Switch to MasterSlaveLKGMSyncStage, which uses <version>-rc<N> versions instead. BUG= chromium:898541 TEST=run_tests Change-Id: Iabb54533b7590a474acacc1dc9c14394c64c77d7 Reviewed-on: https://chromium-review.googlesource.com/c/1363815 Tested-by: Don Garrett <dgarrett@chromium.org> Trybot-Ready: Don Garrett <dgarrett@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/d138913917ce0c16c2932a3e02711477b123627c/cbuildbot/builders/simple_builders.py
,
Dec 7
,
Dec 7
These builders are now in production. I'll file new bugs to cover the known improvements / problems. I've filed these bugs to cover remaining issues: https://crbug.com/913094 https://crbug.com/913095
,
Dec 10
,
Dec 11
,
Dec 11
That was an existing problem, and the large number of new builders has probably made it worse. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by jclinton@google.com
, Oct 24Status: Available (was: Untriaged)
Summary: move binpkg publishing to new -postsubmit builders (was: move binpkg publishing to release builders)