Move cbuildbot's code sync logic into a separate script |
|
Issue descriptionThis script will be responsible for getting the CrOS source code into a requested state, e.g.: * `repo sync`ing to a particular branch or manifest snapshot * applying patches (e.g. Gerrit Changes) It will also attempt to be efficient about the sync: * git-cache * cache gerrit changes (?) This script will aim to replace various Sync stages in cbuildbot. Initially this may be done by replacing those stages' implementation with calls to the script.
,
Jul 18
> Do you plan to include this existing behavior? Ideally this script will only be responsible for checking out existing source to a given state. If there are practical problems with keeping this script focused on that task we can discuss.
,
Jul 18
The current system entangles, with some justification (mostly around optimizations), but maybe not a lot. It would be good to at least think through the workflow of creating a new release version pinned manifest in detail.
,
Aug 4
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/baeba30e635d4f309adb72730f7aa6be0b8e0183 commit baeba30e635d4f309adb72730f7aa6be0b8e0183 Author: Lann Martin <lannm@chromium.org> Date: Sat Aug 04 05:14:48 2018 Add git.RunRepo function This is a simple wrapper around the 'repo' command, just like RunGit wraps 'git'. BUG=chromium:864816 TEST=git_unittest Change-Id: I70a00c9eb322bdcd2ecc82872396ebd54577d3fc Reviewed-on: https://chromium-review.googlesource.com/1161052 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/baeba30e635d4f309adb72730f7aa6be0b8e0183/lib/git.py [modify] https://crrev.com/baeba30e635d4f309adb72730f7aa6be0b8e0183/lib/git_unittest.py
,
Aug 9
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/4b7a664ec8394c699849354cfe88cce4b5d06dd8 commit 4b7a664ec8394c699849354cfe88cce4b5d06dd8 Author: Lann Martin <lannm@chromium.org> Date: Thu Aug 09 09:21:25 2018 Create chromite.lib.repo_util and migrate autotest_evaluator to it This new module will house any (and ideally all) code that interacts directly with the `repo` CLI and '.repo' directories. Start by implementing enough functionality to replace direct calls to `repo` in autotest_evaluator with calls to this module. Remove recently-introduced chromite.lib.git.RunRepo. BUG=chromium:864816 TEST=lib/repo_util_unittest; cros_bisect/autotest_evaluator_unittest Change-Id: I95ec734fa50c16002fa3fc10a44dfb8bdcbbe5e5 Reviewed-on: https://chromium-review.googlesource.com/1168133 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/lib/repo_util_unittest.py [modify] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/cros_bisect/autotest_evaluator_unittest.py [modify] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/lib/git_unittest.py [add] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/lib/repo_util_unittest [modify] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/cros_bisect/autotest_evaluator.py [add] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/lib/repo_util.py [modify] https://crrev.com/4b7a664ec8394c699849354cfe88cce4b5d06dd8/lib/git.py
,
Aug 14
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/b26e129bfd2a0dbd5172e3cd5568b20212c53be2 commit b26e129bfd2a0dbd5172e3cd5568b20212c53be2 Author: Lann Martin <lannm@chromium.org> Date: Tue Aug 14 00:48:55 2018 Add repo_util.Repository.StartBranch and .MustFind Migrate cros_mark_as_stable script to use these methods. Also updated a direct git RunCommand to use git.RunGit. Renamed Repository.RunRepo to ._Run; external callers should use specific subcommand methods. Other changes to support executing repo from a checkout subdir, needed because projects can be referenced by relative path. BUG=chromium:864816 TEST=lib/repo_util_unittest; scripts/cros_mark_as_stable_unittest Change-Id: Ia2a012369e9f44d35d87c6458ad54b4fc93037f6 Reviewed-on: https://chromium-review.googlesource.com/1169745 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Lann Martin <lannm@chromium.org> [modify] https://crrev.com/b26e129bfd2a0dbd5172e3cd5568b20212c53be2/lib/repo_util.py [modify] https://crrev.com/b26e129bfd2a0dbd5172e3cd5568b20212c53be2/scripts/cros_mark_as_stable_unittest.py [modify] https://crrev.com/b26e129bfd2a0dbd5172e3cd5568b20212c53be2/lib/repo_util_unittest.py [modify] https://crrev.com/b26e129bfd2a0dbd5172e3cd5568b20212c53be2/scripts/cros_mark_as_stable.py
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/edc1873c9b879f4d512315c721c483321dd83915 commit edc1873c9b879f4d512315c721c483321dd83915 Author: Lann Martin <lannm@chromium.org> Date: Wed Aug 29 23:11:44 2018 Add repo_util.Repository.Copy method This method will make a copy of the Repository's .repo directory using hard links for git object files. BUG=chromium:864816 TEST=repo_util_unittest --network Change-Id: If5f881b6ba26022f578c00c7e332254f0f810c14 Reviewed-on: https://chromium-review.googlesource.com/1183806 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/edc1873c9b879f4d512315c721c483321dd83915/lib/repo_util.py [modify] https://crrev.com/edc1873c9b879f4d512315c721c483321dd83915/lib/repo_util_unittest.py
,
Sep 1
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/be95864f8b0b0ec1c945e9196c59475d9a405d8a commit be95864f8b0b0ec1c945e9196c59475d9a405d8a Author: Lann Martin <lannm@chromium.org> Date: Sat Sep 01 02:03:21 2018 Add chromite.lib.repo_manifest This module contains code for manipulating repo manifest XML data, to be used in an upcoming script to create reproducible manifest snapshots. BUG=chromium:864816 TEST=lib/git_unittest; lib/repo_manifest_unittest; lib/repo_util_unittest Change-Id: I2949ba453196e8ec32b8ee90bcde7fb0681685e0 Reviewed-on: https://chromium-review.googlesource.com/1195591 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/be95864f8b0b0ec1c945e9196c59475d9a405d8a/lib/repo_manifest_unittest [add] https://crrev.com/be95864f8b0b0ec1c945e9196c59475d9a405d8a/lib/repo_manifest_unittest.py [modify] https://crrev.com/be95864f8b0b0ec1c945e9196c59475d9a405d8a/lib/repo_util.py [add] https://crrev.com/be95864f8b0b0ec1c945e9196c59475d9a405d8a/lib/repo_manifest.py [modify] https://crrev.com/be95864f8b0b0ec1c945e9196c59475d9a405d8a/lib/repo_util_unittest.py
,
Sep 6
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/4fbdf209e2fb35de1d266658f3189a8d267e0f81 commit 4fbdf209e2fb35de1d266658f3189a8d267e0f81 Author: Lann Martin <lannm@chromium.org> Date: Thu Sep 06 04:04:20 2018 Add scripts/create_manifest_snapshot This script creates a manifest "snapshot" that can be used to reproduce a repo checkout on another host. For projects with local commits, those commits may be pushed to a designated remote ref to make those commits available remotely. Added git.IsReachable to help determine when remote refs need to be pushed. Made repo_manifest classes picklable to work with multiprocessing. BUG=chromium:864816 TEST=lib/git_unittest; scripts/create_manifest_snapshot_unittest Change-Id: I182995e8e825fe25826522b559f634ef43b8687f Reviewed-on: https://chromium-review.googlesource.com/1196833 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/scripts/create_manifest_snapshot_unittest.py [add] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/scripts/create_manifest_snapshot [modify] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/lib/git_unittest.py [modify] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/lib/repo_manifest.py [add] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/scripts/create_manifest_snapshot_unittest [add] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/scripts/create_manifest_snapshot.py [modify] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/lib/repo_manifest_unittest.py [modify] https://crrev.com/4fbdf209e2fb35de1d266658f3189a8d267e0f81/lib/git.py
,
Sep 26
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/10f3ee914d209c26f4f0e38c96b44853eb13ebd1 commit 10f3ee914d209c26f4f0e38c96b44853eb13ebd1 Author: Lann Martin <lannm@chromium.org> Date: Wed Sep 26 21:59:55 2018 Add scripts/repo_sync_manifest This script will sync a repo checkout to the given manifest file. The repo checkout can optionally be bootstrapped by copying from an existing repo checkout. BUG=chromium:864816 TEST=lib/repo_util_unittest; scripts/repo_sync_manifest_unittest Change-Id: Id5c5ca85d3f4da3eed0b3b019e2bfa3b833ed790 Reviewed-on: https://chromium-review.googlesource.com/1185889 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/10f3ee914d209c26f4f0e38c96b44853eb13ebd1/lib/repo_util_unittest.py [add] https://crrev.com/10f3ee914d209c26f4f0e38c96b44853eb13ebd1/scripts/repo_sync_manifest_unittest [add] https://crrev.com/10f3ee914d209c26f4f0e38c96b44853eb13ebd1/scripts/repo_sync_manifest_unittest.py [modify] https://crrev.com/10f3ee914d209c26f4f0e38c96b44853eb13ebd1/lib/repo_util.py [add] https://crrev.com/10f3ee914d209c26f4f0e38c96b44853eb13ebd1/scripts/repo_sync_manifest.py [add] https://crrev.com/10f3ee914d209c26f4f0e38c96b44853eb13ebd1/scripts/repo_sync_manifest
,
Nov 13
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c59d48ec1594e5ea2973e156d54f2b7572307ead commit c59d48ec1594e5ea2973e156d54f2b7572307ead Author: Don Garrett <dgarrett@google.com> Date: Tue Nov 13 14:42:23 2018 manifest_version: Add resolution helpers for buildspecs. To help standardize what a "buildspec" is, create two helper functions for resolving one into a filepath to a pinned manifest that matches that spec or version number. ResolveBuildspec ResolveBuildspecVersion BUG=chromium:864816 TEST=Unittests Change-Id: I4fae41c398ae48eed8a4c89a1280bee603f4fad6 Reviewed-on: https://chromium-review.googlesource.com/c/1327633 Tested-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> Reviewed-by: Alec Thilenius <athilenius@google.com> [modify] https://crrev.com/c59d48ec1594e5ea2973e156d54f2b7572307ead/cbuildbot/manifest_version.py [modify] https://crrev.com/c59d48ec1594e5ea2973e156d54f2b7572307ead/cbuildbot/manifest_version_unittest.py
,
Nov 14
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/903fedc7cf84523490a19d5a717d88e3fca15bdc commit 903fedc7cf84523490a19d5a717d88e3fca15bdc Author: Don Garrett <dgarrett@google.com> Date: Wed Nov 14 15:34:29 2018 repo_sync_manifest: Rework repo_sync_manifest. This script now supports syncing based on a variety of manifest sources (branch, buildspecs (based on manifest-verions), or a local manifest file). It also supports repo preloading, and git cache optimization hints. This switches the script from the repo_util library to the older repository library. It also enforces some cleanup of existing checkouts. Source will be detached, and any existing local branches will be deleted. Files not under verssion control will NOT be cleaned up, and other expensive cleaning operations (deleting git locks) will not be performed. It is NOT safe to assume that the checkout can be further updated with a simple "repo sync", instead you should call this script again with the same options. Additional CLs are coming which will add support for cherry-picking changes into the checkout during sync. BUG=chromium:864816 TEST=run_tests Change-Id: If8cd16c60f417e07d16d099d948b8ce53e4afdf6 Reviewed-on: https://chromium-review.googlesource.com/c/1316817 Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Alec Thilenius <athilenius@google.com> [modify] https://crrev.com/903fedc7cf84523490a19d5a717d88e3fca15bdc/cbuildbot/repository.py [modify] https://crrev.com/903fedc7cf84523490a19d5a717d88e3fca15bdc/scripts/repo_sync_manifest.py [modify] https://crrev.com/903fedc7cf84523490a19d5a717d88e3fca15bdc/scripts/repo_sync_manifest_unittest.py
,
Nov 27
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/115df409efa1466f47593767fc05e3d8af95c522 commit 115df409efa1466f47593767fc05e3d8af95c522 Author: Don Garrett <dgarrett@google.com> Date: Tue Nov 27 08:34:05 2018 repo_sync_manifest: Add -g cherry-pick support. Add support for cherry-picking Gerrit CLs into the checkout. This allows them to be cherry-picked in for cases that were not previously possible, such as based on version numbers or build specs. This CL does NOT include support for extracting changes to cherry-pick in from buildspec manifests. BUG=chromium:864816 TEST=run_tests Change-Id: I030b4ca9044ecaa3596a56b5e444a0e99df2fd99 Reviewed-on: https://chromium-review.googlesource.com/1330170 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Alec Thilenius <athilenius@google.com> [modify] https://crrev.com/115df409efa1466f47593767fc05e3d8af95c522/scripts/repo_sync_manifest.py |
|
►
Sign in to add a comment |
|
Comment 1 by dgarr...@chromium.org
, Jul 18