CQ builders failing InitalCheckout and CleanBuildRoot stages due to missing directories. |
|||
Issue descriptionhttps://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fcoral-paladin%2F3509%2F%2B%2Frecipes%2Fsteps%2FCleanBuildRoot%2F0%2Fstdout Traceback (most recent call last): File "/b/rr/tmpXYj6Wq/w/chromite/scripts/cbuildbot_launch", line 169, in <module> DoMain() File "/b/rr/tmpXYj6Wq/w/chromite/scripts/cbuildbot_launch", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/rr/tmpXYj6Wq/w/chromite/lib/commandline.py", line 911, in ScriptWrapperMain ret = target(argv[1:]) File "/b/rr/tmpXYj6Wq/w/chromite/scripts/cbuildbot_launch.py", line 514, in main return _main(argv) File "/b/rr/tmpXYj6Wq/w/chromite/scripts/cbuildbot_launch.py", line 482, in _main CleanBuildRoot(root, repo, metrics_fields, build_state) File "/b/rr/tmpXYj6Wq/w/chromite/scripts/cbuildbot_launch.py", line 71, in wrapped_functor result = functor(*args, **kwargs) File "/b/rr/tmpXYj6Wq/w/chromite/scripts/cbuildbot_launch.py", line 262, in CleanBuildRoot repo.CleanStaleLocks() File "/b/rr/tmpXYj6Wq/w/chromite/cbuildbot/repository.py", line 356, in CleanStaleLocks for attrs in git.ManifestCheckout.Cached(self.directory).ListCheckouts(): File "/b/rr/tmpXYj6Wq/w/chromite/lib/git.py", line 774, in Cached search=search) File "/b/rr/tmpXYj6Wq/w/chromite/lib/git.py", line 598, in _NormalizeArgs raise OSError(errno.ENOENT, "Couldn't find repo root: %s" % (path,)) OSError: [Errno 2] Couldn't find repo root: /b/c/cbuild/repository https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fdaisy_spring-paladin%2F18971%2F%2B%2Frecipes%2Fsteps%2FCleanBuildRoot%2F0%2Fstdout Traceback (most recent call last): File "/b/rr/tmp8ILmqq/w/chromite/scripts/cbuildbot_launch", line 169, in <module> DoMain() File "/b/rr/tmp8ILmqq/w/chromite/scripts/cbuildbot_launch", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/rr/tmp8ILmqq/w/chromite/lib/commandline.py", line 911, in ScriptWrapperMain ret = target(argv[1:]) File "/b/rr/tmp8ILmqq/w/chromite/scripts/cbuildbot_launch.py", line 514, in main return _main(argv) File "/b/rr/tmp8ILmqq/w/chromite/scripts/cbuildbot_launch.py", line 482, in _main CleanBuildRoot(root, repo, metrics_fields, build_state) File "/b/rr/tmp8ILmqq/w/chromite/scripts/cbuildbot_launch.py", line 71, in wrapped_functor result = functor(*args, **kwargs) File "/b/rr/tmp8ILmqq/w/chromite/scripts/cbuildbot_launch.py", line 262, in CleanBuildRoot repo.CleanStaleLocks() File "/b/rr/tmp8ILmqq/w/chromite/cbuildbot/repository.py", line 359, in CleanStaleLocks git.DeleteStaleLocks(repo_git_store) File "/b/rr/tmp8ILmqq/w/chromite/lib/git.py", line 1470, in DeleteStaleLocks raise GitException("Not a valid git repo: %s" % git_repo) chromite.lib.git.GitException: Not a valid git repo: /b/c/cbuild/repository/.repo/projects/src/third_party/kernel/v4.4.git https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fdaisy_spring-paladin%2F18970%2F%2B%2Frecipes%2Fsteps%2FCleanBuildRoot%2F0%2Fstdout [1;33m12:22:32: WARNING: Unable to read /b/c/cbuild/.cbuildbot_build_state.json: [Errno 2] No such file or directory: '/b/c/cbuild/.cbuildbot_build_state.json'[0m Unknown layout: Wiping buildroot.
,
Jun 6 2018
Here's what happened: 0. I landed a change a week ago to clean up stale git locks: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1075727 . It turns out that it doesn't handle the case where the repo is missing entirely or when some of the git repos are corrupted beyond recognition: it will crash in those cases. 1. A user clicked the "Clobber" checkbox while testing coral-paladin for an unrelated bug. This caused the chroot to be wiped when the build 3505 started: https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fcoral-paladin%2F3505%2F%2B%2Frecipes%2Fsteps%2FCleanUp%2F0%2Fstdout 2. The subsequent build started failing due to (0) because now the repo root is gone: https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin/builds/3506 . 3. The coral-paladin was unmarked as EXPERIMENTAL status making it critical for master paladin builds. 4. The next master-paladin build started but then immediately aborted because coral-paladin was failing: https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/18800 . It aborted all slaves including daisy_spring-paladin which cause that paladin to corrupt its git repositories. 5. Subsequent master build failed both coral-paladin and daisy_spring-paladin also because of (0): repo root gone and git repo is corrupted. 6. I chumped a fix in https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1089830 .
,
Jun 6 2018
coral-paladin and daisy_spring are marked experimental until the current run finishes to avoid impact the CL's in flight.
,
Jun 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/46153b54d95c1a5459b00c5a5c0ccc33431d3f42 commit 46153b54d95c1a5459b00c5a5c0ccc33431d3f42 Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Wed Jun 06 22:50:20 2018 Roll src/third_party/chromite 1a3e67e..874afdd (1 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/1a3e67e..874afdd git log 1a3e67e..874afdd --date=short --no-merges --format='%ad %ae %s' 2018-06-06 jclinton@chromium.org cbuildbot_launch: Blow away everything if cleaning up locks fails Created with: gclient setdep -r src/third_party/chromite@874afdd The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. BUG= chromium:850236 TBR=chrome-os-gardeners@chromium.org Change-Id: Id5e16d8dc9ad7f322e079286221bae29d82463aa Reviewed-on: https://chromium-review.googlesource.com/1089694 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#565073} [modify] https://crrev.com/46153b54d95c1a5459b00c5a5c0ccc33431d3f42/DEPS
,
Jun 7 2018
coral-paladin and daisy_spring-paladin are repaired. |
|||
►
Sign in to add a comment |
|||
Comment 1 by bugdroid1@chromium.org
, Jun 6 2018