cros-beefy388-c2 stuck in CleanBuildRoot for hours |
||||||
Issue descriptionconfig: reef-release Build: https://uberchromegw.corp.google.com/i/chromeos/builders/reef-release/builds/2003 Has been stuck in CleanBuildRoot for ~15 hours. Last relevant log line is: 18:14:48: INFO: Checkout cleanup failed, wiping buildroot: Traceback (most recent call last): File "/b/rr/tmpoCuOYR/w/chromite/scripts/cbuildbot_launch.py", line 199, in CleanBuildRoot repo.BuildRootGitCleanup(prune_all=True) File "/b/rr/tmpoCuOYR/w/chromite/cbuildbot/repository.py", line 412, in BuildRootGitCleanup git.ManifestCheckout.Cached(self.directory).ListCheckouts()] File "/b/rr/tmpoCuOYR/w/chromite/lib/git.py", line 771, in Cached search=search) File "/b/rr/tmpoCuOYR/w/chromite/lib/git.py", line 595, in _NormalizeArgs raise OSError(errno.ENOENT, "Couldn't find repo root: %s" % (path,)) OSError: [Errno 2] Couldn't find repo root: /b/c/cbuild/repository 18:14:48: INFO: RunCommand: sudo -- find /b/c/cbuild/repository -mindepth 1 -maxdepth 1 -exec rm -rf '{}' +
,
Mar 8 2018
,
Mar 8 2018
still stuck. I forced a clobber restart.
,
Mar 8 2018
It's a GCE instance (cros-beefy388-c2). I generally suggest reinstancing them when something like this happens, so you get back to a 100% known state.
,
Mar 8 2018
This is still happening. That's expected. A clobber restart only affects cbuildbot's behavior, but this problem is happening during cbuildbot_launch. I'm investigating the state of the builder so we can try to harden cbuildbot_launch, then I'll reinstance it.
,
Mar 8 2018
I'm unable to ssh in to investigate. Reinstanced. This should fix the problem, but doesn't explain it. Looks like some kind of general file system corruption?
,
Mar 8 2018
,
Mar 8 2018
Believed fixed, but waiting for the current build to get far enough along to confirm. https://uberchromegw.corp.google.com/i/chromeos/builders/reef-release/builds/2006
,
Mar 8 2018
2006 is on SetupBoard |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by pprabhu@chromium.org
, Mar 8 2018