CQ chroot cleanup failed |
|||
Issue descriptionhttps://luci-milo.appspot.com/buildbot/chromeos/nyan-full-compile-paladin/12641 ************************************************************ ** Start Stage CleanUp - Tue, 15 May 2018 09:33:02 -0700 (PDT) ** ** Stages that cleans up build artifacts from previous runs. ** ** This stage cleans up previous KVM state, temporary git commits, ** clobbers, and wipes tmp inside the chroot. ************************************************************ 09:33:02: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f9d4fe2ff90> Preconditions for the stage successfully met. Beginning to execute stage... 09:33:02: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f9d4fe2f650> 09:33:02: INFO: Refreshing access_token 09:33:03: INFO: Previous local build buildbucket_id=8946446495671154720 finished in state fail. 09:33:03: INFO: Previous local build 12640 did not pass. Cannot reuse chroot. 09:33:03: INFO: Attempting to revert chroot to clean-chroot snapshot 09:33:03: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk --snapshot-list in /b/c/cbuild/repository [1;33m09:33:04: WARNING: Failed to activate VG on try 1.[0m [1;33m09:33:08: WARNING: Failed to activate VG on try 2.[0m [1;33m09:33:08: WARNING: Failed to activate VG on try 3.[0m cros_sdk: Unhandled exception: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/bin/cros_sdk", line 169, in <module> DoMain() File "/b/c/cbuild/repository/chromite/bin/cros_sdk", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 911, in ScriptWrapperMain ret = target(argv[1:]) File "/b/c/cbuild/repository/chromite/scripts/cros_sdk.py", line 948, in main cros_build_lib.MountChroot(options.chroot, create=False) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 1622, in MountChroot SudoRunCommand(cmd, capture_output=True, print_cmd=False) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 294, in SudoRunCommand return RunCommand(cmd, **kwargs) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 658, in RunCommand raise RunCommandError(msg, cmd_result) chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: vgchange -q -ay cros_b+c+cbuild+repository+chroot_000 Check of thin pool cros_b+c+cbuild+repository+chroot_000/thinpool failed (status:1). Manual repair required (thin_dump --repair /dev/mapper/cros_b+c+cbuild+repository+chroot_000-thinpool_tmeta)! 0 logical volume(s) in volume group "cros_b+c+cbuild+repository+chroot_000" now active cmd=['vgchange', '-q', '-ay', 'cros_b+c+cbuild+repository+chroot_000'] [1;31m09:33:09: ERROR: return code: 1; command: /b/c/cbuild/repository/chromite/bin/cros_sdk --snapshot-list cmd=['/b/c/cbuild/repository/chromite/bin/cros_sdk', '--snapshot-list'], cwd=/b/c/cbuild/repository[0m [1;31m09:33:09: ERROR: <class 'chromite.lib.failures_lib.BuildScriptFailure'>: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1) Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor return functor(*args, **kwargs) File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 353, in PerformStage delete_chroot = not self._RevertChrootToCleanSnapshot() File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 68, in _RevertChrootToCleanSnapshot snapshots = commands.ListChrootSnapshots(self._build_root) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 196, in ListChrootSnapshots redirect_stdout=True) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript raise failures_lib.BuildScriptFailure(ex, cmd[0]) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript raise failures_lib.BuildScriptFailure(ex, cmd[0]) BuildScriptFailure: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1) [0m 09:33:09: INFO: Translating result <class 'chromite.lib.failures_lib.BuildScriptFailure'>: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1) Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor return functor(*args, **kwargs) File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 353, in PerformStage delete_chroot = not self._RevertChrootToCleanSnapshot() File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 68, in _RevertChrootToCleanSnapshot snapshots = commands.ListChrootSnapshots(self._build_root) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 196, in ListChrootSnapshots redirect_stdout=True) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript raise failures_lib.BuildScriptFailure(ex, cmd[0]) File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript raise failures_lib.BuildScriptFailure(ex, cmd[0]) BuildScriptFailure: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1) to fail. 09:33:09: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f9d4fd83710> 09:33:09: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f9d4fd83a90> 09:33:09: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f9d4fd83b10> ************************************************************ ** Finished Stage CleanUp - Tue, 15 May 2018 09:33:09 -0700 (PDT) ************************************************************
,
May 16 2018
I cleaned up that particular builder and confirmed that subsequent builds are green. I'll send out a CL to make the CleanUp stage handle it by itself.
,
May 16 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/516b94cffe0090291073fea0c6029233fcacf05a commit 516b94cffe0090291073fea0c6029233fcacf05a Author: Benjamin Gordon <bmgordon@chromium.org> Date: Wed May 16 23:42:59 2018 build_stages: Handle exceptions in snapshot commands If chroot.img is corrupt, the snapshot commands will raise exceptions when cros_sdk fails. Trap these and report that snapshot restoration failed so that the chroot can be deleted. BUG= chromium:843441 TEST=unit tests Change-Id: I02beed268d4582b3a679a7693717101a6e477b15 Reviewed-on: https://chromium-review.googlesource.com/1061592 Commit-Ready: Benjamin Gordon <bmgordon@chromium.org> Tested-by: Benjamin Gordon <bmgordon@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/516b94cffe0090291073fea0c6029233fcacf05a/cbuildbot/stages/build_stages.py [modify] https://crrev.com/516b94cffe0090291073fea0c6029233fcacf05a/cbuildbot/stages/build_stages_unittest.py
,
May 16 2018
,
May 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b6e9f9e54e907eb474b62cc926eb40744bb01ae5 commit b6e9f9e54e907eb474b62cc926eb40744bb01ae5 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Thu May 17 00:53:03 2018 Roll src/third_party/chromite/ 4c1562dfd..966fe6267 (6 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/4c1562dfdeef..966fe62677b1 $ git log 4c1562dfd..966fe6267 --date=short --no-merges --format='%ad %ae %s' 2018-05-14 ayatane sysmon: Add ignored metric to prod manifest 2018-05-16 bmgordon build_stages: Handle exceptions in snapshot commands 2018-05-16 manojgupta cros_fuzz_test_env: Fix cleanup option. 2018-04-27 dgarrett remote_try: Rename to request_build. 2018-04-27 dgarrett remote_try: Rearrange a little. 2018-04-26 dgarrett cros_tryjob: Move tryjob specific logic into cros tryjob. Created with: roll-dep src/third_party/chromite BUG=chromium:None,chromium:843441,chromium:843627,chromium:824550,chromium:824550,chromium:824550 The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: Ibd98b2152ec46624b2c80e06bb60514e17ca9e2a Reviewed-on: https://chromium-review.googlesource.com/1063030 Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#559372} [modify] https://crrev.com/b6e9f9e54e907eb474b62cc926eb40744bb01ae5/DEPS |
|||
►
Sign in to add a comment |
|||
Comment 1 by seanpaul@chromium.org
, May 16 2018