New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 843441 link

Starred by 0 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

CQ chroot cleanup failed

Project Member Reported by vapier@chromium.org, May 16 2018

Issue description

https://luci-milo.appspot.com/buildbot/chromeos/nyan-full-compile-paladin/12641

************************************************************
** Start Stage CleanUp - Tue, 15 May 2018 09:33:02 -0700 (PDT)
** 
** Stages that cleans up build artifacts from previous runs.
** 
**   This stage cleans up previous KVM state, temporary git commits,
**   clobbers, and wipes tmp inside the chroot.
************************************************************
09:33:02: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f9d4fe2ff90>
Preconditions for the stage successfully met. Beginning to execute stage...
09:33:02: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f9d4fe2f650>
09:33:02: INFO: Refreshing access_token
09:33:03: INFO: Previous local build buildbucket_id=8946446495671154720 finished in state fail.
09:33:03: INFO: Previous local build 12640 did not pass. Cannot reuse chroot.
09:33:03: INFO: Attempting to revert chroot to clean-chroot snapshot
09:33:03: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk --snapshot-list in /b/c/cbuild/repository
09:33:04: WARNING: Failed to activate VG on try 1.
09:33:08: WARNING: Failed to activate VG on try 2.
09:33:08: WARNING: Failed to activate VG on try 3.
cros_sdk: Unhandled exception:
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/bin/cros_sdk", line 169, in <module>
    DoMain()
  File "/b/c/cbuild/repository/chromite/bin/cros_sdk", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/c/cbuild/repository/chromite/scripts/cros_sdk.py", line 948, in main
    cros_build_lib.MountChroot(options.chroot, create=False)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 1622, in MountChroot
    SudoRunCommand(cmd, capture_output=True, print_cmd=False)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 294, in SudoRunCommand
    return RunCommand(cmd, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 658, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: vgchange -q -ay cros_b+c+cbuild+repository+chroot_000
  Check of thin pool cros_b+c+cbuild+repository+chroot_000/thinpool failed (status:1). Manual repair required (thin_dump --repair /dev/mapper/cros_b+c+cbuild+repository+chroot_000-thinpool_tmeta)!
  0 logical volume(s) in volume group "cros_b+c+cbuild+repository+chroot_000" now active
cmd=['vgchange', '-q', '-ay', 'cros_b+c+cbuild+repository+chroot_000']
09:33:09: ERROR: 
return code: 1; command: /b/c/cbuild/repository/chromite/bin/cros_sdk --snapshot-list
cmd=['/b/c/cbuild/repository/chromite/bin/cros_sdk', '--snapshot-list'], cwd=/b/c/cbuild/repository
09:33:09: ERROR: <class 'chromite.lib.failures_lib.BuildScriptFailure'>: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1)
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 353, in PerformStage
    delete_chroot = not self._RevertChrootToCleanSnapshot()
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 68, in _RevertChrootToCleanSnapshot
    snapshots = commands.ListChrootSnapshots(self._build_root)
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 196, in ListChrootSnapshots
    redirect_stdout=True)
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript
    raise failures_lib.BuildScriptFailure(ex, cmd[0])
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript
    raise failures_lib.BuildScriptFailure(ex, cmd[0])
BuildScriptFailure: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1)

09:33:09: INFO: Translating result <class 'chromite.lib.failures_lib.BuildScriptFailure'>: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1)
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 353, in PerformStage
    delete_chroot = not self._RevertChrootToCleanSnapshot()
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 68, in _RevertChrootToCleanSnapshot
    snapshots = commands.ListChrootSnapshots(self._build_root)
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 196, in ListChrootSnapshots
    redirect_stdout=True)
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript
    raise failures_lib.BuildScriptFailure(ex, cmd[0])
  File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 136, in RunBuildScript
    raise failures_lib.BuildScriptFailure(ex, cmd[0])
BuildScriptFailure: /b/c/cbuild/repository/chromite/bin/cros_sdk failed (code=1)
 to fail.
09:33:09: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f9d4fd83710>
09:33:09: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f9d4fd83a90>
09:33:09: INFO: Running cidb query on pid 32616, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f9d4fd83b10>
************************************************************
** Finished Stage CleanUp - Tue, 15 May 2018 09:33:09 -0700 (PDT)
************************************************************
 
Status: Started (was: Available)
I cleaned up that particular builder and confirmed that subsequent builds are green.  I'll send out a CL to make the CleanUp stage handle it by itself.
Project Member

Comment 3 by bugdroid1@chromium.org, May 16 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/516b94cffe0090291073fea0c6029233fcacf05a

commit 516b94cffe0090291073fea0c6029233fcacf05a
Author: Benjamin Gordon <bmgordon@chromium.org>
Date: Wed May 16 23:42:59 2018

build_stages: Handle exceptions in snapshot commands

If chroot.img is corrupt, the snapshot commands will raise exceptions
when cros_sdk fails.  Trap these and report that snapshot restoration
failed so that the chroot can be deleted.

BUG= chromium:843441 
TEST=unit tests

Change-Id: I02beed268d4582b3a679a7693717101a6e477b15
Reviewed-on: https://chromium-review.googlesource.com/1061592
Commit-Ready: Benjamin Gordon <bmgordon@chromium.org>
Tested-by: Benjamin Gordon <bmgordon@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/516b94cffe0090291073fea0c6029233fcacf05a/cbuildbot/stages/build_stages.py
[modify] https://crrev.com/516b94cffe0090291073fea0c6029233fcacf05a/cbuildbot/stages/build_stages_unittest.py

Status: Fixed (was: Started)
Project Member

Comment 5 by bugdroid1@chromium.org, May 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b6e9f9e54e907eb474b62cc926eb40744bb01ae5

commit b6e9f9e54e907eb474b62cc926eb40744bb01ae5
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Thu May 17 00:53:03 2018

Roll src/third_party/chromite/ 4c1562dfd..966fe6267 (6 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/4c1562dfdeef..966fe62677b1

$ git log 4c1562dfd..966fe6267 --date=short --no-merges --format='%ad %ae %s'
2018-05-14 ayatane sysmon: Add ignored metric to prod manifest
2018-05-16 bmgordon build_stages: Handle exceptions in snapshot commands
2018-05-16 manojgupta cros_fuzz_test_env: Fix cleanup option.
2018-04-27 dgarrett remote_try: Rename to request_build.
2018-04-27 dgarrett remote_try: Rearrange a little.
2018-04-26 dgarrett cros_tryjob: Move tryjob specific logic into cros tryjob.

Created with:
  roll-dep src/third_party/chromite
BUG=chromium:None,chromium:843441,chromium:843627,chromium:824550,chromium:824550,chromium:824550


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ibd98b2152ec46624b2c80e06bb60514e17ca9e2a
Reviewed-on: https://chromium-review.googlesource.com/1063030
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#559372}
[modify] https://crrev.com/b6e9f9e54e907eb474b62cc926eb40744bb01ae5/DEPS

Sign in to add a comment