New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 816986 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

reef canary won't get past CleanUp stage after build crash

Project Member Reported by ejcaruso@chromium.org, Feb 27 2018

Issue description

reef-release was passing on this build: https://luci-milo.appspot.com/buildbot/chromeos/reef-release/1972

It looks like the build crashed during the UnitTest phase of https://luci-milo.appspot.com/buildbot/chromeos/reef-release/1973 and now every build since then has failed with the following error:

umount: /b/c/cbuild/repository/chroot: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
18:10:31: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: sudo -n 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- umount -d /b/c/cbuild/repository/chroot
cmd=['sudo', '-n', 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache', 'CROS_SUDO_KEEP_ALIVE=unknown', '--', 'umount', '-d', '/b/c/cbuild/repository/chroot']
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 190, in PerformStage
    cros_build_lib.CleanupChrootMount(buildroot=self._build_root)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 1730, in CleanupChrootMount
    osutils.UmountTree(chroot)
  File "/b/c/cbuild/repository/chromite/lib/osutils.py", line 866, in UmountTree
    UmountDir(mount_pt, lazy=False, cleanup=False)
  File "/b/c/cbuild/repository/chromite/lib/osutils.py", line 828, in UmountDir
    runcmd(cmd, print_cmd=False)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 333, in SudoRunCommand
    return RunCommand(sudo_cmd, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 658, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: sudo -n 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- umount -d /b/c/cbuild/repository/chroot
cmd=['sudo', '-n', 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache', 'CROS_SUDO_KEEP_ALIVE=unknown', '--', 'umount', '-d', '/b/c/cbuild/repository/chroot']

18:10:31: INFO: Translating result <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: sudo -n 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- umount -d /b/c/cbuild/repository/chroot
cmd=['sudo', '-n', 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache', 'CROS_SUDO_KEEP_ALIVE=unknown', '--', 'umount', '-d', '/b/c/cbuild/repository/chroot']
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/build_stages.py", line 190, in PerformStage
    cros_build_lib.CleanupChrootMount(buildroot=self._build_root)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 1730, in CleanupChrootMount
    osutils.UmountTree(chroot)
  File "/b/c/cbuild/repository/chromite/lib/osutils.py", line 866, in UmountTree
    UmountDir(mount_pt, lazy=False, cleanup=False)
  File "/b/c/cbuild/repository/chromite/lib/osutils.py", line 828, in UmountDir
    runcmd(cmd, print_cmd=False)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 333, in SudoRunCommand
    return RunCommand(sudo_cmd, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 658, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: sudo -n 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- umount -d /b/c/cbuild/repository/chroot
cmd=['sudo', '-n', 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache', 'CROS_SUDO_KEEP_ALIVE=unknown', '--', 'umount', '-d', '/b/c/cbuild/repository/chroot']
 to fail.

Assigning to deputy.
 

Comment 1 by nxia@chromium.org, Mar 3 2018

I forced a clobber run on reef-release

Comment 2 by nxia@chromium.org, Mar 3 2018

Status: Fixed (was: Assigned)
the latest reef-release passed the CleanUp stage.

Sign in to add a comment