New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 711578 link

Starred by 1 user
Status: Untriaged
Owner: ----
Cc:
█ jrbarnette@chromium.org
EstimatedDays: ----
NextAction: 2019-07-09
OS: ----
Pri: 3
Type: Bug

Blocked on:
issue 710552


Sign in to add a comment

CQ master hung and sync stage failed at clearing chroot folders

Project Member Reported by nxia@chromium.org, Apr 14 2017 
https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fmaster-paladin%2F14291%2F%2B%2Frecipes%2Fsteps%2FCommitQueueSync%2F0%2Fstdout


RELEASETAG: 9459.0.0-rc5

19:04:45: INFO: LKGM version was found in the manifest: 9459.0.0-rc2

@@@STEP_TEXT@(Using fresh chroot)@@@

@@@STEP_FAILURE@@@
19:04:45: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: sudo -n 'CROS_CACHEDIR=/b/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- rm -rf -- /b/cbuild/repository/chroot
rm: cannot remove &#8216;/b/cbuild/repository/chroot/mnt/host/depot_tools&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/mnt/host/source&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/dev&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/home/chrome-bot/.cidb_creds&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/var/cache/chromeos-cache&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/sys&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/run&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/proc&#8217;: Device or resource busy

cmd=['sudo', '-n', 'CROS_CACHEDIR=/b/cbuild/repository/.cache', 'CROS_SUDO_KEEP_ALIVE=unknown', '--', 'rm', '-rf', '--', '/b/cbuild/repository/chroot']
Traceback (most recent call last):
  File "/b/cbuild/repository/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 767, in PerformStage
    self.ManifestCheckout(new_manifest)
  File "/b/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 1083, in ManifestCheckout
    chroot_manager.EnsureChrootAtVersion(lkgm_version)
  File "/b/cbuild/repository/chromite/cbuildbot/chroot_lib.py", line 75, in EnsureChrootAtVersion
    osutils.RmDir(chroot, ignore_missing=True, sudo=True)
  File "/b/cbuild/repository/chromite/lib/osutils.py", line 299, in RmDir
    redirect_stdout=True, redirect_stderr=True)
  File "/b/cbuild/repository/chromite/lib/cros_build_lib.py", line 300, in SudoRunCommand
    return RunCommand(sudo_cmd, **kwargs)
  File "/b/cbuild/repository/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: sudo -n 'CROS_CACHEDIR=/b/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- rm -rf -- /b/cbuild/repository/chroot
rm: cannot remove &#8216;/b/cbuild/repository/chroot/mnt/host/depot_tools&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/mnt/host/source&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/dev&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/home/chrome-bot/.cidb_creds&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/var/cache/chromeos-cache&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/sys&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/run&#8217;: Device or resource busy
rm: cannot remove &#8216;/b/cbuild/repository/chroot/proc&#8217;: Device or resource busy

cmd=['sudo', '-n', 'CROS_CACHEDIR=/b/cbuild/repository/.cache', 'CROS_SUDO_KEEP_ALIVE=unknown', '--', 'rm', '-rf', '--', '/b/cbuild/repository/chroot']

Comment 1 by nxia@chromium.org, Apr 14 2017 

chrome-bot@cros-wimpy0-c2:(Linux 14.04):/b/cbuild/repository/chroot$ lsof +D /b/cbuild/repository/chroot/
COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
python2    6136 chrome-bot  rtd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot
bash      15270 chrome-bot  cwd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot
lsof      15938 chrome-bot  cwd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot
lsof      15939 chrome-bot  cwd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot
setup_boa 18988 chrome-bot  rtd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot
cros_choo 22886 chrome-bot  rtd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot
cros_choo 23014 chrome-bot  rtd    DIR    8,1     4096 8915362 /b/cbuild/repository/chroot

Comment 2 by nxia@chromium.org, Apr 14 2017 

[1] chrome-bot@cros-wimpy0-c2:(Linux 14.04):/b/cbuild/repository/chroot$ ps aux  | grep cbuildbot 
root      6127  0.0  0.1 116808 44508 ?        S    18:46   0:00 python2 /b/cbuild/repository/chromite/bin/cros_sdk PARALLEL_EMERGE_STATUS_FILE=/tmp/tmp38n73X -- /mnt/host/source/chromite/cbuildbot/binhost_test --log-level=debug
root      6132  0.0  0.1 116808 44672 ?        S    18:46   0:00 python2 /b/cbuild/repository/chromite/bin/cros_sdk PARALLEL_EMERGE_STATUS_FILE=/tmp/tmp38n73X -- /mnt/host/source/chromite/cbuildbot/binhost_test --log-level=debug
chrome-+  6136  0.0  0.1 169180 57008 ?        S    18:46   0:00 python2 /mnt/host/source/chromite/cbuildbot/binhost_test --log-level=debug

Comment 3 by nxia@chromium.org, Apr 14 2017 

It could be https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/14289 was canceled by the process didn't get killed. This needs to be fixed. now I'm going to reboot the machine manually.


[1] chrome-bot@cros-wimpy0-c2:(Linux 14.04):/b/cbuild/repository/chroot$ ps aux  | grep 14289
chrome-+ 16014  0.0  0.0  13440   904 pts/1    S+   21:33   0:00 grep --color=auto 14289
chrome-+ 18807  0.0  0.1 127768 58156 ?        S    18:30   0:00 python2 /b/build/slave/master-paladin-master/build
/chromite/scripts/cbuildbot_launch --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 
--git-cache-dir /b/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_acc
ounts/service-account-goma-client.json master-paladin
chrome-+ 29426  0.0  0.3 270432 104952 ?       S    18:31   0:09 python2 /b/cbuild/repository/chromite/bin/cbuildbo
t master-paladin --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 --git-cache-dir /b
/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-acco
unt-goma-client.json
chrome-+ 29445  0.0  0.1 137992 52492 ?        Ss   18:31   0:00 python2 /b/cbuild/repository/chromite/bin/cbuildbo
t master-paladin --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 --git-cache-dir /b
/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-acco
unt-goma-client.json
chrome-+ 29447  0.0  0.1 591176 56924 ?        Sl   18:31   0:00 python2 /b/cbuild/repository/chromite/bin/cbuildbo
t master-paladin --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 --git-cache-dir /b
/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-acco
unt-goma-client.json
chrome-+ 29451  0.0  0.1 227440 61608 ?        S    18:31   0:00 python2 /b/cbuild/repository/chromite/bin/cbuildbo
t master-paladin --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 --git-cache-dir /b
/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-acco
unt-goma-client.json
chrome-+ 29453  0.0  0.1 443712 56908 ?        Sl   18:31   0:00 python2 /b/cbuild/repository/chromite/bin/cbuildbo
t master-paladin --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 --git-cache-dir /b
/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-acco
unt-goma-client.json
chrome-+ 29460  0.0  0.1 845380 59408 ?        Sl   18:31   0:02 python2 /b/cbuild/repository/chromite/bin/cbuildbo
t master-paladin --buildroot /b/cbuild/repository --buildbot --branch master --buildnumber 14289 --git-cache-dir /b
/cros_git_cache --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-acco
unt-goma-client.json

Comment 4 by nxia@chromium.org, Apr 14 2017

Blockedon: 710552
Reboot fixed the issue temporarily

Comment 5 by nxia@chromium.org, Jun 8 2018

Cc: -nxia@chromium.org

Comment 6 by benhenry@google.com, Jan 10

Labels: Pri-3
NextAction: 2019-07-09
Downgrading P2s that haven't been modified in more than 6 months, which have no component or owner.
►

Sign in to add a comment