New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 919889 link

Starred by 1 user
Status: Duplicate
Owner:
mikenichols@chromium.org
Closed: Jan 8
Cc:
█ martinroth@chromium.org
chromeos-continuous-integration-fire-hose@google.com
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: ----
Type: ----



Sign in to add a comment

chroot cleanup failure.

Project Member Reported by npoojary@chromium.org, Jan 8 
tael-paladin failed with childbot_launch failure

  /b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch --buildroot /b/swarming/w8SRsOa/ir/cache/cbuild --branch master --buildbucket-id 8924893985518862784 --master-build-id 3328122 --git-cache-dir /b/swarming/w8SRsOa/ir/cache/git --goma_dir /b/swarming/w8SRsOa/ir/cache/goma/client --goma_client_json /creds/service_accounts/service-account-goma-client.json --buildbot tael-paladin
  in dir /b/swarming/w8SRsOa/ir/kitchen-workdir:
   allow_subannotations: True
   base_name: 'cbuildbot_launch [tael-paladin]'
   cmd: ['/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch',
   '--buildroot',
   '/b/swarming/w8SRsOa/ir/cache/cbuild',
   '--branch',
   'master',
   '--buildbucket-id',
   '8924893985518862784',
   '--master-build-id',
   '3328122',
   '--git-cache-dir',
   '/b/swarming/w8SRsOa/ir/cache/git',
   '--goma_dir',
   '/b/swarming/w8SRsOa/ir/cache/goma/client',
   '--goma_client_json',
   '/creds/service_accounts/service-account-goma-client.json',
   '--buildbot',
   'tael-paladin']
   env: {'DEPOT_TOOLS_UPDATE': '0'}
   env_prefixes: {'PATH': '/b/swarming/w8SRsOa/ir/kitchen-workdir/python_bin'}
   env_suffixes: {'PATH': '/b/swarming/w8SRsOa/ir/kitchen-workdir/depot_tools'}
   infra_step: False
   name: 'cbuildbot_launch [tael-paladin]'
   nest_level: 0
   ok_ret: frozenset([0])
   trigger_specs: ()
  full environment:
   ADDRFAM: inet
   BUILDBUCKET_EXPERIMENTAL: FALSE
   CIPD_CACHE_DIR: /b/swarming/cipd_cache/cache
   CIPD_PROTOCOL: v2
   DEPOT_TOOLS_UPDATE: 0
   DOCKER_CONFIG: /b/swarming/w8SRsOa/ir/tmp/docker_cfg_task
   DOCKER_TMPDIR: /b/swarming/w8SRsOa/ir/tmp/docker_tmp_task
   HOME: /home/chrome-bot
   IFACE: eth0
   LOGDOG_COORDINATOR_HOST: logs.chromium.org
   LOGDOG_STREAM_PREFIX: buildbucket/cr-buildbucket.appspot.com/8924893985518862784
   LOGDOG_STREAM_PROJECT: chromeos
   LOGDOG_STREAM_SERVER_PATH: unix:/b/swarming/w8SRsOa/ir/tmp/ld.sock
   LOGICAL: eth0
   LUCI_CONTEXT: /b/swarming/w8SRsOa/itrSqraF/luci_context.284132199
   MAC_CHROMIUM_TMPDIR: /b/swarming/w8SRsOa/ir/tmp/t
   METHOD: dhcp
   NO_GCE_CHECK: False
   PATH: /b/swarming/w8SRsOa/ir/kitchen-workdir/python_bin:/b/swarming/w8SRsOa/ir/cipd_bin_packages:/b/swarming/w8SRsOa/ir/cipd_bin_packages/bin:/b/swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/b/swarming/w8SRsOa/ir/kitchen-workdir/depot_tools
   PWD: /
   PYTHONIOENCODING: UTF-8
   PYTHONUNBUFFERED: 1
   ROOT_SETUP_LOG: /var/log/messages/chromebuild/root-setup.log
   STARTUP_LOG: /var/log/messages/chromebuild/startup.log
   SWARMING_BOT_ID: swarm-cros-431
   SWARMING_EXTERNAL_BOT_SETUP: 1
   SWARMING_HEADLESS: 1
   SWARMING_SERVER: https://chrome-swarming.appspot.com
   SWARMING_TASK_ID: 4246834f1a087e11
   TEMP: /b/swarming/w8SRsOa/ir/tmp/t
   TEMPDIR: /b/swarming/w8SRsOa/ir/tmp/t
   TERM: linux
   TMP: /b/swarming/w8SRsOa/ir/tmp/t
   TMPDIR: /b/swarming/w8SRsOa/ir/tmp/t
   UPSTART_EVENTS: net-device-up
   UPSTART_INSTANCE: 
   UPSTART_JOB: chromebuild-startup
   USER: chrome-bot
   USERNAME: chrome-bot
   VPYTHON_VIRTUALENV_ROOT: /b/swarming/w8SRsOa/ir/cache/vpython
  05:47:39: INFO: Starting new HTTP connection (1): metadata.google.internal
  step returned non-zero exit code: 1

Comment 1 by ayatane@chromium.org, Jan 8

Components: -Infra>Client>ChromeOS>Test>LabAutomation Infra>Client>ChromeOS>CI
Owner: ----

Comment 2 by dgarr...@chromium.org, Jan 8

Owner: mikenichols@chromium.org
Summary: chroot cleanup failure. (was: tael-paladin failed with childbot_launch failure)
It would be really useful to have a link to the exact build that failed.

I'm going to guess you meant this one:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924893985518862784

In which case, there are multiple stages that failed. The important one was CleanupChroot.

This is a chroot umount failure:


06:38:09: INFO: Cleaning up chroot at /b/swarming/w8SRsOa/ir/cache/cbuild/repository/chroot
umount: /b/swarming/w8SRsOa/ir/cache/cbuild/repository/chroot: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

06:38:09: INFO: Waiting for ts_mon flushing process to finish...
06:39:09: INFO: RetriableHttp: attempt 1 receiving status 503, will retry
06:39:10: INFO: Finished waiting for ts_mon process.
cbuildbot_launch: Unhandled exception:
Traceback (most recent call last):
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 169, in <module>
    DoMain()
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 498, in main
    return _main(options, argv)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 481, in _main
    CleanupChroot(buildroot)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 72, in wrapped_functor
    result = functor(*args, **kwargs)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 399, in CleanupChroot
    cros_sdk_lib.CleanupChrootMount(chroot_dir, delete=False)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/timeout_util.py", line 191, in TimeoutWrapper
    return func(*args, **kwargs)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/cros_sdk_lib.py", line 394, in CleanupChrootMount
    osutils.UmountTree(chroot)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/osutils.py", line 901, in UmountTree
    UmountDir(mount_pt, lazy=False, cleanup=False)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/osutils.py", line 863, in UmountDir
    runcmd(cmd, print_cmd=False)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/cros_build_lib.py", line 321, in SudoRunCommand
    return RunCommand(sudo_cmd, **kwargs)
  File "/b/swarming/w8SRsOa/ir/kitchen-workdir/chromite/lib/cros_build_lib.py", line 646, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: sudo -- umount -d /b/swarming/w8SRsOa/ir/cache/cbuild/repository/chroot
cmd=['sudo', '--', 'umount', '-d', '/b/swarming/w8SRsOa/ir/cache/cbuild/repository/chroot']

Comment 3 by dgarr...@chromium.org, Jan 8 

I would classify this as build flake, and unless we have a sudden uptake in frequency, blow it off until after the builder kernel upgrade is finished, since we hope that will fix / reduce this type of flake.

Comment 4 by jclinton@chromium.org, Jan 8

Mergedinto: 902482
Status: Duplicate (was: Untriaged)
Guessing this will be fixed by the Xenial upgrade.
►

Sign in to add a comment