New issue
Advanced search Search tips

Issue 895955 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 22
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Cros builders not always rebooting between builds.

Project Member Reported by dgarr...@chromium.org, Oct 16

Issue description

ChromeOS builders can't always clean up loopback devices after themselves, because of umount kernel bugs. So, we are supposed to reboot between builds to ensure things are clean for the next build.

That does not always appear to be happening, which can kill builds at startup in confusing ways.


https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8932745453427979040

cbuildbot_launch: Unhandled exception:
Traceback (most recent call last):
  File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 169, in <module>
    DoMain()
  File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 497, in main
    return _main(argv)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 464, in _main
    CleanBuildRoot(root, repo, metrics_fields, build_state)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 72, in wrapped_functor
    result = functor(*args, **kwargs)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 260, in CleanBuildRoot
    osutils.RmDir(root, ignore_missing=True, sudo=True)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/osutils.py", line 394, in RmDir
    redirect_stdout=True, redirect_stderr=True)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/cros_build_lib.py", line 322, in SudoRunCommand
    return RunCommand(sudo_cmd, **kwargs)
  File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: sudo -- rm -rf -- /b/swarming/w/ir/cache/cbuild
rm: cannot remove &#8216;/b/swarming/w/ir/cache/cbuild/repository/new-sdk-chroot&#8217;: Device or resource busy
cmd=['sudo', '--', 'rm', '-rf', '--', '/b/swarming/w/ir/cache/cbuild']
 
With maruel@'s help, I'm preparing a tweak to the bot configs to force reboots between builds.
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 16

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1020cffb25c8418629f7e2cb75264606cf7a7dc4

commit 1020cffb25c8418629f7e2cb75264606cf7a7dc4
Author: Don Garrett <dgarrett@google.com>
Date: Tue Oct 16 20:36:36 2018

Status: Fixed (was: Started)
Status: Started (was: Fixed)
I've got an example that confirms this isn't fixed.
Status: Fixed (was: Started)
This build:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931960173807897536

On this bot:

https://chrome-swarming.appspot.com/bot?id=swarm-cros-413&sort_stats=total%3Adesc

Ran this task:

https://chrome-swarming.appspot.com/task?id=40b4d8bca55c0810&refresh=10

Looking at GCE serial console, there WERE reboots. I thought there weren't because of a mix of multiple timezones in different logs.


I knew there was something in the web UI that did that, but didn't find it this morning (probably not enough coffee).

Thanks!

Sign in to add a comment