Cros builders not always rebooting between builds. |
||||
Issue descriptionChromeOS builders can't always clean up loopback devices after themselves, because of umount kernel bugs. So, we are supposed to reboot between builds to ensure things are clean for the next build. That does not always appear to be happening, which can kill builds at startup in confusing ways. https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8932745453427979040 cbuildbot_launch: Unhandled exception: Traceback (most recent call last): File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 169, in <module> DoMain() File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/commandline.py", line 912, in ScriptWrapperMain ret = target(argv[1:]) File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 497, in main return _main(argv) File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 464, in _main CleanBuildRoot(root, repo, metrics_fields, build_state) File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 72, in wrapped_functor result = functor(*args, **kwargs) File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 260, in CleanBuildRoot osutils.RmDir(root, ignore_missing=True, sudo=True) File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/osutils.py", line 394, in RmDir redirect_stdout=True, redirect_stderr=True) File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/cros_build_lib.py", line 322, in SudoRunCommand return RunCommand(sudo_cmd, **kwargs) File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/cros_build_lib.py", line 647, in RunCommand raise RunCommandError(msg, cmd_result) chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: sudo -- rm -rf -- /b/swarming/w/ir/cache/cbuild rm: cannot remove ‘/b/swarming/w/ir/cache/cbuild/repository/new-sdk-chroot’: Device or resource busy cmd=['sudo', '--', 'rm', '-rf', '--', '/b/swarming/w/ir/cache/cbuild']
,
Oct 16
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/1020cffb25c8418629f7e2cb75264606cf7a7dc4 commit 1020cffb25c8418629f7e2cb75264606cf7a7dc4 Author: Don Garrett <dgarrett@google.com> Date: Tue Oct 16 20:36:36 2018
,
Oct 16
,
Oct 22
I've got an example that confirms this isn't fixed.
,
Oct 22
This build: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931960173807897536 On this bot: https://chrome-swarming.appspot.com/bot?id=swarm-cros-413&sort_stats=total%3Adesc Ran this task: https://chrome-swarming.appspot.com/task?id=40b4d8bca55c0810&refresh=10 Looking at GCE serial console, there WERE reboots. I thought there weren't because of a mix of multiple timezones in different logs.
,
Oct 22
,
Oct 22
I knew there was something in the web UI that did that, but didn't find it this morning (probably not enough coffee). Thanks! |
||||
►
Sign in to add a comment |
||||
Comment 1 by dgarr...@chromium.org
, Oct 16