cros-beefy409-c2 slave fails to start |
||||
Issue description
[D2017-06-19T18:11:04.056538 1059 140492224227136 chromebuild-startup:181] Deleting /b/build...
[D2017-06-19T18:11:04.056662 1059 140492224227136 chromebuild-startup:181] Deleting /b/c...
[D2017-06-19T18:11:04.056800 1059 140492224227136 chromebuild-startup:181] Traceback (most recent call last):
[D2017-06-19T18:11:04.056959 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/run.py", line 29, in <module>
[D2017-06-19T18:11:04.057137 1059 140492224227136 chromebuild-startup:181] sys.exit(run_helper.run_py_main(sys.argv[1:], RUNPY_PATH, ENV_PATH, 'infra'))
[D2017-06-19T18:11:04.057295 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/bootstrap/run_helper.py", line 123, in run_py_main
[D2017-06-19T18:11:04.057438 1059 140492224227136 chromebuild-startup:181] runpy.run_module(args[0], run_name='__main__', alter_sys=True)
[D2017-06-19T18:11:04.057576 1059 140492224227136 chromebuild-startup:181] File "/usr/lib/python2.7/runpy.py", line 176, in run_module
[D2017-06-19T18:11:04.057697 1059 140492224227136 chromebuild-startup:181] fname, loader, pkg_name)
[D2017-06-19T18:11:04.057837 1059 140492224227136 chromebuild-startup:181] File "/usr/lib/python2.7/runpy.py", line 82, in _run_module_code
[D2017-06-19T18:11:04.058001 1059 140492224227136 chromebuild-startup:181] mod_name, mod_fname, mod_loader, pkg_name)
[D2017-06-19T18:11:04.058152 1059 140492224227136 chromebuild-startup:181] File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[D2017-06-19T18:11:04.058273 1059 140492224227136 chromebuild-startup:181] exec code in run_globals
[D2017-06-19T18:11:04.058486 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/.cipd/pkgs/infra_python_linux-amd64-ubuntu14_04_tap59FFGuW/6246e5df
5e5e39e9e03c12f7acce0a1dddded143/infra/tools/bot_setup/start/__main__.py", line 58, in <module>
[D2017-06-19T18:11:04.058616 1059 140492224227136 chromebuild-startup:181] sys.exit(main())
[D2017-06-19T18:11:04.058844 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/.cipd/pkgs/infra_python_linux-amd64-ubuntu14_04_tap59FFGuW/6246e5df
5e5e39e9e03c12f7acce0a1dddded143/infra/tools/bot_setup/start/__main__.py", line 54, in main
[D2017-06-19T18:11:04.059012 1059 140492224227136 chromebuild-startup:181] chrome.start(root_dir, depot_tools, password_file, slave_name)
[D2017-06-19T18:11:04.059227 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/.cipd/pkgs/infra_python_linux-amd64-ubuntu14_04_tap59FFGuW/6246e5df
5e5e39e9e03c12f7acce0a1dddded143/infra/tools/bot_setup/start/chrome.py", line 228, in start
[D2017-06-19T18:11:04.059365 1059 140492224227136 chromebuild-startup:181] ensure_checkout(root_dir, depot_tools, is_internal)
[D2017-06-19T18:11:04.059606 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/.cipd/pkgs/infra_python_linux-amd64-ubuntu14_04_tap59FFGuW/6246e5df
5e5e39e9e03c12f7acce0a1dddded143/infra/tools/bot_setup/start/chrome.py", line 131, in ensure_checkout
[D2017-06-19T18:11:04.059740 1059 140492224227136 chromebuild-startup:181] infra_libs.rmtree(full_path)
[D2017-06-19T18:11:04.059978 1059 140492224227136 chromebuild-startup:181] File "/opt/infra-bot-setup/infra-python/.cipd/pkgs/infra_python_linux-amd64-ubuntu14_04_tap59FFGuW/_current
/ENV/local/lib/python2.7/site-packages/infra_libs/utils.py", line 137, in rmtree
[D2017-06-19T18:11:04.060105 1059 140492224227136 chromebuild-startup:181] os.chmod(root, 0770)
[D2017-06-19T18:11:04.060257 1059 140492224227136 chromebuild-startup:181] OSError: [Errno 1] Operation not permitted: '/b/c/cbuild/repository/chroot/build/bin'
[E2017-06-19T18:11:04.070439 1059 140492224227136 chromebuild-startup:188] '/usr/bin/python' exited with code 1
[E2017-06-19T18:11:04.070661 1059 140492224227136 chromebuild-startup:84] Ran into exception /usr/bin/python run.py infra.tools.bot_setup.start --root_dir /b --password_file /home/chro
me-bot/.password_json --slave_name cros-beefy409-c2 failed with error 1
Traceback (most recent call last):
File "/opt/chromebuild/chromebuild-startup.py", line 82, in _wrapper
return fn(*args, **kwargs)
File "/opt/chromebuild/chromebuild-startup.py", line 284, in run_setup_cmd
call(cmd, cwd=cwd)
File "/opt/chromebuild/chromebuild-startup.py", line 192, in call
raise SubprocessFailed('%s failed with error %s' % (' '.join(args), code))
My current guess is that this may be related to canarying the new recipe launcher for cros slaves that dnj@ did last week.
If not, please re-assign to me.
,
Jun 19 2017
This looks related, not specifically to the new launcher, but rather to the new location for "cbuild", which moved from "/b/cbuild" to "/b/c/cbuild". It looks like the "chromebuild-startup" script is trying to nuke that directory and failing b/c of root-owned files. It wasn't doing this before (which was, itself an oversight!) +hinoka@ says that this sort of behavior only happens when the script things something has gone horribly wrong, so good news is that this shouldn't affect working bots. Proposed solution is to fix the script to properly handle nuking files that are owned by root.
,
Jun 19 2017
This is a CQ slave, so it is important. I've recovered other slaves that were also down (see issue 734638) so this isn't affecting the CQ right now. I'd say, keep it around till the end of the day if needed, then we'll recreate the instance to fill up the CQ slave pool again.
,
Jun 19 2017
This should be solved well before that, I'll keep you posted. Thanks!
,
Jun 20 2017
The underlying issue here should be fixed. If a buildslave exhibits this behavior, just reboot it and it will pick up the fix on reboot.
,
Jun 20 2017
,
Jun 20 2017
That buildslave is back without the need for a reboot. |
||||
►
Sign in to add a comment |
||||
Comment 1 by pprabhu@chromium.org
, Jun 19 2017