container won't start 64-bit container on aarch64 due to seccomp errors |
|||
Issue description
On Kevin with a container running with network set up
I mount /dev/vdb to /mnt/stateful, set the date, run lxd and then run lxd_setup.sh
I try to launch a container with:
lxc launch ubuntu:16.04 foobar
but get seccomp errors when trying to start the container:
Creating foobar
Starting foobar
EROR[04-12|00:07:15] Failed starting container action=start created=2018-04-12T00:07:08+0000 ephemeral=false name=foobar stateful=false used=1970-01-01T00:00:00+0000
error: Failed to run: /usr/sbin/lxd forkstart foobar /mnt/stateful/lxd/containers /mnt/stateful/lxd/logs/foobar/lxc.conf:
Try `lxc info --show-log local:foobar` for more info
(termina) localhost ~ # date
Thu Apr 12 00:07:20 UTC 2018
(termina) localhost ~ # lxc info --show-log local:foobar
Name: foobar
Remote: unix://
Architecture: aarch64
Created: 2018/04/12 00:07 UTC
Status: Stopped
Type: persistent
Profiles: default
Log:
lxc 20180412000715.442 ERROR lxc_seccomp - seccomp.c:get_new_ctx:249 - Seccomp error -17 (File exists) adding arch: 5
lxc 20180412000715.442 ERROR lxc_start - start.c:lxc_init:610 - Failed loading seccomp policy.
lxc 20180412000715.442 ERROR lxc_start - start.c:__lxc_start:1426 - Failed to initialize container "foobar".
lxc 20180412000715.443 ERROR lxc_container - lxccontainer.c:wait_on_daemonized_start:751 - No such file or directory - Failed to receive the container state
The guest kernel has CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER enabled
The current theory is that lxc is getting confused because we're running a 32-bit userspace on a 64-bit guest kernel and then trying to start a 64-bit container (ubuntu:16.04)
,
Apr 13 2018
re #1 - not sure -- I think that's what's available for Ubuntu ARM (server) but I need to double check to see if there's a 32-bit container available
,
Apr 13 2018
I can disable CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER in the tael kernel and then we get further. Apparently not having kernel support is not fatal for lxc.
,
Apr 13 2018
Can you show a grep SECCOMP of the kernel config, please?
,
Apr 13 2018
So this is a funny problem. When you run a 64bit kernel and a 32bit userspace what does uname -a show you. For you to be able to even get that far of creating the container it must report x86_64. Assume you now nest LXD inside As an fyi, if you want to venture into even weirder territory: Assume, you run a 64bit kernel with a 64bit userspace on the host and you start a *32bit container* via LXD, then LXC will by default use the personality() syscall and change the personality to 32bit at which point uname -a will report i686 instead of x86_64. Now further assume you nest LXD, i.e. you run LXD inside LXD which is perfectly possible. Now creating a 64bit container would fail since the utsname() syscall would report a 32bit kernel. :) Note, that it's still possible to tell the container to not change the architecture to 32bit by doing "lxc config edit" and then changing the "architecture" entry to "x86_64" or you use "lxc config set <container-name> raw.lxc 'lxc.arch = x86_64'". Then you could even manage that.
,
Apr 13 2018
Chirantan, Sonny, can you please take a look at and possibly try: https://github.com/lxc/lxc/pull/2274 This should unblock you and allow you to do what you want.
,
Apr 16 2018
re #6 -- thanks for that I'll give it a try
,
Apr 16 2018
There is some weirdness wrt to seccomp and stacking userspace architectures. I'm talking to Paul Moore - the libseccomp maintainer - about this. If you need anything feel free to reach out.
,
Apr 17 2018
re #8 -- it seems to work, thanks! I backported two patches and patched them into our version of lxc: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1014019
,
Apr 17 2018
Ok, good to hear.
,
Apr 17 2018
re #10 -- I might have spoken too soon. Those patches do allow us to load the seccomp policy and then try to start the container. I had to fix a couple other issues to get further, but once the container starts I'm seeing a seccomp kill that shoots down the init process almost immediately. The syscall in question doesn't seem to be one on the blacklist -- so maybe this is what you were referring to re wierdness?
,
Apr 17 2018
re #11 - The patch is still needed so if you can pick it up. It will allow you to run 32bit containers on 64bit kernel + 64bit userspace with seccomp correctly. Paul Moore and I currently believe that this is a seccomp or libseccomp bug that we're tracking down. If I know more I'll ping you.
,
Apr 17 2018
also to answer Chirantan's question -- there is a 32-bit ARM image that we can use. We need to specify it as "armhf" architecture. So you'd run lxc launch ubuntu:16.04/armhf foobar32
,
Apr 17 2018
Yes, usually for any distro we support we usually support all architectures they support.
,
Apr 17 2018
We discussed this during the meeting today and I think we want to stick to using a 32-bit ARM container.
,
Apr 17 2018
That's probably the proper solution for now. I still want to fix the inverted case though since this is bugging me. Paul pointed out that it may simply be that the 64bit ABI is not loaded into the kernel on an inverted arch. I'm going to test this and see if that is indeed true.
,
Apr 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/bca58f43214d5f0b42a106100afefceca893904a commit bca58f43214d5f0b42a106100afefceca893904a Author: Sonny Rao <sonnyrao@chromium.org> Date: Tue Apr 17 22:47:39 2018 app-emulation/lxc: add patches to fix seccomp arch inversion This applies upstream fixes to handle the case where we hae mixed 32/64-bit architectures like on ARM. BUG= chromium:832366 TEST=manual, launch container on arm Change-Id: I40995bacfcc00bd61a412784f5aaa7d888376363 Reviewed-on: https://chromium-review.googlesource.com/1014019 Commit-Ready: Sonny Rao <sonnyrao@chromium.org> Tested-by: Sonny Rao <sonnyrao@chromium.org> Reviewed-by: Chirantan Ekbote <chirantan@chromium.org> [add] https://crrev.com/bca58f43214d5f0b42a106100afefceca893904a/app-emulation/lxc/lxc-2.1.1-r1.ebuild [modify] https://crrev.com/bca58f43214d5f0b42a106100afefceca893904a/app-emulation/lxc/lxc-2.1.1.ebuild [add] https://crrev.com/bca58f43214d5f0b42a106100afefceca893904a/app-emulation/lxc/files/lxc-3.0.0-seccomp-handle-all-errors.patch [add] https://crrev.com/bca58f43214d5f0b42a106100afefceca893904a/app-emulation/lxc/files/lxc-3.0.0-seccomp-handle-arch-inversion.patch
,
Apr 18 2018
re #15 -- ok yeah that seems fine, I'll leave this open but downgrade priority
,
Apr 18 2018
Ok, I now have a tested fix for this: https://github.com/lxc/lxc/pull/2281 that you might want to backport once we have merged this. In essence: LXC generates and loads the seccomp-bpf filter in the host/container which spawns the new container. In other words, userspace N is responsible for generating and loading the seccomp-bpf filter which restricts userspace N + 1. Assume 64bit kernel and 32bit userspace running a 64bit container. In this case the 32-bit userspace is used to create a seccomp-bpf filter for a 64-bit userspace. Unless one explicitly adds the 64-bit ABI to the libseccomp filter, or adjusts the default behavior for "BAD_ARCH", *all* 64-bit syscalls will be blocked which is what you were seeing. I've added logic to check for all architectures that are compatible with each other.
,
Apr 18 2018
Ok, we just merged https://github.com/lxc/lxc/pull/2281 so you should backport and this will unblock you and enable you to run 64bit containers on 64bit kernel + 32bit userspace.
,
Apr 18 2018
So you want: https://github.com/lxc/lxc/commit/d648e178f1b3fa9f261b890157d2ee6e9e5e14fa https://github.com/lxc/lxc/commit/94d56054143a8634852989819acee06bf4aaf9f9 https://github.com/lxc/lxc/commit/7e84441ec3f973609bc2462528d55888ab1a084f https://github.com/lxc/lxc/commit/eca6736eb019f33a6243fc20a61c658da0662827 in this order with the last one being the actual fix.
,
Apr 20 2018
Fwiw, I have backported all of the seccomp arch inversion handling to LXC 3.0.0 as this is a bug. That should've always worked.
,
May 3 2018
re #22 -- thanks! I've applied those patches to our version of lxc and it is working. CL posted here: https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/1041205
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a9536fb2399d04892b6992b5a44212b4e8ea9082 commit a9536fb2399d04892b6992b5a44212b4e8ea9082 Author: Sonny Rao <sonnyrao@chromium.org> Date: Fri May 04 01:32:15 2018 app-emulation/lxc: more patches for to fix seccomp for mixed architecture The first set of backported patches allowed the policy to load but the container would quickly run into a seccomp violation. Upstream has posted a more complete solution to their 3.0 branch which I apply here. BUG= chromium:832366 TEST=manual, run 64-bit container on tael Change-Id: Icbb98147e65f3567f8ab63b281100f3f63d6621e Reviewed-on: https://chromium-review.googlesource.com/1041205 Commit-Ready: Sonny Rao <sonnyrao@chromium.org> Tested-by: Sonny Rao <sonnyrao@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> [add] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-use-return-instead-of-attribution-and-return.patch [add] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-non-functional-changes.patch [modify] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-handle-all-errors.patch [modify] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/lxc-2.1.1.ebuild [add] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-cleanup-architecture-handling.patch [add] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-filter-syscalls-based-on-arguments.patch [rename] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/lxc-2.1.1-r2.ebuild [add] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-handle-arch-inversion-ii.patch [add] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-improve-logging.patch [modify] https://crrev.com/a9536fb2399d04892b6992b5a44212b4e8ea9082/app-emulation/lxc/files/lxc-3.0.0-seccomp-handle-arch-inversion.patch
,
May 4 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by chirantan@chromium.org
, Apr 13 2018