New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 860565 link

Starred by 100 users

Issue metadata

Status: Fixed
Owner:
Closed: Oct 31
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug-Regression

Restricted
  • Only users with EditIssue permission may comment.



Sign in to add a comment

Crostini Containers Unable to Run Docker (kernel keyring support)

Reported by david.dy...@gmail.com, Jul 5

Issue description

UserAgent: Mozilla/5.0 (X11; CrOS x86_64 10820.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3473.0 Safari/537.36
Platform: 10820.0.0 (Official Build) dev-channel eve

Steps to reproduce the problem:
1. Enable Linux/Crostini
2. Install Docker (wget https://get.docker.com|bash)
3. Try to run a docker container (docker run --rm -ti busybox)

What is the expected behavior?
The container should run as expected

What went wrong?
Docker complains that required keyring functionality is not present in the kernel:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"could not create session key: function not implemented\"": unknown

Did this work before? N/A 

Chrome version: 69.0.3473.0  Channel: dev
OS Version: 10820.0.0
Flash Version: 30.0.0.127

This used to work in earlier kernel versions (4.14.37) but now doesn't
 
Components: OS>Systems>Containers
Cc: smbar...@chromium.org
Labels: Proj-Containers
The kernel config hasn't been changed recently.

0b4c318f5a4b4a (Stephen Barber     2017-02-06 18:06:44 -0800 2960) CONFIG_KEYS=y
5d0d3cb4375d64 (Dylan Reid         2017-11-20 11:47:19 -0800 2961) CONFIG_KEYS_COMPAT=y

Is your docker runtime 32-bit or 64-bit? Which cros-termina component version are you using (under chrome://components)?

If you have any additional security stuff turned on, particularly with seccomp policies, that would explain the ENOSYS. But your kernel should support it. You can check by installing the keyutils package and running `keyctl session`.
Cc: jkwang@chromium.org
The kernel keyring is not namespaced so it is absolutely not safe to use it inside containers and won't work correctly. Since LXD on ChromeOS is only running unprivileged containers it will also not be possible to use the keyring API in any meaningful anyway. We always suggest to set:

security.syscalls.blacklist: keyctl errno 38

either per container or in the default profile all containers belong to. This will ensure that e.g. systemd services will function correctly in unprivileged containers.
This could easily be a either a regression in runC (The underlying runtime for Docker.) or the runC version is too old. If they muck with the kernel keyring they should definitely gracefully move on when ENOSYS is returned.
Owner: smbar...@chromium.org
Status: Assigned (was: Unconfirmed)
I have had no issues running docker on the pixelbook dev channel. I installed through apt. Perhaps it is a configuration issue with your docker installation?
Project Member

Comment 7 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tremplin/+/c482ee9c54e32b02615e2cd42b27b3029a3fdabd

commit c482ee9c54e32b02615e2cd42b27b3029a3fdabd
Author: Stephen Barber <smbarber@chromium.org>
Date: Fri Jul 20 03:12:50 2018

tremplin: blacklist keyctl syscall in default profile

BUG= chromium:860565 
TEST=keyctl syscall fails with ENOSYS

Change-Id: I6861de1638834b8529581a37c0c294880c003b1a
Reviewed-on: https://chromium-review.googlesource.com/1144042
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Jingkui Wang <jkwang@google.com>

[modify] https://crrev.com/c482ee9c54e32b02615e2cd42b27b3029a3fdabd/src/chromiumos/tremplin/main.go

Status: Fixed (was: Assigned)
Thanks Christian! Should be blacklisted in the next update or so.
I'm on 10895.21.0 (Official Build) beta-channel fizz-unibuild (fizz kench sion teemo wukong), which is an Acer Chromebox CXI3 i5, running 4.14.56-05965-ge7d0ede66eb4 kernel.

Is it expected that adding this new security.syscalls.blacklist entry[1] will prevent Docker containers from running?  I get the error in the original message above ("could not create session key: function not implemented") and the only way I can get around it is by _unsetting_ that profile key every time I reboot the hardware, and then reapplying the profile to penguin.

```
(termina) chronos@localhost ~ $ lxc profile unset default security.syscalls.blacklist
(termina) chronos@localhost ~ $ lxc profile apply penguin default
Profiles default applied to penguin
```

If I do not do this, my Docker containers (even hello-world) will not function.  After unsetting, they function exactly as expected.

On my Pixelbook i5 running the exact same kernel but a slightly older OS build (10895.10.0 (Official Build) dev-channel eve), all Docker container use works fine, but it doesn't have that blacklist entry in the lxc profile.

If I've made a mistake by misunderstanding what was or was not fixed in this bug, please let me know.

1: https://chromium.googlesource.com/chromiumos/platform/tremplin/+/c482ee9c54e32b02615e2cd42b27b3029a3fdabd/src/chromiumos/tremplin/main.go#84
Cc: dgreid@chromium.org
Owner: ----
Status: Available (was: Fixed)
Reopening to track an upstream solution.

As Christian mentioned in #4, we can't allow the kernel keyring to be accessed in containers if it's not namespace-safe. The problem here is in libcontainer, which shouldn't be falling flat on its face if we return ENOSYS here: https://github.com/opencontainers/runc/blob/master/libcontainer/keys/keyctl.go#L15
I tried unsetting the security.syscalls.blacklist per Comment 9.  After doing so   the restart of penguin container stalled and it and appeared to be still running but unresponsive. Setting the blacklist back to what I'm guessing it should be per Comment 4, I stopped/started up penguin again and it was working again:

Setting it back (undoing steps in Comment 9)
$ lxc stop penguin
$ lxc profile set default security.syscalls.blacklist "keyctl errno 38"
$ lxc profile apply penguin default
Profiles default applied to penguin
$ lxc start penguin

I'm on a Pixelbook using the beta channel running 10895.33.0 with Build date of Aug 23rd 2018.

Just will wait for the upstream solution.

If you perform it within about 15 seconds of login, it will apply to the not-yet-booted penguin.  You don't actually need the apply step it turns out.  So my actions are:

1. Login to Chrome OS as usual
2. CTRL+ALT+T
3. vmc start termina
4. lxc profile unset default security.syscalls.blacklist
5. CTRL+D
6. CTRL+D

That setting does not persist between login sessions, so simply logging out of Chrome OS is enough to reset it.

Also dev on my Pixelbook and beta on my CXI3 are both unstable enough that (even without penguin able to run docker) they sessions crash somewhat often.  When that happens access to penguin seems to get blown away, so I end up login-cycling often to deal with that.  But in doing so, the above must be done again each time.
I opened a corresponding issue in the opencontainers project.

https://github.com/opencontainers/runc/issues/1889
 Issue 882204  has been merged into this issue.
Upstream PR has been merged: https://github.com/opencontainers/runc/pull/1893
Thank you all for your time and attention to this.

As a Docker end user, when should I expect for this to work again within Chrome OS? I realize this is not a chromium responsibility, so I'm just asking here for this extra information as a courtesy.

I noticed that project repo doesn't do releases often (last one is Feb 2018) so I'm not sure how that runc commit to master gets to my local machine.
runc is currently versioned by git hash.

If you run docker info, you'll see which version you of runc is in use. I don't have my chromebook at hand.

I *suspect* that if you install a nightly release built after today, docker will work!

Nightly is not built every day, maybe every week?

https://download.docker.com/linux/debian/dists/stretch/
Not sure if this is informative, but I attempted to run with the nightly Docker (5:0.0.0-20180926205650-117ee8c-0~debian-stretch) to pick up this change. The Docker daemon does not start, with log entries:

> Oct 01 13:00:10 penguin systemd[1]: docker.service: Failed to reset devices.list: Operation not permitted
> Oct 01 13:00:10 penguin systemd[1]: docker.service: Failed to set invocation ID on control group /system.slice/docker.service, ignoring: Operation not permitted
> Oct 01 13:01:09 penguin systemd[1]: docker.service: Failed to reset devices.list: Operation not permitted
> Oct 01 13:01:09 penguin systemd[1]: docker.service: Failed to set invocation ID on control group /system.slice/docker.service, ignoring: Operation not permitted
@chris: There is another PR needed for docker to work: https://github.com/opencontainers/runc/pull/1862

Testing vs. nightly is a good idea, I just built a custom runc and set it as the default: https://www.reddit.com/r/Crostini/comments/9jabhq/docker_now_working/

Comment 21 Deleted

It looks like https://github.com/opencontainers/runc/pull/1862 was merged but the latest nightly build that I can find from Docker is from 9/26. Does anybody else know what I'm missing? I want to test the latest fixes on my Pixelbook.
Docker has changed its release process.

Right now, your best bet is testing with whatever build comes after 18.09.0-beta3:
https://github.com/docker/docker-ce/releases

I don't know when they will update the pinned version of runc. There was a significant regression in runc functionality and I *hope* they're waiting for it to be fixed - if so, 18.09 might include the chromeos work that's been done.

If you want to build the current version yourself, you can build moby:
https://github.com/moby/moby

From https://mobyproject.org:

Moby IS RECOMMENDED for anyone who wants to assemble a container-based system.
 * Hackers who want to customize or patch their Docker build
 * System engineers or integrators building a container system
 * Infrastructure providers looking to adapt existing container systems to their environment
 * Container enthusiasts who want to experiment with the latest container tech
 * Open-source developers looking to test their project in a variety of different systems
 * Anyone curious about Docker internals and how it’s built
I pulled the latest runc, built, and installed.

I then created a custom runtime configuration for Docker that pointed to the patched runc version.

No more errors! I'm exercising with more complex containers, but I'm hopeful that things seem to be working correctly now.
I also got docker (edge) working (again) on ChromeOS (beta channel)

Docker version 18.06.1-ce, build e68fc7a

As crshover pointed out, a home-built runc is required.

1. Install Docker via Debian instructions, I used the edge version: (https://docs.docker.com/install/linux/docker-ce/debian/#install-docker-ce-1)
2. Install Golang (https://golang.org/doc/install#install)
3. Install seccomp dev, (required to build runc with this flag enabled, highly recommended)
  `sudo apt install libseccomp-dev`
4. Getting runc source
  `go get -v github.com/opencontainers/runc`
5. Building runc
  `cd $GOPATH/src/github.com/opencontainers/runc`
  `make BUILDTAGS='seccomp apparmor'`
6. Linking runc
  `sudo ln -s $(realpath ./runc) /usr/local/bin/runc-master`
7. Configuring dockerd to use our new runc
  `sudo vim /etc/docker/daemon.json` (see below for config)
8. Reload the config
  `sudo kill -s SIGHUP $(pgrep dockerd)`
9. Enjoy
  `docker run -it ubuntu:xenial /bin/bash`


*daemon.json*
```
{
  "runtimes": {
    "runc-master": {
      "path": "/usr/local/bin/runc-master"
    }
  },
  "default-runtime": "runc-master"
}
```


Additional Resources:

https://ops.tips/blog/run-docker-with-forked-runc/#forking-runc
https://docs.nvidia.com/dgx/nvidia-container-runtime-upgrade/index.html
https://github.com/moby/moby/issues/18946
https://unix.stackexchange.com/questions/219341/how-to-apt-delete-repository
Owner: smbar...@chromium.org
Status: Fixed (was: Available)
sounds like there's nothing left here on the CrOS side
I tried out centerorbit@'s instructions which almost worked, but I needed one extra step because I was encountering this error:

$ sudo service docker restart
A dependency job for docker.service failed. See 'journalctl -xe' for details.

$ sudo systemctl status containerd.service 
● containerd.service - containerd container runtime
   Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-11-19 07:17:48 UTC; 6s ago
     Docs: https://containerd.io
  Process: 387 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
      CPU: 1ms

Nov 19 07:17:48 penguin modprobe[387]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.74-07
Nov 19 07:17:48 penguin modprobe[387]: modprobe: FATAL: Module overlay not found in directory /lib/modules/4.14.74-07727-g7815dfea1ba2

I don't know why `/sbin/modprobe overlay` is failing. To workaround this:

1. Remove the ExecSartPre line from `/lib/systemd/system/containerd.service`
2. `sudo systemctl daemon-release`
3. `sudo service docker restart`

I'm running Chrome OS Beta (71.0.3578.49 (Official Build), November 13, 2018), with a container I created today.
@za...@chromium.org,

Docker had released a new update recently, and it switches to using containerd service. In Docker version 18.06.1-ce, that wasn't the case.

I've found a fix/workaround for this though. You no longer need to custom built runc. (but it doesn't hurt any to leave that config in place).

I believe it's rooted in the fact that the debian for chromebook doesn't come with overlay driver kernel module, AND that the filesystem is btrfs, so it wouldn't work anyways (I think).

Anyways, here's how you get docker working: 
https://github.com/docker/for-linux/issues/475#issuecomment-437373774

Here is the relevant piece (copied from github comment):

Run systemctl edit containerd.service. This automatically creates a draft override file, and opens it in your editor.
Edit the file to look like this;

```
[Service]
ExecStartPre=
```

Save the file

sudo systemctl daemon-reload
sudo systemctl restart containerd.service
sudo system docker restart

Note: you will also need to add the group 'docker' to your user to use docker without sudo.

Enjoy!

Comment 29 Deleted

Thanks!  This worked for me too.  What I did instead was this:

```
[Service]
ExecStartPre=
ExecStartPre=-/sbin/modprobe overlay
```

It's exactly the same thing as stock, but with the hyphen added so it will ignore the error of `modprobe overlay` failing.  The blank assignment above it is to unset the value first, instead of adding it as a second entry.

(Edit: My original comment thanked David, but that was the reporter, not the person I was commenting to.  Oops!)
Actually, I might have jumped the gun on my response.  I tried again on a stock penguin, installed Docker 18.09.0, configured as above with the `ExecStartPre` and it doesn't work, giving the same `could not create session key: function not implemented` error as before.  Sorry about the spam.
Hey Emmaly,

You were right. My 'new fix' didn't fully fix the changes/issues. It turns out that the switched to containerd didn't fully fix the issue.

Here's what I did to get it working. I'm not sure what steps are _actually_ needed, but it seemed to be effective at some point!

First: Perform all of [Comment 28](https://bugs.chromium.org/p/chromium/issues/detail?id=860565#c28)

This will now let containerd run without crashing at startup.

Second: Perform all of [Comment 25](https://bugs.chromium.org/p/chromium/issues/detail?id=860565#c25)

Because that 'session key' issue is related to an old version of runc. Now in my trials, it seemed that the config I provided didn't seem to really "take affect", even after a reboot, so here's what I ended up doing:

1. Made a backup of contaienrd config: `cp /etc/containerd/config.toml /etc/containerd/config.toml.bak`
2. Replaced that config with the default using instructions here: https://github.com/containerd/containerd/blob/master/docs/getting-started.md
(I had to run this as full-on root by doing `sudo su` first)
`containerd config default > /etc/containerd/config.toml`
3. Edit that file and replaced:
    runtime = "runc"
With:
    runtime = "/usr/local/bin/runc-master"


Ran these again: 
```
sudo systemctl daemon-reload
sudo systemctl restart containerd.service
sudo system docker restart
```

Heck, maybe even reboot your system.

Test it now. If it works, you're lucky, if not you'll need this _extra-special_ hack:

1. Download and install docker-compose:
https://docs.docker.com/compose/install/#install-compose

2. Create an example.yml file with the following:
```
version: '2.3'
services:
  example:
    runtime: runc-master  
    image: "alpine:latest"
```
(Note: needs to be version: '2.3' in the yaml file, when runtime was introduced)

3. Run it `docker-compose -f example.yml up`

Tada! You are now free to make whatever containers you want whenever. For whatever reason, it took this last docker-compose with specific configuration to convince docker and containerd that it really could use an alternate runtime! (May require reboot... may break if rebooted... who knows!)

I found the hint for this here: https://stackoverflow.com/a/52063921

Best of luck, lot of duct tape on this one! I'll let you know if it breaks on me, or if I find a simpler/more solid way.
This is not fixed as of 70.0.3538.110 (Official Build) (64-bit):
```
. ~ docker run hello-world

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"could not create session key: function not implemented\"": unknown.
ERRO[0001] error waiting for container: context canceled
```
Labels: Restrict-AddIssueComment-EditIssue

Sign in to add a comment