New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 860874 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 11
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

Termina Kernel Missing VXLAN and IPVLAN

Reported by jamessmi...@gmail.com, Jul 7

Issue description

UserAgent: Mozilla/5.0 (X11; CrOS x86_64 10820.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3473.0 Safari/537.36
Platform: 10820.0.0 (Official Build) dev-channel eve

Steps to reproduce the problem:
1. docker swarm init --advertise-addr eth0
2. docker stack <DESIRED_STACK>
3. docker service ps --no-trunc <DESIRED_SERVICE>

What is the expected behavior?
Service is running.

What went wrong?
Service is shutdown. Error message is the following : "starting container failed: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: operation not supported"

Did this work before? N/A 

Chrome version: 69.0.3473.0  Channel: dev
OS Version: 10820.0.0
Flash Version: 

Looking into the error message, issue seems to be kernel-related. As Crostini is running Termina, I'm unsure what the resolution of the fix should be. For other kernels, the solution was to ensure CONFIG_IPVLAN and CONFIG_VXLAN were not missing.

The following script was provided to check if the system had everything needed : https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh

Here is my system's output :

./check-config.sh 
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
- CONFIG_MEMCG_SWAP_ENABLED: missing
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: missing
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: missing
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled
    - CONFIG_NF_CONNTRACK_FTP: enabled
    - CONFIG_NF_NAT_TFTP: enabled
    - CONFIG_NF_CONNTRACK_TFTP: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

And filtering by the desired configs :

./check-config.sh | grep -e CONFIG_IPVLAN -e CONFIG_VXLAN
    - CONFIG_VXLAN: missing
    - CONFIG_IPVLAN: missing

Lastly, here are my error logs from Docker :

docker service ps --no-trunc func_alertmanager
ID                          NAME                      IMAGE                                                                                                    NODE                DESIRED STATE       CURRENT STATE          ERROR                                                                                                                                PORTS
fkapie0f2uhxaacu2kcs9aftm   func_alertmanager.1       prom/alertmanager:v0.15.0-rc.0@sha256:db97c29cac1a94133a3cd20df995fd7a6193173183932994c24c7253e5f0cb5d   penguin             Ready               Ready 2 seconds ago                                                                                                                                         
ykxvxvwng78zhltql2fmp2pko    \_ func_alertmanager.1   prom/alertmanager:v0.15.0-rc.0@sha256:db97c29cac1a94133a3cd20df995fd7a6193173183932994c24c7253e5f0cb5d   penguin             Shutdown            Failed 2 minutes ago   "starting container failed: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: operation not supported"   
yl1qom87yu5qvy2f257mifp41    \_ func_alertmanager.1   prom/alertmanager:v0.15.0-rc.0@sha256:db97c29cac1a94133a3cd20df995fd7a6193173183932994c24c7253e5f0cb5d   penguin             Shutdown            Failed 4 minutes ago   "starting container failed: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: operation not supported"   
zkd9dnpnp0ki5m4pbrewnhqho    \_ func_alertmanager.1   prom/alertmanager:v0.15.0-rc.0@sha256:db97c29cac1a94133a3cd20df995fd7a6193173183932994c24c7253e5f0cb5d   penguin             Shutdown            Failed 5 minutes ago   "starting container failed: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: operation not supported"   
yk51ahjbavzj35pk8dnm9twbl    \_ func_alertmanager.1   prom/alertmanager:v0.15.0-rc.0@sha256:db97c29cac1a94133a3cd20df995fd7a6193173183932994c24c7253e5f0cb5d   penguin             Shutdown            Failed 6 minutes ago   "starting container failed: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: operation not supported"
 
Components: OS>Systems>Containers
Labels: -Pri-2 Proj-Containers Pri-3
Owner: smbar...@chromium.org
Status: Assigned (was: Unconfirmed)
Cc: jkwang@chromium.org
Project Member

Comment 3 by bugdroid1@chromium.org, Jul 11

Labels: merge-merged-chromeos-4.14
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3e378b7ea9cb9d4f9aec4640f4bd068c194b7fd4

commit 3e378b7ea9cb9d4f9aec4640f4bd068c194b7fd4
Author: Stephen Barber <smbarber@chromium.org>
Date: Wed Jul 11 19:13:21 2018

CHROMIUM: arch: x86: configs: enable IPVLAN and other configs

Enable IPVLAN and VXLAN for Docker swarms. Also enable POSIX
ACLs for btrfs, which is optional for Docker.

BUG= chromium:860874 
TEST=emerge-tatl chromeos-kernel-4_14

Signed-off-by: Stephen Barber <smbarber@chromium.org>

Change-Id: I7570699cd5aad87e30d4a39e88a1197b3f2e8778
Reviewed-on: https://chromium-review.googlesource.com/1131922
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>

[modify] https://crrev.com/3e378b7ea9cb9d4f9aec4640f4bd068c194b7fd4/arch/x86/configs/chromiumos-container-vm-x86_64_defconfig

Project Member

Comment 4 by bugdroid1@chromium.org, Jul 11

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c7037cbdad1257b10f07a482e223b4abdef9110c

commit c7037cbdad1257b10f07a482e223b4abdef9110c
Author: Stephen Barber <smbarber@chromium.org>
Date: Wed Jul 11 19:13:01 2018

CHROMIUM: arch: arm64: configs: enable IPVLAN and other configs

Enable IPVLAN and VXLAN for Docker swarms. Also enable POSIX
ACLs for btrfs, which is optional for Docker.

BUG= chromium:860874 
TEST=emerge-tael chromeos-kernel-4_14

Change-Id: I7ce999a7ca389ad8ca0579389499fdca18017d13
Signed-off-by: Stephen Barber <smbarber@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1132076

[modify] https://crrev.com/c7037cbdad1257b10f07a482e223b4abdef9110c/arch/arm64/configs/chromiumos-container-vm-arm64_defconfig

Status: Fixed (was: Assigned)
Should be fixed in the next component push.
Has this been released in the latest update (https://chromereleases.googleblog.com/2018/07/dev-channel-update-for-chrome-os.html)? I am still having issues, although the errors are different.

Here is an update on my system's output, this time CONFIG_VXLAN is now enabled, but CONFIG_IPVLAN is still missing :

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
- CONFIG_MEMCG_SWAP_ENABLED: missing
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: missing
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: missing
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled
    - CONFIG_NF_CONNTRACK_FTP: enabled
    - CONFIG_NF_NAT_TFTP: enabled
    - CONFIG_NF_CONNTRACK_TFTP: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

I am getting this error from some of the services, which has been mentioned in this GitHub issue with Linode kernels (https://github.com/moby/moby/issues/36008) : 

container ingress-sbox is already present in sandbox ingress_sbox

And this error from other services, which seems to be a networking issue but the solution mentioned didn't work (https://stackoverflow.com/questions/40524602/error-creating-default-bridge-network-cannot-create-network-docker0-conf) :

starting container failed: error creating external connectivity network: cannot create network ... (docker_gwbridge): conflicts with network ... (docker_gwbridge): networks have same bridge name

If it hasn't been released yet, can someone comment the day it will come out. Thanks!
Project Member

Comment 7 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6c8b4866c7d9db48b8b5802e23b9f52d430c5dce

commit 6c8b4866c7d9db48b8b5802e23b9f52d430c5dce
Author: Stephen Barber <smbarber@chromium.org>
Date: Fri Jul 20 15:36:35 2018

CHROMIUM: arch: x86: configs: enable IPVLAN

The last CL enabled the dependencies for IPVLAN, but not
the config itself. So enable CONFIG_IPVLAN.

BUG= chromium:860874 
TEST=emerge-tatl chromeos-kernel-4_14
TBR=dgreid

Signed-off-by: Stephen Barber <smbarber@chromium.org>

Change-Id: Id5f351dc47e4c8a10d28c76038b79af5b60ff3d1
Reviewed-on: https://chromium-review.googlesource.com/1144785
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[modify] https://crrev.com/6c8b4866c7d9db48b8b5802e23b9f52d430c5dce/arch/x86/configs/chromiumos-container-vm-x86_64_defconfig

Project Member

Comment 8 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/976d3bc6f2d61fe8d443adb970c2e19bc89206e0

commit 976d3bc6f2d61fe8d443adb970c2e19bc89206e0
Author: Stephen Barber <smbarber@chromium.org>
Date: Fri Jul 20 15:36:34 2018

CHROMIUM: arch: arm64: configs: enable IPVLAN

The last CL enabled the dependencies for IPVLAN, but not
the config itself. So enable CONFIG_IPVLAN.

BUG= chromium:860874 
TEST=emerge-tael chromeos-kernel-4_14
TBR=dgreid

Change-Id: I4c6af7cf88e7a6a0237f7eb01ddda3226cc9711d
Signed-off-by: Stephen Barber <smbarber@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1144786

[modify] https://crrev.com/976d3bc6f2d61fe8d443adb970c2e19bc89206e0/arch/arm64/configs/chromiumos-container-vm-arm64_defconfig

Sign in to add a comment