Deploy glusterfs/heketi inside Kubernetes 1.6.1 on Google container Engine cos image
Reported by
d.berard...@infonial.com,
Apr 25 2017
|
||
Issue description**Kubernetes version**: 1.6.1 **Environment**: - **Cloud provider or hardware configuration**: Google container engine - **OS**: BUILD_ID=9000.104.0 NAME="Container-Optimized OS" GOOGLE_CRASH_ID=Lakitu VERSION_ID=56 BUG_REPORT_URL=https://crbug.com/new PRETTY_NAME="Container-Optimized OS from Google" VERSION=56 GOOGLE_METRICS_PRODUCT_ID=26 HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/" ID=cos - **Kernel**: Linux gke-gup-kube-1-default-pool-1b1de7e8-3jh9 4.4.21+ #1 SMP Wed Apr 5 14:40:46 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux - **Install tools**: Google Container engine - **Others**: **What happened**: I am facing a problem when deploying Glusterfs + Heketi (thanks to [this excellent repo](https://github.com/gluster/gluster-kubernetes)) inside a GKE Kubernetes 1.6.1 cluster with nodes using the cos image. The gk-deploy script went well until it tries to execute heketi-cli -s http://${heketi_service} setup-openshift-heketi-storage that requires the dm_thin_pool module to be loaded. The command fails with: ``` Error: Unable to execute command on glusterfs-xxxxx: /usr/sbin/modprobe failed: 1 thin: Required device-mapper target(s) not detected in your kernel. Run `lvcreate --help' for more information. Failed on setup openshift heketi storage ``` The fix normally consists in using `modeprobe dm_thin_pool` to each Kubernetes node, But it won't work on the cos image type. **What you expected to happen**: We cannot use `modprobe dm_thin_pool` on the cos image as it will not find it, and there is no way to install new package to my knowledge. I would expect it to be included in cos image. **How to reproduce it**: ssh into one of the worker intances and `modprobe dm_thin_pool` **Anything else we need to know**: To be honest, I do not really know where is the best place to file this kind of issue related to cos image. My apologies if I am not choosing the right place. In this case, may you help me to choose a better one ?
,
May 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/5fe5cda98e552a7d5f55d9270276067d0aaf8d06 commit 5fe5cda98e552a7d5f55d9270276067d0aaf8d06 Author: Aditya Kali <adityakali@google.com> Date: Wed May 10 21:24:36 2017 lakitu: kernel: enable DM_THIN_PROVISIONINIG module Allows users to use device mapper with other filesystems like glusterfs. BUG= chromium:715009 TEST=built image verified that the module can be loaded RELEASE_NOTE=Enabled DM_THIN_PROVISIONING support Change-Id: Ic5999856d583b7d2a5b3aa11a9276d7fb7dc1ce5 Reviewed-on: https://chromium-review.googlesource.com/501268 Commit-Ready: Aditya Kali <adityakali@google.com> Tested-by: Aditya Kali <adityakali@google.com> Reviewed-by: Andrey Ulanov <andreyu@google.com> [modify] https://crrev.com/5fe5cda98e552a7d5f55d9270276067d0aaf8d06/overlay-lakitu/sys-kernel/lakitu-kernel-4_4/files/lakitu_kernel_config_4_4
,
Jun 1 2017
Update: cos-dev-60-9540-0-0 (https://cloud.google.com/container-optimized-os/docs/release-notes#cos-dev-60-9540-0-0) and newer images now have DM_THIN_PROVISIONING enabled. You can choose this image while creating your kubernetes cluster by setting following env vars before doing kube_up.sh: export KUBE_OS_DISTRIBUTION=cos export KUBE_GCE_MASTER_PROJECT=cos-cloud export KUBE_GCE_MASTER_IMAGE=cos-dev-60-9540-0-0 export KUBE_GCE_NODE_PROJECT=cos-cloud export KUBE_GCE_NODE_IMAGE=cos-dev-60-9540-0-0 So now `modprobe dm_thin_pool` works and loads the module. Though you will have to manually run it on all nodes because of issue https://github.com/gluster/gluster-kubernetes/issues/19. I tried following the instructions at https://github.com/gluster/gluster-kubernetes and https://github.com/heketi/heketi/wiki/Kubernetes-Integration. Everything went fine until I ran into issue https://github.com/heketi/heketi/issues/764 (heketi-cli uses xfs to configure the glusterfs, but Container-Optimized OS only supports ext4). Next step for this would be for heketi to support ext4 based glusterfs volumes. I will follow up on the upstream issue for it and closing this one.
,
Jun 1 2017
,
Jun 2 2017
Thanks a lot ! |
||
►
Sign in to add a comment |
||
Comment 1 by adityakali@google.com
, May 5 2017