New issue
Advanced search Search tips

Issue 715009 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2017
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 3
Type: Bug



Sign in to add a comment

Deploy glusterfs/heketi inside Kubernetes 1.6.1 on Google container Engine cos image

Reported by d.berard...@infonial.com, Apr 25 2017

Issue description

**Kubernetes version**: 
1.6.1

**Environment**:
- **Cloud provider or hardware configuration**: 
Google container engine

- **OS**:
BUILD_ID=9000.104.0
NAME="Container-Optimized OS"
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=56
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=56
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos

- **Kernel**:
Linux gke-gup-kube-1-default-pool-1b1de7e8-3jh9 4.4.21+ #1 SMP Wed Apr 5 14:40:46 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

- **Install tools**:
Google Container engine

- **Others**:


**What happened**:
I am facing a problem when deploying Glusterfs + Heketi (thanks to [this excellent repo](https://github.com/gluster/gluster-kubernetes)) inside a GKE Kubernetes 1.6.1 cluster with nodes using the cos image. The gk-deploy script went well until it tries to execute heketi-cli -s http://${heketi_service} setup-openshift-heketi-storage that requires the dm_thin_pool module to be loaded.

The command fails with:
```
Error: Unable to execute command on glusterfs-xxxxx:   /usr/sbin/modprobe failed: 1
thin: Required device-mapper target(s) not detected in your kernel.
Run `lvcreate --help' for more information.
Failed on setup openshift heketi storage
```

The fix normally consists in using `modeprobe dm_thin_pool` to each Kubernetes node, But it won't work on the cos image type.

**What you expected to happen**:
We cannot use `modprobe dm_thin_pool` on the cos image as it will not find it, and there is no way to install new package to my knowledge. I would expect it to be included in cos image.

**How to reproduce it**:
ssh into one of the worker intances and `modprobe dm_thin_pool`

**Anything else we need to know**:
To be honest, I do not really know where is the best place to file this kind of issue related to cos image. My apologies if I am not choosing the right place. In this case, may you help me to choose a better one ?
 
Owner: adityakali@google.com
Thanks for reporting the issue. I will look into this.

And also possibly better documentation for reporting issues :) Sorry for the inconvenience.

Project Member

Comment 2 by bugdroid1@chromium.org, May 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/5fe5cda98e552a7d5f55d9270276067d0aaf8d06

commit 5fe5cda98e552a7d5f55d9270276067d0aaf8d06
Author: Aditya Kali <adityakali@google.com>
Date: Wed May 10 21:24:36 2017

lakitu: kernel: enable DM_THIN_PROVISIONINIG module

Allows users to use device mapper with other filesystems like glusterfs.

BUG= chromium:715009 
TEST=built image verified that the module can be loaded
RELEASE_NOTE=Enabled DM_THIN_PROVISIONING support

Change-Id: Ic5999856d583b7d2a5b3aa11a9276d7fb7dc1ce5
Reviewed-on: https://chromium-review.googlesource.com/501268
Commit-Ready: Aditya Kali <adityakali@google.com>
Tested-by: Aditya Kali <adityakali@google.com>
Reviewed-by: Andrey Ulanov <andreyu@google.com>

[modify] https://crrev.com/5fe5cda98e552a7d5f55d9270276067d0aaf8d06/overlay-lakitu/sys-kernel/lakitu-kernel-4_4/files/lakitu_kernel_config_4_4

Update:
cos-dev-60-9540-0-0 (https://cloud.google.com/container-optimized-os/docs/release-notes#cos-dev-60-9540-0-0) and newer images now have DM_THIN_PROVISIONING enabled.

You can choose this image while creating your kubernetes cluster by setting following env vars before doing kube_up.sh:

export KUBE_OS_DISTRIBUTION=cos
export KUBE_GCE_MASTER_PROJECT=cos-cloud
export KUBE_GCE_MASTER_IMAGE=cos-dev-60-9540-0-0
export KUBE_GCE_NODE_PROJECT=cos-cloud
export KUBE_GCE_NODE_IMAGE=cos-dev-60-9540-0-0

So now `modprobe dm_thin_pool` works and loads the module. Though you will have to manually run it on all nodes because of issue https://github.com/gluster/gluster-kubernetes/issues/19.

I tried following the instructions at https://github.com/gluster/gluster-kubernetes and https://github.com/heketi/heketi/wiki/Kubernetes-Integration.
Everything went fine until I ran into issue https://github.com/heketi/heketi/issues/764 (heketi-cli uses xfs to configure the glusterfs, but Container-Optimized OS only supports ext4).

Next step for this would be for heketi to support ext4 based glusterfs volumes. I will follow up on the upstream issue for it and closing this one.
Status: Fixed (was: Unconfirmed)
Thanks a lot !

Sign in to add a comment