k8s glusterfs heketi

Posted 2021-01-02 fengjian2016

　　kubernetes集群上node节点安装glusterfs的服务端集群（DaemonSet方式），并将heketi以deployment的方式部署到kubernetes集群，主要是实现 storageclass

Heketi是一个具有resetful接口的glusterfs管理程序, Heketi提供了一个RESTful管理界面，可用于管理GlusterFS卷的生命周期。借助Heketi，Kubernetes 可以动态地配置GlusterFS卷和任何支持的持久性类型。Heketi将自动确定整个集群的brick位置，确保将brick及其副本放置在不同的故障域中。Heketi还支持任意数量的GlusterFS集群，允许云服务提供网络文件存储，而不受限于单个GlusterFS集群。”

标注：

1.每个node 节点安装 glusterfs 客户端  yum -y  install glusterfs-client

2. 加载内核模块 : modprobe dm_thin_pool

3. 至少要3个节点， 每个节点上必须要一个裸硬盘。

 

glusterfs 和 heketi 在kubernetes 部署

1. 下载heketi 和 heketi-cli

git clone https://github.com/heketi/heketi.git

wget  https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-client-v7.0.0.linux.amd64.tar.gz

 

2. 进入到 heketi kubernetes 的安装目录

cd /root/heketi/heketi/extras/kubernetes

 

3. 部署 glusterfs daemonset 方式

kubectl create -f glusterfs-daemonset.json

 

 4. node 节点设置标签

查看标签
kubectl get nodes --show-labels

设置标签
kubectl label node node1 storagenode-glusterfs
kubectl label node node2 storagenode-glusterfs
kubectl label node node3 storagenode-glusterfs
kubectl label node node4 storagenode-glusterfs
kubectl label node node5 storagenode-glusterfs

###删除标签命令####
kubectl label node node5 storagenode-
#######

 

5.创建heketi 服务器账号（serviceaccount）

kubectl create -f heketi-service-account.json

 

6. heketi服务帐户的授权绑定相应的权限来控制gluster的pod

kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account

 

7.创建一个Kubernetes secret来保存Heketi实例的配置

kubectl create secret generic heketi-config-secret --from-file=./heketi.json

 

8.需要部署一个初始（bootstrap）Pod和一个服务来访问该Pod

 

使用 bootstrap heketi 服务 是为了  把 heketi  db 放置在 glusterfs卷上。

 

deploy-heketi服务端程序运行后， 我们可以使用heketi-cli 与服务进行通信。测试heketi 服务运行是否正常

[[email protected] kubernetes]# curl  192.168.20.178:31791/hello
Hello from Heketi
[[email protected] kubernetes]#

 

 

通过拓扑文件,向Heketi提供有关要管理的GlusterFS集群的信息

[[email protected] kubernetes]# cat topology.json 
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "node1"
              ],
              "storage": [
                "192.168.20.174"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node2"
              ],
              "storage": [
                "192.168.20.175"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node3"
              ],
              "storage": [
                "192.168.20.176"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node4"
              ],
              "storage": [
                "192.168.20.177"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "node5"
              ],
              "storage": [
                "192.168.20.178"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        }
      ]
    }
  ]
}

 

其实就是glusterfs  添加对端 gluster  peer  status

[[email protected] kubernetes]# /root/heketi/heketi-client/bin/heketi-cli -s  http://192.168.20.178:31791  topology  load --json=topology.json
Creating cluster ... ID: 6a73c32f28af53fc4eda6d5b4c9769bf
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node node1 ... ID: 46bd3a9e2ac7fe3bd80bf5d418cdcf08
        Adding device /dev/sdb ... OK
    Creating node node2 ... ID: e50ab42920c804afd010b8f739f81d1a
        Adding device /dev/sdb ... OK
    Creating node node3 ... ID: 3b5dd4c09c8216be7c5a2d919581891b
        Adding device /dev/sdb ... OK
    Creating node node4 ... ID: b42de16ea1863e7133493876dd93d00e
        Adding device /dev/sdb ... OK
    Creating node node5 ... ID: 47d1de245a14091473cda2626e605f16
        Adding device /dev/sdb ... OK
You have new mail in /var/spool/mail/root

 

使用heketi为其存储其数据库提供一个卷

/root/heketi/heketi-client/bin/heketi-cli -s  http://192.168.20.178:31791  setup-openshift-heketi-storage --image 192.168.200.10/source/heketi/heketi:dev

生成文件 heketi-storage.json

[[email protected] ~]# kubectl create -f heketi-storage.json 
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

 

heketi-storage-copy-job-c57vd   status 状态是有问题的， 没有关系，继续进行。

 

 

endpoints 中的 heketi-storage-endpoints 一定要存在，否则 创建 heketi-deployment.json 时会报错

 

 删除bootstrap Heketi实例相关的组件

[[email protected] ~]# kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
pod "deploy-heketi-7bbb8bc596-nkppj" deleted
service "deploy-heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-7bbb8bc596" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted

 

创建长期使用的Heketi实例（存储持久化的）

[[email protected] kubernetes]# kubectl create -f heketi-deployment.json 
secret/heketi-db-backup created
service/heketi created
deployment.extensions/heketi created

 

 

 

[[email protected] gluster-storage]# cat  storage-class-slow.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow                                   #-------------SC的名字
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://192.168.20.177:31791"       #-------------heketi service的cluster ip 和端口
  restuser: "admin"                            #-------------随便填，因为没有启用鉴权模式
  gidMin: "40000"
  gidMax: "50000"
  volumetype: "replicate:3"                    #-------------申请的默认为3副本模式

 

 

 

 

 

 

 

清理集群

清理有状态服务，清理daemonset部署的 glusterfs集群

kubectl delete  secret  heketi-config-secret

kubectl delete  clusterrolebinding heketi-gluster-admin

 

 2. 清理node节点gluster文件

（1）删除不运行的docker, 其中包括glusterfs
docker ps -a | grep Exited | awk ‘{print $1}‘ | xargs docker rm

（2） 删除/var/lib/glusterd 目录
pkill -9 gluster; rm /var/lib/glusterd/  -rf

(3) 清除vg，pv
[[email protected] ~]# vgremove  vg_ebaa3e7189cabac78a48426a90a14bd9
[[email protected] ~]# pvremove  /dev/sdb

（4） 删除node vg
 rm  -rf  /dev/vg_*