k8s持久化存储

Posted 2022-01-22 Qiao_Zhi

k8s持久化存储

1. 以前数据持久化方式

　　通过volumes 数据卷挂载

1. web3.yaml 内容如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: web3
  name: web3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web3
  strategy: 
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: web3
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: 
        volumeMounts:
        - name: varlog
          mountPath: /tmp/log
      volumes:
      - name: varlog
        hostPath:
          path: /tmp/log/web3log
status: 

2. 创建资源后查看

[root@k8smaster1 volumestest]# kubectl get pods | grep web3
web3-6c6557674d-xt7kr                           1/1     Running   0          6m38s
[root@k8smaster1 volumestest]# kubectl describe pods web3-6c6557674d-xt7kr

相关挂载信息如下：

 3. 到容器内部创建一个文件

[root@k8smaster1 volumestest]# kubectl exec -it web3-6c6557674d-xt7kr
error: you must specify at least one command for the container
[root@k8smaster1 volumestest]# kubectl exec -it web3-6c6557674d-xt7kr -- bash
root@web3-6c6557674d-xt7kr:/# echo "123" > /tmp/log/test.txt
root@web3-6c6557674d-xt7kr:/# exit
exit

4. 到pod 调度的节点查看宿主机目录是否挂载成功

(1) master 节点查看pod 调度节点

[root@k8smaster1 volumestest]# kubectl get pods -o wide | grep web3
web3-6c6557674d-xt7kr                           1/1     Running   0          11m     10.244.2.108     k8snode2     <none>           <none>

(2) 到k8snode2 节点查看

[root@k8snode2 web3log]# ll
total 4
-rw-r--r-- 1 root root 4 Jan 21 05:49 test.txt
[root@k8snode2 web3log]# cat test.txt 
123

5. 测试k8snode2 节点宕机，pod 自动调度到k8snode1 节点再次查看

[root@k8smaster1 volumestest]# kubectl get pods -o wide | grep web3
web3-6c6557674d-6wlh4                           1/1     Running       0          4m22s   10.244.1.110     k8snode1     <none>           <none>
web3-6c6557674d-xt7kr                           1/1     Terminating   0          22m     10.244.2.108     k8snode2     <none>           <none>
[root@k8smaster1 volumestest]# kubectl exec -it web3-6c6557674d-6wlh4 -- bash
root@web3-6c6557674d-6wlh4:/# ls /tmp/log/
root@web3-6c6557674d-6wlh4:/# 

 　　发现自动调度到k8snode1 节点，进入容器之后发现之前新建的文件丢失。

6. 从k8snode1 宿主机查看发现也没有文件

[root@k8snode1 web3log]# pwd
/tmp/log/web3log
[root@k8snode1 web3log]# ls
[root@k8snode1 web3log]# 

　　造成的现象就是pod 所在的节点宕机后，volume 数据卷挂载的文件也丢失，因此需要一种解决方案。

1. nfs 持久化存储

　　网络文件系统，是一种共享文件系统，实际上相当于客户端将文件上传到服务器，实现共享。

1. 下载nfs

1. 找一台服务器安装nfs

(1) 安装nfs以及查看nfs 服务状态

yum install -y nfs-utils

(2) 设置挂载路径, 注意需要将挂载路径创建出来

[root@k8smaster2 logs]# cat /etc/exports
/data/nfs *(rw,no_root_squash)

解释： rw 代表读写访问， no_root_squash 代表root 用户具有根目录的完全管理访问权限

2. k8s 集群node 节点安装nfs-utils

yum install -y nfs-utils

3. nfs 服务器启动nfs 服务且查看服务状态

[root@k8smaster2 nfs]# systemctl start nfs    # 启动nfs
[root@k8smaster2 nfs]# systemctl status nfs    # 查看状态
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
   Active: active (exited) since Fri 2022-01-21 19:55:38 EST; 5min ago
  Process: 51947 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
  Process: 51943 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 51941 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS)
  Process: 51977 ExecStartPost=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status=0/SUCCESS)
  Process: 51960 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 51958 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 51960 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
   CGroup: /system.slice/nfs-server.service

Jan 21 19:55:38 k8smaster2 systemd[1]: Starting NFS server and services...
Jan 21 19:55:38 k8smaster2 systemd[1]: Started NFS server and services.
[root@k8smaster2 nfs]# showmount -e localhost    # 查看挂载的nfs 信息
Export list for localhost:
/data/nfs *

也可以查看nfs 的进程信息

[root@k8smaster2 nfs]# ps -ef | grep nfs
root      51962      2  0 19:55 ?        00:00:00 [nfsd4_callbacks]
root      51968      2  0 19:55 ?        00:00:00 [nfsd]
root      51969      2  0 19:55 ?        00:00:00 [nfsd]
root      51970      2  0 19:55 ?        00:00:00 [nfsd]
root      51971      2  0 19:55 ?        00:00:00 [nfsd]
root      51972      2  0 19:55 ?        00:00:00 [nfsd]
root      51973      2  0 19:55 ?        00:00:00 [nfsd]
root      51974      2  0 19:55 ?        00:00:00 [nfsd]
root      51975      2  0 19:55 ?        00:00:00 [nfsd]
root      54774  45013  0 20:02 pts/2    00:00:00 grep --color=auto nfs

2. 客户端安装

 1. 在所有k8snode 节点安装客户端，并且查看远程nfs 信息

yum install -y nfs-utils

2. 查看远程信息

[root@k8snode1 ~]# showmount -e 192.168.13.106
Export list for 192.168.13.106:
/data/nfs *

3. 本地测试nfs

(1) 创建挂载并进行测试

[root@k8snode1 ~]# mkdir /share
[root@k8snode1 ~]# mount 192.168.13.106:/data/nfs /share
[root@k8snode1 ~]# df -h | grep 13.106
192.168.13.106:/data/nfs   17G   12G  5.4G  69% /share

(2) node 节点创建文件

[root@k8snode1 ~]# echo "hello from 104" >> /share/104.txt
[root@k8snode1 ~]# cat /share/104.txt 
hello from 104

(3) nfs 服务器查看

[root@k8smaster2 nfs]# cat 104.txt 
hello from 104

(4) 客户端取消挂载

[root@k8snode1 ~]# umount /share
[root@k8snode1 ~]# df -h | grep 13.106

取消挂载之后，nfs 服务器上的文件仍然存在。

3. k8s 集群使用nfs

1. 编写nfs-nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dep1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: wwwroot
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
      volumes:
        - name: wwwroot
          nfs:
            server: 192.168.13.106
            path: /data/nfs

2. 创建资源

[root@k8smaster1 nfs]# kubectl apply -f nfs-nginx.yaml 
deployment.apps/nginx-dep1 created

然后查看pod describe 信息

3. 我们进入容器然后创建一个文件导/usr/share/nginx/html

root@nginx-dep1-6d7f9c85dc-lqfbf:/# cat /usr/share/nginx/html/index.html 
hello

4. 然后到nfs 服务器查看

[root@k8smaster2 nfs]# pwd
/data/nfs
[root@k8smaster2 nfs]# ls
104.txt  index.html
[root@k8smaster2 nfs]# cat index.html 
hello

4. pv 和 pvc

上面使用nfs 有一个问题，就是每个需要持久化的都需要知道远程nfs 服务器的地址以及相关权限，可能不太安全。下面研究pv和pvc 使用。

pv pvc 对应PersistentVolume和PersistentVolumeClaim。  pv 类似于一个声明nfs 地址等信息，抽象成配置文件； pvc 通过引用pv 中声明的信息，然后即可实现nfs 持久化存储。

pv 有好多实现方式，实际上是对nfs进行一层包装，因为我们已经安装了nfs， 所以基于nfs 实现。

参考： https://kubernetes.io/docs/concepts/storage/persistent-volumes/

1.  创建pv

1. 创建 pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /data/nfs
    server: 192.168.13.106

2. 创建并查看

[root@k8smaster1 nfs]# kubectl apply -f pv.yaml 
persistentvolume/my-pv created
[root@k8smaster1 nfs]# kubectl get pv -o wide
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE    VOLUMEMODE
my-pv   5Gi        RWX            Retain           Available                                   2m4s   Filesystem

2. 创建pvc 使用上面的pv

1. 创建pvc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dep1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: wwwroot
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
      volumes:
      - name: wwwroot
        persistentVolumeClaim:
          claimName: my-pvc

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

2. 创建并查看

[root@k8smaster1 nfs]# kubectl apply -f pvc.yaml 
deployment.apps/nginx-dep1 created
persistentvolumeclaim/my-pvc created
[root@k8smaster1 nfs]# kubectl get pvc -o wide
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
my-pvc   Bound    my-pv    5Gi        RWX                           60s   Filesystem
[root@k8smaster1 nfs]# kubectl get pods -o wide
NAME                                            READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
nginx-dep1-58b7bf955f-4jhbq                     1/1     Running   0          75s     10.244.2.112     k8snode2     <none>           <none>
nginx-dep1-58b7bf955f-m69dm                     1/1     Running   0          75s     10.244.2.110     k8snode2     <none>           <none>
nginx-dep1-58b7bf955f-qh6pg                     1/1     Running   0          75s     10.244.2.111     k8snode2     <none>           <none>
nginx-f89759699-vkf7d                           1/1     Running   3          4d16h   10.244.1.106     k8snode1     <none>           <none>
tomcat-58767d5b5-f5qwj                          1/1     Running   2          4d15h   10.244.1.103     k8snode1     <none>           <none>
weave-scope-agent-ui-kbq7b                      1/1     Running   2          45h     192.168.13.105   k8snode2     <none>           <none>
weave-scope-agent-ui-tg5q4                      1/1     Running   2          45h     192.168.13.103   k8smaster1   <none>           <none>
weave-scope-agent-ui-xwh2b                      1/1     Running   2          45h     192.168.13.104   k8snode1     <none>           <none>
weave-scope-cluster-agent-ui-7498b8d4f4-zdlk7   1/1     Running   2          45h     10.244.1.104     k8snode1     <none>           <none>
weave-scope-frontend-ui-649c7dcd5d-7gb9s        1/1     Running   2          45h     10.244.1.107     k8snode1     <none>           <none>
web3-6c6557674d-6wlh4                           1/1     Running   0          14h     10.244.1.110     k8snode1     <none>           <none>
[root@k8smaster1 nfs]# 

3. 随便进入一个pod的第一个容器，然后创建文件

[root@k8smaster1 nfs]# kubectl exec -it nginx-dep1-58b7bf955f-4jhbq -- bash
root@nginx-dep1-58b7bf955f-4jhbq:/# echo "111222" >> /usr/share/nginx/html/1.txt
root@nginx-dep1-58b7bf955f-4jhbq:/# exit
exit

4. 到nfs 服务器查看与其他容器查看

(1) nfs 服务器查看

[root@k8smaster2 nfs]# ls
104.txt  1.txt  index.html
[root@k8smaster2 nfs]# cat 1.txt 
111222

(2) 进入其他pod 的第一个容器查看

[root@k8smaster1 nfs]# kubectl exec -it nginx-dep1-58b7bf955f-qh6pg -- bash
root@nginx-dep1-58b7bf955f-qh6pg:/# ls /usr/share/nginx/html/
1.txt  104.txt  index.html

 

　　至此简单实现了基于nfs 和 pv、pvc 的持久化存储。

 

【当你用心写完每一篇博客之后,你会发现它比你用代码实现功能更有成就感!】