Kubelet运行在容器下使用LocalPV磁盘地址不存在问题

Posted 2021-09-10 朱世伟

问题

今天在通过kubernetes部署elasticsearch的过程中出现一个问题，特此记录一下。因为elasticsearch是有状态应用，需要使用持久化存储，手头既没有云存储，也没有nfs，cephfs之类的存储。kubernetes给的elasticsearch的yaml资源文件默认的是emptydir方式，并且文档强调：

Storage

The Elasticsearch StatefulSet will use the EmptyDir volume to store data. EmptyDir is erased when the pod terminates, here it is used only for testing purposes. Important: please change the storage to persistent volume claim before actually using this StatefulSet in your setup!

说白了，这块需要自由发挥，选择适合自己的持久化存储。所以理所应当使用本地磁盘，Local PV的方式，PV的资源文件如下：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-holder-01
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /home/k8s/localpv  # 节点上的目录
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kubernetes-node03

其中指定了/home/k8s/localpv目录作为存储目录，这些都不是关键点，关键点在于，最后elasticsearch的pod启动时会报错：

Error: MountVolume.NewMounter initialization failed for volume “local-pv-xxxxx” : path “/home/k8s/localpv” does not exist

找不到/home/k8s/localpv路径。

原因

因为我的Kubernetes集群是通过Rancher安装的RKE集群，并不是原生通过Kubeadm安装的，所以集群组件都运行在Docker容器内，所以kubelet 运行在容器内部，无法读到 Node 节点上的路径。

官方FAQ也说明了这个问题

Volume does not exist with containerized kubelet

If your kubelet is running in a container, it may not be able to access the path on the host.

In order to allow the kubelet to access the path on the host, you must prefix hostDir with the prefix of the host filesystem in the kubelet container or mount the directory of the local volumes into the kubelet container at the same path.

For example, if the root filesystem of the host is mounted at /rootfs in the kubelet container, you need to prefix the hostDir with /rootfs. This requires recreating the local PV objects. You can delete them all and wait for them to be discovered again.

Another solution is to add a bind in the kubelet deployment configuration to mount the parent directory of local volumes into the kubelet container at the same path. This requires restarting the kubelet container.

For Rancher clusters, users must configure additional local volumes (or the parent directory) via Extra Binds.

解决方法如下：

services:
    kubelet:
      extra_binds:
        - "/host/dev:/dev"
        - "/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins:z"