Kubernetes 部署 RabbitMQ 3.7.X 集群遇到的问题

Posted 2023-02-25 回归心灵

问题描述

在 Kubernetes 上部署 RabbitMQ 集群时，我们可以参考 https://github.com/rabbitmq/diy-kubernetes-examples/tree/master/minikube 下面的编排文件部署，使用上述编排文件可以部署一个正确的 RabbitMQ 集群。RabbitMQ 的版本号是 3.8。

由于我们公司用的是在 3.7.24 版本上修改过的 RabbitMQ，因此修改 RabbitMQ 版本号部署，但是却遇到了问题（3.8 版本没有问题）。在 statefulset.yaml 编排文件中添加了默认用户名和密码的环境变量配置。

启动时日志报错：

sed: couldn't open temporary file /etc/rabbitmq/sed7ee9aW: Read-only file system

又使用 initContainers 容器修改 /etc/rabbitmq 权限为 777 ，依然不能正常启动。

问题解决

经过不断地修改验证最终采用拷贝配置文件到 /etc/rabbitmq 目录的方式解决，编排文件如下：

apiVersion: apps/v1
# See the Prerequisites section of https://www.rabbitmq.com/cluster-formation.html#peer-discovery-k8s.
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: test-rabbitmq1
spec:
  serviceName: rabbitmq
  # Three nodes is the recommended minimum. Some features may require a majority of nodes
  # to be available.
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      nodeSelector:
        # Use Linux nodes in a mixed OS kubernetes cluster.
        # Learn more at https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#kubernetes-io-os
        kubernetes.io/os: linux
      containers:
      - name: rabbitmq-k8s
        image: rabbitmq:3.7.24
        volumeMounts:
          - name: rabbitmq-config-rw
            mountPath: /etc/rabbitmq
        # Learn more about what ports various protocols use
        # at https://www.rabbitmq.com/networking.html#ports
        ports:
          - name: http
            protocol: TCP
            containerPort: 15672
          - name: amqp
            protocol: TCP
            containerPort: 5672
        livenessProbe:
          exec:
            # This is just an example. There is no "one true health check" but rather
            # several rabbitmq-diagnostics commands that can be combined to form increasingly comprehensive
            # and intrusive health checks.
            # Learn more at https://www.rabbitmq.com/monitoring.html#health-checks.
            #
            # Stage 2 check:
            command: ["rabbitmq-diagnostics", "status"]
          initialDelaySeconds: 60
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
          periodSeconds: 60
          timeoutSeconds: 15
        readinessProbe:
          exec:
            # This is just an example. There is no "one true health check" but rather
            # several rabbitmq-diagnostics commands that can be combined to form increasingly comprehensive
            # and intrusive health checks.
            # Learn more at https://www.rabbitmq.com/monitoring.html#health-checks.
            #
            # Stage 1 check:
            command: ["rabbitmq-diagnostics", "ping"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        imagePullPolicy: Always
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: RABBITMQ_USE_LONGNAME
            value: "true"
          # See a note on cluster_formation.k8s.address_type in the config file section
          - name: K8S_SERVICE_NAME
            value: rabbitmq
          - name: RABBITMQ_NODENAME
            value: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
          - name: K8S_HOSTNAME_SUFFIX
            value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
          - name: RABBITMQ_ERLANG_COOKIE
            value: "mycookie"
          - name: RABBITMQ_DEFAULT_USER
            value: guest
          - name: RABBITMQ_DEFAULT_PASS
            value: aaaaaa
          - name: RABBITMQ_DEFAULT_VHOST
            value: host
      initContainers:
      # Since k8s 1.9.4, config maps mount read-only volumes. Since the Docker image also writes to the config file,
      # the file must be mounted as read-write. We use init containers to copy from the config map read-only
      # path, to a read-write path
      - name: init
        image: d8tlzzxs.mirror.aliyuncs.com/library/busybox:1.31.1
        volumeMounts:
        - name: config-volume
          mountPath: /tmp/rabbitmq
        - name: rabbitmq-config-rw
          mountPath: /etc/rabbitmq
        command:
        - sh
        - -c
        # the newline is needed since the Docker image entrypoint scripts appends to the config file
        - cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf;
          cp /tmp/rabbitmq/enabled_plugins /etc/rabbitmq/enabled_plugins
      volumes:
        - name: config-volume
          configMap:
            name: rabbitmq-config
            items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
            - key: enabled_plugins
              path: enabled_plugins
        - name: rabbitmq-config-rw
          emptyDir: 
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "rabbitmq"
            topologyKey: "kubernetes.io/hostname"

参考

• https://github.com/rabbitmq/diy-kubernetes-examples/tree/master/minikube
• https://github.com/rabbitmq/diy-kubernetes-examples/blob/55b9ae92f64bd11ae894daaef870841da56f9503/gke/statefulset.yaml