k8s部署Rabbitmq集群

Posted 2021-04-24 运维菌

rabbitmq-peer-discovery-k8s是RabbitMQ官方基于第三方开源项目rabbitmq-autocluster开发，对3.7.X版本提供的Kubernetes下的对等发现插件，可实现rabbitmq集群在k8s中的自动化部署；低于3.7.X版本请使用rabbitmq-autocluster。

创建

1. 创建RBAC授权

假设namesapce为ingress

rbac.yaml

--- apiVersion: v1 kind: ServiceAccount metadata: name: rabbitmq namespace: ingress --- kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: endpoint-reader namespace: ingress rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: endpoint-reader namespace: ingress subjects: - kind: ServiceAccount name: rabbitmq roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: endpoint-reader

创建

kubectl create -f rbac.yaml

2. 创建configmap

configmap.yaml

--- apiVersion: v1 kind: ConfigMap metadata: name: rabbitmq-config namespace: ingress data: enabled_plugins: | [rabbitmq_management,rabbitmq_peer_discovery_k8s]. rabbitmq.conf: | ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more. cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s cluster_formation.k8s.host = kubernetes.default.svc.cluster.local ## Should RabbitMQ node name be computed from the pod's hostname or IP address? ## IP addresses are not stable, so using [stable] hostnames is recommended when possible. ## Set to "hostname" to use pod hostnames. ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME ## environment variable. cluster_formation.k8s.address_type = hostname ## How often should node cleanup checks run? cluster_formation.node_cleanup.interval = 30 ## Set to false if automatic removal of unknown/absent nodes ## is desired. This can be dangerous, see ## * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup ## * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ cluster_formation.node_cleanup.only_log_warning = true cluster_partition_handling = autoheal ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality queue_master_locator=min-masters ## See https://www.rabbitmq.com/access-control.html#loopback-users loopback_users.guest = false cluster_formation.randomized_startup_delay_range.min = 0 cluster_formation.randomized_startup_delay_range.max = 2 # 必须设置service_name，否则Pod无法正常启动，这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量 cluster_formation.k8s.service_name = rabbitmq-cluster # 必须设置hostname_suffix，否则节点不能成为集群，同时保证namespace正确 cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.ingress.svc.cluster.local # 内存上限 vm_memory_high_watermark.absolute = 1GB # 硬盘上限 disk_free_limit.absolute = 2GB

rabbitmq_peer_discovery_k8s参数解析：

• cluster_formation.peer_discovery_backend：对待发现后台插件

• cluster_formation.k8s.host：k8s服务主机(默认即可)

• cluster_formation.k8s.address_type：基于IP/Hostname

• cluster_formation.node_cleanup.interval：清理节点检查时间间隔

创建

kubectl create -f configmap.yaml

3. 创建headless service

service.yaml

--- kind: Service apiVersion: v1 metadata: namespace: ingress name: rabbitmq-cluster labels: app: rabbitmq-cluster type: LoadBalancer spec: clusterIP: None # 由于使用DNS访问Pod需Pod和Headless service启动之后才能访问，publishNotReadyAddresses设置成true，防止readinessProbe在服务没启动时找不到DNS publishNotReadyAddresses: true ports: - name: http protocol: TCP port: 15672 targetPort: 15672 - name: amqp protocol: TCP port: 5672 targetPort: 5672 selector: app: rabbitmq

创建

kubectl create -f service.yaml

4. 创建statefulset

statefulset.yaml

--- apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: rabbitmq namespace: ingress spec: serviceName: rabbitmq-cluster replicas: 3 template: metadata: labels: app: rabbitmq spec: serviceAccountName: rabbitmq terminationGracePeriodSeconds: 10 containers: - name: rabbitmq image: 192.168.100.100/library/rabbitmq:3.7.4 volumeMounts: - name: config-volume mountPath: /etc/rabbitmq - name: rabbitmq-data mountPath: /var/lib/rabbitmq ports: - name: http protocol: TCP containerPort: 15672 - name: amqp protocol: TCP containerPort: 5672 livenessProbe: exec: command: ["rabbitmqctl", "status"] initialDelaySeconds: 60 # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations. periodSeconds: 60 timeoutSeconds: 15 readinessProbe: exec: command: ["rabbitmqctl", "status"] initialDelaySeconds: 20 periodSeconds: 60 timeoutSeconds: 10 imagePullPolicy: Always env: - name: HOSTNAME valueFrom: fieldRef: fieldPath: metadata.name - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: RABBITMQ_USE_LONGNAME value: "true" # See a note on cluster_formation.k8s.address_type in the config file section - name: RABBITMQ_NODENAME value: "rabbit@$(HOSTNAME).rabbitmq-cluster.$(NAMESPACE).svc.cluster.local" # 若在ConfigMap中设置了service_name，则此处无需再次设置 # - name: K8S_SERVICE_NAME # value: "rabbitmq-cluster" - name: RABBITMQ_ERLANG_COOKIE value: "mycookie" volumes: - name: config-volume configMap: name: rabbitmq-config items: - key: rabbitmq.conf path: rabbitmq.conf - key: enabled_plugins path: enabled_plugins volumeClaimTemplates: - metadata: name: rabbitmq-data labels: name: rabbitmq-data spec: storageClassName: managed-nfs-storage accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 100Mi

创建

kubectl create -f statefulset.yaml

等待所有Pod全是Running状态后，登陆到pod内部查询集群信息

[root@k8s-node1 services]# kubectl -n ingress exec -it rabbitmq-0 bash root@rabbitmq-0:/# rabbitmqctl cluster_status Cluster status of node [e-mail] ... [{nodes, [{disc, ['[e-mail]', '[e-mail]', '[e-mail]']}]}, {running_nodes, ['[e-mail]', '[e-mail]', '[e-mail]']}, {cluster_name, <<"[e-mail]">>}, {partitions,[]}, {alarms, [{'[e-mail]',[]}, {'[e-mail]',[]}, {'[e-mail]',[]}]}]

5. 创建ingress

ingress.yaml

apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: "ingress" name: rabbitmq-ingress namespace: ingress spec: rules: - host: rabbitmq-ingress.test.com http: paths: - backend: serviceName: rabbitmq-cluster servicePort: 15672 path: /

创建

kubectl create -f ingress.yaml

登录http://rabbitmq-ingress.test.com:15672(保证域名解析正确)即可访问web管理页面（默认用户密码均为guest）

6. 节点伸缩

可以直接使用scale命令行来伸缩工作节点：

kubectl scale statefulset rabbitmq --replicas=4 -n ingress