使用mongodb-exporter+prometheus+grafana实现mongodb集群监控

Posted 2020-11-29

1 创建Mongodb集群

直接使用下面的yaml文件创建mongodb集群，本环境中，只启动了两节点的mongodb。使用hostPath进行数据存储，创建存储目录

[root@master ~]# mkdir -p /home/storage/mongo/db/
[root@master mongo]# cat mongo.yaml
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    app: mongo
spec:
  ports:
  - name: mongo
    port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    app: mongo
---
apiVersion: v1
kind: Service
metadata:
  name: mongo-service
  labels:
    app: mongo
spec:
  ports:
  - name: mongo-http
    port: 27017
  selector:
    app: mongo
  type: NodePort
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  selector:
    matchLabels:
      app: mongo
  serviceName: "mongo"
  replicas: 2
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: mongo
    spec:
      terminationGracePeriodSeconds: 10
      affinity:
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
               matchExpressions:
               - key: "app"
                 operator: In
                 values:
                 - mongo
             topologyKey: "kubernetes.io/hostname"
      containers:
      - name: mongo
        image: mongo:latest
        command:
        - mongod
        - "--bind_ip_all"
        - "--replSet"
        - rs0
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-data
          mountPath: /data/db
      volumes:
      - name: mongo-data
        hostPath:
          path: /home/storage/mongo/db
[root@master mongo]# kubectl apply -f mongo.yaml
[root@master mongo]# kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-74f858558f-rckf4       2/2     Running   6          9d
**mongo-0                           2/2     Running   2          22h
mongo-1                           2/2     Running   2          22h**
productpage-v1-76589d9fdc-vp6h5   2/2     Running   6          9d
ratings-v1-7855f5bcb9-fd98w       2/2     Running   6          9d
reviews-v1-64bc5454b9-knhcs       2/2     Running   6          9d
reviews-v2-76c64d4bdf-wmsgq       2/2     Running   6          9d
reviews-v3-5545c7c78f-hsw5h       2/2     Running   6          9d
[root@master mongo]# kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
details         ClusterIP   10.10.179.23    <none>        9080/TCP          9d
kubernetes      ClusterIP   10.10.0.1       <none>        443/TCP           14d
mongo           ClusterIP   None            <none>        27017/TCP         22h
mongo-service   NodePort    10.10.27.148    <none>        27017:31014/TCP   22h
productpage     ClusterIP   10.10.112.221   <none>        9080/TCP          9d
ratings         ClusterIP   10.10.183.94    <none>        9080/TCP          9d
reviews         ClusterIP   10.10.86.216    <none>        9080/TCP          9d
[root@master mongo]#

如上所示，mongodb集群创建完成

2 测试是否创建成功

使用curl测试mongodb是否正常工作

[root@master mongo]# curl http://192.168.122.7:31014
It looks like you are trying to access MongoDB over HTTP on the native driver port.

出现上面提示，则说明mongodb启动成功

3 准备mongodb-exporter镜像

pull mongodb-exporter镜像

[root@master mongo]# docker pull noenv/mongo-exporter:latest

4 创建mongodb-exporter应用、服务、serviceaccount、servicemonitor

使用mongodb-exporter镜像创建deployment，此镜像作者在创建此镜像时，指定了mongodb-uri的镜像为172.17.0.1，所以在创建deployment时，需要重新指定mongodb的uri地址，还要注意，此deployment需要创建在monitoring namespace下，此namespace为kube-prometheus命名空间，关于kube-prometheus的部署方式，后续文章会出。具体yaml文件如下：

[root@master mongo]# cat mongo-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-exporter
  namespace: monitoring
  labels:
    k8s-app: mongo-exporter
spec:
  selector:
    matchLabels:
      k8s-app: mongo-exporter
  template:
    metadata:
      labels:
        k8s-app: mongo-exporter
    spec:
      containers:
      - name: mongo-exporter
        image: noenv/mongo-exporter:latest
        args: ["--web.listen-address=:9104", "--mongodb.uri", "mongodb://192.168.122.7:31014"]
        ports:
        - containerPort: 9104
          name: http

创建deployment

[root@master mongo]# kubectl apply -f mongo-exporter.yaml
deployment.apps/mongo-exporter created
[root@master mongo]# kubectl get pod -n monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
alertmanager-main-0                   2/2     Running   12         14d
alertmanager-main-1                   2/2     Running   12         14d
alertmanager-main-2                   2/2     Running   12         14d
grafana-77978cbbdc-szzbd              1/1     Running   6          14d
kube-state-metrics-7f6d7b46b4-wnkn6   3/3     Running   18         14d
**mongo-exporter-7557899dcc-xpzh5       1/1     Running   0          114s**
node-exporter-55259                   2/2     Running   12         14d
node-exporter-rr2bf                   2/2     Running   2          29h
prometheus-adapter-68698bc948-65blv   1/1     Running   6          14d
prometheus-k8s-0                      3/3     Running   19         14d
prometheus-k8s-1                      3/3     Running   19         14d
prometheus-operator-6685db5c6-7dbvj   1/1     Running   6          14d
[root@master mongo]#

创建service

[root@master mongo]# cat mongo-exporter-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: mongo-exporter
  name: mongo-exporter
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - name: http
    port: 9104
    nodePort: 30017
    targetPort: http
  selector:
    k8s-app: mongo-exporter
[root@master mongo]# kubectl create -f mongo-exporter-service.yaml
service/mongo-exporter created
[root@master mongo]# kubectl describe svc -n monitoring mongo-exporter
Name:                     mongo-exporter
Namespace:                monitoring
Labels:                   k8s-app=mongo-exporter
Annotations:              <none>
Selector:                 k8s-app=mongo-exporter
Type:                     NodePort
IP:                       10.10.84.94
Port:                     http  9104/TCP
TargetPort:               http/TCP
NodePort:                 http  30017/TCP
Endpoints:                10.124.72.5:9104
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
[root@master mongo]#

创建serviceaccout

[root@master mongo]# kubectl create serviceaccount -n monitoring mongo-exporter
serviceaccount/mongo-exporter created

创建serviceMonitor

[root@master mongo]# cat mongo-exporter-serviceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: mongo-exporter
  name: mongo-exporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: http
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: mongo-exporter
[root@master mongo]# kubectl apply -f mongo-exporter-serviceMonitor.yaml
servicemonitor.monitoring.coreos.com/mongo-exporter created

5 查看prometheus Targets

以上资源都创建完成后，在prometheus中查看mongodb-exporter Targets

此时prometheus可以正常收集mongodb集群性能数据

6 在grafana中创建监控面板

在grafana页面中选择官方的mongodb模板


点击Import，然后即可实现对mongodb监控

为何有的数据没有，因为：1 mongodb中没有存入数据 2 这个grafana模板中的prometheus-mongodb-exporter语句中有的指标名称不存在。