第三章 Promethus监控服务

Posted 2021-05-21 陪你去流浪

一、概述

普罗米修斯监控分为两种：
	1、携带metric接口的服务
	2、不携带metric接口的服务

普罗米修斯监控携带metric接口的服务的流程:
	1、通过EndPrints获取需要监控的ETCD的地址
	2、创建Service，给予集群内部的ServiceMoniter使用
	3、创建ServiceMoniter部署需要访问证书
	4、重启普罗米修斯监控Pod,载入监控项

二、监控携带metrics接口服务

携带metric接口的服务就表示可以通过metric接口获取服务的监控项和监控信息。本次课以ETCD作为载体。

1.测试ETCD服务的metrics接口

curl -k --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key https://127.0.0.1:2379/metrics

2.通过普罗米修斯监控ETCD

普罗米修斯监控携带metric接口的服务的流程：

​	1、通过EndPrints获取需要监控的ETCD的地址

​	2、创建Service，给予集群内部的ServiceMoniter使用

​	3、创建ServiceMoniter部署需要访问证书，给予prometheus-k8s-0来使用

​	4、重启普罗米修斯监控Pod（prometheus-k8s-0）,载入监控项

因为ETCD是携带metric接口的服务，所以会用上述流程。

1）通过EndPrints获取需要监控的ETCD的地址

kind: Endpoints
apiVersion: v1
metadata:
  namespace: kube-system
  name: etcd-moniter
  labels:
    k8s: etcd
subsets:
  - addresses:
      - ip: "192.168.12.50"
    ports:
      - port: 2379
        protocol: TCP
        name: etcd

• 创建结果

[root@kubernetes-master-01 etcd]# kubectl get endpoints -n kube-system 
NAME                   ENDPOINTS                        AGE
etcd-moniter           192.168.12.50:2379              7m24s

2）创建Service，给予集群内部的ServiceMoniter使用

kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: etcd-moniter
  labels:
    k8s: etcd
spec:
  ports:
    - port: 2379
      targetPort: 2379
      name: etcd
      protocol: TCP

• 创建的结果

[root@kubernetes-master-01 etcd]# kubectl get svc -n kube-system 
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
etcd-moniter   ClusterIP   10.101.187.75   <none>        2379/TCP              6m5s

3）创建ServiceMoniter部署需要访问证书

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    k8s: etcd
  name: etcd-monitor
  namespace: monitoring
spec:
  endpoints:
  - interval: 3s
    port: etcd
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
      certFile: /etc/prometheus/secrets/etcd-certs/peer.crt
      keyFile: /etc/prometheus/secrets/etcd-certs/peer.key
      insecureSkipVerify: true
  selector:
    matchLabels:
      k8s: etcd
  namespaceSelector:
    matchNames:
      - "kube-system"

• 创建的结果

[root@kubernetes-master-01 etcd]# kubectl get ServiceMonitor -n monitoring 
NAME                      AGE
etcd-monitor              22s

4）重启普罗米修斯监控Pod（prometheus-k8s-0）,载入监控项

kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
      - name: alertmanager-main-etcd
        namespace: kube-system
        port: etcd
  image: quay.io/prometheus/prometheus:v2.15.2
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.15.2
  secrets:
    - etcd-certs

• 创建一个secrets，用来保存prometheus监控的etcd的证书

[root@kubernetes-master-01 ~]# kubectl create secret generic etcd-certs -n monitoring --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/peer.crt --from-file=/etc/kubernetes/pki/etcd/peer.key

• 创建的结果

[root@kubernetes-master-01 etcd]# kubectl apply -f prometheus-k8s.yaml 
prometheus.monitoring.coreos.com/k8s created
[root@kubernetes-master-01 etcd]# kubectl get pods -n monitoring 
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-k8s-0                       2/3     Running   1          7s
prometheus-k8s-1                       3/3     Running   1          7s

5）测试是否监控成功

6）加入Grafana