prometheus 收集kube-state-metrics指标信息

Posted 2021-03-03 gaofeng-henu

安装kube-state-metrics

1. 下载

   $ cd /opt/k8s/work/
   $ wget -O kube-state-metrics.zip  https://github.com/kubernetes/kube-state-metrics/archive/master.zip
   $ unzip kube-state-metrics.tar.gz
   $ cd kube-state-metrics-master/examples/standard
   $ ls
   cluster-role-binding.yaml  cluster-role.yaml  deployment.yaml  service-account.yaml  service.yaml
   

2. 修改namespace

   将默认的namespace从kube-system修改成monitoring

   $ cd /opt/k8s/work/kube-state-metrics-master/examples/standard
   $ ls | xargs -I {} sed -i "s/namespace: kube-system/namespace: monitoring/g" {}
   

3. 在service.yaml追加如下annotation

   annotations:
     prometheus.io/scraped: "true"
     
   

   修改后的server.yml文件

   cat service.yaml
    apiVersion: v1
    kind: Service
    metadata:
      annotations:
        prometheus.io/scraped: "true"
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: 1.9.5
      name: kube-state-metrics
      namespace: monitoring
    spec:
      clusterIP: None
      ports:
      - name: http-metrics
        port: 8080
        targetPort: http-metrics
      - name: telemetry
        port: 8081
        targetPort: telemetry
      selector:
        app.kubernetes.io/name: kube-state-metrics
    
   

4. 启动服务

   $ cd /opt/k8s/work/kube-state-metrics-master/examples/standard
   $ kubectl create -f .
   
   

prometheus 收集指标

1. prometheus 追加配置，仅抓取带有prometheus.io/scraped: "true" 注解的endpoint

   - job_name: "kubernetes-service-endpoints"
     kubernetes_sd_configs:
     - role: endpoints
     relabel_configs:
     - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scraped]
       action: keep
       regex: true
     - action: labelmap
       regex: __meta_kubernetes_service_label_(.+)
     - source_labels: [__meta_kubernetes_namespace]
       action: replace
       target_label: kubernetes_namespace
     - source_labels: [__meta_kubernetes_service_name]
       action: replace
       target_label: service_name
    
   

2. prometheus重新加载配置后，查看对应的target

   

   kube-state-metrics对应的pod有两个端口，在prometheus中显示了两个target

配置grafana

1. 从grafana.com 下载state-metrics监控模版1. Kubernetes Deployment Statefulset Daemonset metrics,导入模版

   grafana效果图