prometheus-adapter 空项目列表但成功的 HTTP 请求

Posted 2023-02-16

【中文标题】prometheus-adapter 空项目列表但成功的 HTTP 请求

【英文标题】：prometheus-adapter empty item list but successful HTTP request

【发布时间】：2021-12-05 13:44:02

【问题描述】：

我将以下 values.yaml 用于 prometheus-adapter Helm 部署：

rules:
  default: false
  custom:
  - seriesQuery: '__name__=~"go_routines_total"'
    resources:
      overrides:
        job: resource: "namespace"
        queue: resource: "service"
    name:
      matches: ^(.*)
      as: "go_routines_total"
    metricsQuery: go_routines_totaljob=~"rt-scrape"

这些是我通过 kubectl 检查得到的响应：

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq .

  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    
      "name": "namespaces/go_routines_total",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    
  ]

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/go_routines_total" | jq .

  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": 
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/%2A/metrics/go_routines_total"
  ,
  "items": []

将 prometheus-adapter 的详细程度增加到 10 后，我可以确认对 Prometheus 的 HTTP 请求成功并且响应中实际存在数据（162）：

I1018 13:33:05.254264       1 api.go:76] GET http://**.**.**.**:80/api/v1/query?query=go_routines_total%7Bjob%3D~%22rt-scrape%22%7D&time=1634563985.252&timeout= 200 OK                                                                                                        

I1018 13:33:05.254351       1 api.go:95] Response Body: "status":"success","data":"resultType":"vector","result":["metric":"__name__":"go_routines_total","instance":"**.**.**.**:80","job":"rt-scrape","value":[1634563985.252,"162"]]                               I1018 13:33:05.254743       1 httplog.go:104] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/%2A/metrics/go_routines_total" latency="21.148501ms" userAgent="kubectl/v1.21.3 (darwin/amd64) kubernetes/ca643a4" audit-ID="cae29c7f-6915-4b49-aee3-164bbfdede9e" srcIP="10.132.0.189:44084" resp=200

我不确定为什么 kubectl 命令的“items”下没有列出“go_routines_total”（162）的值。任何帮助将不胜感激。

【参考方案1】：

我能够通过向 prometheus-operator 添加自定义标签来解决此问题：

prometheus: 
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: 
    additionalScrapeConfigs:
      - job_name: 'scrape-job'
        metrics_path: '/api/v1/metrics'
        scrape_interval: 5s
        static_configs:
          - targets: ['http://**']
            labels:
              namespace: 'project'
              service: 'app-svc'