prometheus alertmanager 规则的自动标签

Posted

技术标签:

【中文标题】prometheus alertmanager 规则的自动标签【英文标题】:automatic labels to prometheus alertmanager rules 【发布时间】:2021-11-23 23:34:17 【问题描述】:

我正在使用prometheus-community/prometheus 图表

我想将以下标签自动添加到任何警报管理器规则触发

env=产品 cluster=project-prod-eks

这样我就不会手动将这些标签添加到每个警报规则中。

 - alert: nginxTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requestsstatus=~"4.+" ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
     env: prod
     cluster: project-prod-eks              <---------------HOW to inject them?
   annotations:
     description: Too many 4XXs
     summary: More than 5% of all requests returned 4XX, this requires your attention

这样我就可以做类似的事情

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requestsstatus=~"4.+" ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
   annotations:
     description: Too many 4XXs on  $labels.env  /  $labels.cluster   <----- THIS
     summary: More than 5% of all requests returned 4XX, this requires your attention

有什么想法吗?

【问题讨论】:

【参考方案1】:

您可以将external_labels 添加到您的prometheus.yml:

global:
  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    env: prod
    cluster: project-prod-eks

社区图表在values.yml:

serverFiles:
  prometheus.yml:
    global:
      external_labels:
        foo: bar
    ...

【讨论】:

【参考方案2】:

所以我的做法与修改“serverFiles”略有不同,见下文

server:
  nodeSelector: 
    prometheus: "true"
  baseURL: "https://prometheus.project.io"
  enabled: true
  retention: "30d"
  strategy:
    type: RollingUpdate
  global:
    scrape_interval: 30s
    external_labels:
      env: prod
      client: client-name
      project: project-name
      cluster: project-prod-eks

【讨论】:

以上是关于prometheus alertmanager 规则的自动标签的主要内容,如果未能解决你的问题,请参考以下文章

linux(centos8):prometheus使用alertmanager发送报警邮件(prometheus 2.18.1/alertmanager 0.20.0)

Prometheus+alertmanager告警配置-2

Prometheus配合Alertmanager报警系统

Prometheus配置alertmanager告警邮件通知

2, Prometheus之部署Alertmanager

小姐姐带你入门Alertmanager与Prometheus告警规则