prometheus alertmanager 规则的自动标签
Posted
技术标签:
【中文标题】prometheus alertmanager 规则的自动标签【英文标题】:automatic labels to prometheus alertmanager rules 【发布时间】:2021-11-23 23:34:17 【问题描述】:我正在使用prometheus-community/prometheus 图表
我想将以下标签自动添加到任何警报管理器规则触发
env=产品 cluster=project-prod-eks这样我就不会手动将这些标签添加到每个警报规则中。
- alert: nginxTooMany400s
expr: 100 * ( sum( nginx_ingress_controller_requestsstatus=~"4.+" ) / sum(nginx_ingress_controller_requests) ) > 5
for: 1m
labels:
severity: warning
env: prod
cluster: project-prod-eks <---------------HOW to inject them?
annotations:
description: Too many 4XXs
summary: More than 5% of all requests returned 4XX, this requires your attention
这样我就可以做类似的事情
- alert: NGINXTooMany400s
expr: 100 * ( sum( nginx_ingress_controller_requestsstatus=~"4.+" ) / sum(nginx_ingress_controller_requests) ) > 5
for: 1m
labels:
severity: warning
annotations:
description: Too many 4XXs on $labels.env / $labels.cluster <----- THIS
summary: More than 5% of all requests returned 4XX, this requires your attention
有什么想法吗?
【问题讨论】:
【参考方案1】:您可以将external_labels
添加到您的prometheus.yml:
global:
# The labels to add to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
env: prod
cluster: project-prod-eks
社区图表在values.yml:
serverFiles:
prometheus.yml:
global:
external_labels:
foo: bar
...
【讨论】:
【参考方案2】:所以我的做法与修改“serverFiles”略有不同,见下文
server:
nodeSelector:
prometheus: "true"
baseURL: "https://prometheus.project.io"
enabled: true
retention: "30d"
strategy:
type: RollingUpdate
global:
scrape_interval: 30s
external_labels:
env: prod
client: client-name
project: project-name
cluster: project-prod-eks
【讨论】:
以上是关于prometheus alertmanager 规则的自动标签的主要内容,如果未能解决你的问题,请参考以下文章
linux(centos8):prometheus使用alertmanager发送报警邮件(prometheus 2.18.1/alertmanager 0.20.0)