生产者和消费者的 Kafka Metrics 导出器

Posted 2023-02-16

【中文标题】生产者和消费者的 Kafka Metrics 导出器

【英文标题】：Kafka Metrics exporter for producer and consumer

【发布时间】：2021-10-31 20:54:04

【问题描述】：

我安装了一个包含 3 个代理的 Cloudera CDP 集群。我使用 Prometheus JMX 导出器导出指标。但是，我不能在 Prometheus 中使用生产者和消费者指标。这些客户端指标是否必须由生产者和消费者导出？

Kafka 代理的指标可用。有什么遗漏或者可以指出我正确的方向吗？

我的 Kafka 导出器 YAML 如下所示：

#https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
lowercaseOutputName: true

rules:
# Special cases and very specific rules
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
- pattern : kafka.coordinator.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_coordinator_$1_$2_$3
  type: GAUGE


- pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
  name: kafka_server_$1_connections_tls_info
  type: GAUGE
  labels:
    listener: "$2"
    networkProcessor: "$3"
    protocol: "$4"
    cipher: "$5"
- pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
  name: kafka_server_$1_connections_software
  type: GAUGE
  labels:
    clientSoftwareName: "$2"
    clientSoftwareVersion: "$3"
    listener: "$4"
    networkProcessor: "$5"
- pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
  name: kafka_server_$1_$4
  type: GAUGE
  labels:
    listener: "$2"
    networkProcessor: "$3"
- pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
  name: kafka_server_$1_$4
  type: GAUGE
  labels:
    listener: "$2"
    networkProcessor: "$3"

# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER

- pattern: kafka.server<type=(.+), client-id=(.+)><>([a-z-]+)
  name: kafka_server_quota_$3
  type: GAUGE
  labels:
    resource: "$1"
    clientId: "$2"

- pattern: kafka.server<type=(.+), user=(.+), client-id=(.+)><>([a-z-]+)
  name: kafka_server_quota_$4
  type: GAUGE
  labels:
    resource: "$1"
    user: "$2"
    clientId: "$3"

# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE

# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
    quantile: "0.$8"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    quantile: "0.$6"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    quantile: "0.$4"
  
#Client
- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w*
  name: kafka_producer_$2

- pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w*
  name: kafka_consumer_$2

- pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w*
  name: kafka_consumer_$2
  # "kafka.consumer:type=app-info,client-id=*"
  # "kafka.producer:type=app-info,client-id=*"
- pattern: "kafka.(.+)<type=app-info, client-id=(.+)><>(.+): (.+)"
  value: 1
  name: kafka_$1_app_info
  labels:
    client_type: $1
    client_id: $2
    $3: $4
  type: UNTYPED

- pattern: "kafka.(.+)<type=(.+), (.+)=(.+), (.+)=(.+), (.+)=(.+)><>(.+):"
  name: kafka_$1_$2_$9
  type: GAUGE
  labels:
    client_type: $1
    $3: "$4"
    $5: "$6"
    $7: "$8"
- pattern: "kafka.(.+)<type=(.+), (.+)=(.+), (.+)=(.+)><>(.+):"
  name: kafka_$1_$2_$7
  type: GAUGE
  labels:
    client_type: $1
    $3: "$4"
    $5: "$6"
  # "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*"
  # "kafka.consumer:type=consumer-metrics,client-id=*"
  # "kafka.producer:type=producer-metrics,client-id=*"
- pattern: "kafka.(.+)<type=(.+), (.+)=(.+)><>(.+):"
  name: kafka_$1_$2_$5
  type: GAUGE
  labels:
    client_type: $1
    $3: "$4"
- pattern: "kafka.(.+)<type=(.+)><>(.+):"
  name: kafka_$1_$2_$3
  labels:
    client_type: $1

感谢您的帮助！

【参考方案1】：

生产者和消费者必须公开指标端点并作为数据源添加到 Prometheus 配置中。

查看监控 Spring boot 应用的演示here。

另一种选择是为生产者和消费者构建自定义 Prometheus 导出器，并将导出器的端点作为数据源添加到 Prometheus 配置中。

查看使用自定义开发的导出器 here 的监控应用演示。

【讨论】：

JMX Exporter 或 Spring Actuator 端点应该可以正常工作，为什么要使用自定义端点？

如果我们无法控制生产者和消费者代码

@OneCricketeer JMX Exporter 或 Spring Actuator 端点应该可以正常工作是什么意思？你能更详细地描述一下这个案例吗？

@rok 是的，我无法控制生产者和消费者代码。你的例子和代码可以独立于生产者和消费者添加吗？

@chrootENV 是的，你可以找到代码示例和解释here