fabric运维监控：prometheus配置热更新

Posted 2021-10-27 孙绿如叶～

在上一篇fabric运维监控：prometheus+grafana框架搭建中搭建了prometheus+grafana监控框架，但这里存在一个问题，所有监控job都提前写好在配置文件prometheus.yml中，后面新增job时，都得手动修改配置文件、重启服务，这在生产中是非常麻烦的，如当用prometheus去监控baas平台时，如何做到prometheus动态监控新增的节点？

Prometheus提供了多种动态服务发现的功能，这里以consul为例介绍动态加载配置文件过程。

1 . 安装consul（docker 方式）

docker pull consul

docker run --name consul -d -p 8500:8500 consul

consul 提供标准API方式注册/注销服务：

//注册服务
 curl -X PUT -d '{"id": "orderer0.ordererOrg.com","name": "orderer0.ordererOrg.com-172.30.12.167","address": "192.168.20.111","port": 8443,"tags": ["test"],"checks": [{"http": "http://192.168.20.111:8443/metrics", "interval": "5s"}]}'  http://192.168.20.84:8500/v1/agent/service/register
 
 //注册增加meta，meta支持自定义标签信息：
 curl -X PUT -d '{"id": "orderer0.ordererOrg.com","name": "orderer0.ordererOrg.com-172.30.12.167",  "Meta": { "app": "spring-boot",    "team": "appgroup",    "project": "bigdata" },"address": "192.168.20.111","port": 8443,"tags": ["test"],"checks": [{"http": "http://192.168.20.111:8443/metrics", "interval": "5s"}]}'  http://192.168.20.84:8500/v1/agent/service/register
 
 //注销服务：
 curl -X PUT http://192.168.20.84:8500/v1/agent/service/deregister/orderer0.ordererOrg.com

2 . 修改 prometheus.yml 文件支持consul 服务发现：relabel_configs中去除consul 服务，支持Meta标签

• keep: 丢弃 source_labels 的值中没有匹配到 regex 正则表达式内容的 Target 实例
• labelmap: 根据 regex 去匹配 Target 实例所有标签的名称（注意是名称），并且将捕获到的内容作为为新的标签名称，regex 匹配到标签的的值作为新标签的值

global:
  scrape_interval:     60s
  evaluation_interval: 60s

scrape_configs:
  - job_name: 'consul-prometheus'
    consul_sd_configs:
    -  server: '192.168.20.84:8500'
    -  services: []
    relabel_configs:
    - source_labels: [__meta_consul_tags]
      regex: .*test.*
      action: keep
    - regex: __meta_consul_service_metadata_(.+)
      action: labelmap

3 . 使用go 向consul动态注册/注销服务

import "github.com/hashicorp/consul/api"

//注册
func Register(consortiumName,nodeName,NodeType string ,port int) error {
	config :=api.DefaultConfig()
	config.Address = ""

	check := api.AgentServiceCheck{
		Interval: "5s",
		HTTP: "",
	} 

	m := make(map[string]string)  //自定义标签信息
	m["consortium"] = consortiumName
	m["NodeType"] = NodeType
	m["NodeName"] = nodeName

	reg := api.AgentServiceRegistration{
		ID: nodeName,
		Name: consortiumName,
		Address: ip,
		Port: port,
		Tags: []string{"baas"},
		Check: &check,
		Meta: m,
	}

	client, err := api.NewClient(config) 
	if err != nil {
		return err
	}

	return client.Agent().ServiceRegister(&reg)
}

//注销
func Deregister(id string) error {
	config :=api.DefaultConfig()
	config.Address = ""

	client, err := api.NewClient(config)
	if err != nil {
		return err
	}

	return client.Agent().ServiceDeregister(id)
}