filebeat修改自带modules模版

Posted 2022-12-10 MMMMMMMMMMemory

Filebeat针对流行应用集成了大量的module，方便直接拿来使用收集日志比如

[root@ecs-nginx2 ingest]# filebeat modules list
Enabled:
nginx

Disabled:
activemq
apache
auditd
aws
awsfargate
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
crowdstrike
cyberark
cyberarkpas
cylance
elasticsearch
envoyproxy
f5
fortinet
gcp
google_workspace
googlecloud
gsuite
haproxy
ibmmq
icinga
iis
imperva
infoblox
iptables
juniper
kafka
kibana
logstash
microsoft
misp
mongodb
mssql
mysql
mysqlenterprise
nats
netflow
netscout
o365
okta
oracle
osquery
panw
pensando
postgresql
proofpoint
rabbitmq
radware
redis
santa
snort
snyk
sonicwall
sophos
squid
suricata
system
threatintel
tomcat
traefik
zeek
zookeeper
zoom
zscaler

这里可以看到支持的module和已经启用的module，但是有时候默认的模版不能完全满足需求，就得自己修改达到需要，比如我已经自定义了nginx-access.log的日志输出格式为如下，按默认模版会无法解析

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"'
                      ' "$upstream_addr" "$upstream_response_time" "$request_time" "$http_host"';

如果使用yum或者rpm安装的filebeat，module的路径在/usr/share/filebeat/module/下，以我当前需要修改的nginx的access日志为例路径是/usr/share/filebeat/module/nginx/access/ingest/pipeline.yml

将grok字段改为这样，可以对应上修改后的日志

- grok:
    field: event.original
    patterns:
    - (%NGINX_HOST )?"?(?:%NGINX_ADDRESS_LIST:nginx.access.remote_ip_list|%NOTSPACE:source.address)
      - (-|%DATA:user.name) \\[%HTTPDATE:nginx.access.time\\] "%DATA:nginx.access.info"
      %NUMBER:http.response.status_code:long %NUMBER:http.response.body.bytes:long
      "(-|%DATA:http.request.referrer)" "(-|%DATA:user_agent.original)" "(-|%DATA:nginx.access.xff)" "(-|%DATA:nginx.access.upstream_addr)" "%NUMBER:nginx.access.upstream_response_time:float" "%NUMBER:nginx.access.request_time:float" "%DATA:nginx.access.http_host"

192.168.0.10 - - [18/Nov/2022:03:50:50 +0800] "GET /a/b/c.png HTTP/1.1" 200 93094 "-" "Zabbix" "111.111.111.111" "192.168.0.104:9999" "0.001" "0.008" "a.b.com"

修改完成后在module已经启用的情况下使用命令filebeat setup --pipelines --modules=nginx进行在线更新

如果第一次启用module，使用命令filebeat modules enable nginx