fluentd 推送 mariadb audit log

Posted 2020-12-26 运维从入门到放弃

说明:

mariadb audit log是 mariadb 的审计日志

目的是把日志拆分成 tab 键分隔的字段

 

直接附上 fluentd 配置文件

<system>
  log_level error
</system>
<source>
  @type tail
  path /data/logs/mariadb/server_audit.log
  tag mysql_audit
  pos_file /data/logs/mariadb/fluentd.pos
  <parse>
    @type multiline
    format_firstline /^d{8}/
    format1 /^(?<dt>d{8} d{2}:d{2}:d{2}),(?<hostname>[^,]+),(?<user>[^,]+),(?<ip>[^,]+),(?<connid>[^,]+),(?<queryid>[^,]+),(?<action>[^,]+),(?<db>[^,]+),(?<message>.*),(?<retcode>d+)$/
  </parse>
</source>

<filter mysql_audit>
  @type grep
  <regexp>
    key action
    pattern QUERY
  </regexp>
  <exclude>
    key user
    pattern lagou_status
  </exclude>
  <exclude>
    key db
    pattern information_schema
  </exclude>
</filter>

<filter mysql_audit>
  @type record_transformer
  enable_ruby
  <record>
    message ${record["message"].gsub(/s/, ‘ ‘)}
    message ${record["message"].gsub(/s+/, ‘ ‘)}
  </record>
</filter>

<match mysql_audit>
  #@type stdout
  @type file
  path "/data/logs/mariadb/#{ENV[‘HOSTNAME‘]}"
  time_slice_format %Y%m%d%H
  time_slice_wait 5m
  add_path_suffix false
  append true
  compress gzip
  <format>
    @type csv
    fields dt,hostname,user,ip,action,db,message,retcode
    delimiter ‘    ‘
  </format>
</match>

fluentd 比 logstash 内存占用大大下降

分析同样的日志 logstash 占用700M, fluentd 占用35M

不过 cpu 占用相当,对于日志量大的机器 cpu 到100%

看来对日志做正则过滤很损耗 cpu