filebeat

Posted 2021-01-04 xiaobaozi-95

 filebeat用于收集和转发日志。filebeat监视指定的日志文件和位置，收集日志事件，并将它们转发到es或logstash等。

配置说明：

input ：

#============== Filebeat prospectors ===========
filebeat.inputs:   # 6.3以前是 filebeat.prospectors:
- type: log        # 默认为log  。6.0以前配置是 - input_type: log
  paths:
    - /var/log/java/test-java.log
    - /var/log/java/*.log
    - /var/log/*/*.log
  fields:          
    log_topic: java_log  #附加的可选字段,以向output添加额外的信息
  multiline.pattern: ‘^[‘   #多行合并 同logstash的multiline用法
  multiline.negate: true
  multiline.match: after
 
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  fields:
    log_topic: nginx_log

 

processor处理器配置

#====================Preocessors=====================
processors:
  - drop_fields:    #删除字段
      fields: ["beat.hostname","beat.version","beat.name","prospector.type","input.type"]


也可以在logstash里面通过remove_field 删除字段
mutate{
        remove_field => ["[beat][hostname]","[beat][version]","[input][type]","[prospector][type]"]
    }

output 输出配置

#================================ Outputs =====================================
#-------------------------Kafka output ------------------------------
output.kafka:
  hosts: ["localhost:9092"]
  topic: ‘%{[fields.log_topic]}‘
  partition.round_robin:
  reachable_only: false
  version: 0.10.0.0
  required_acks: 1
  compression: none
  max_message_bytes: 1000000
#output.logstash:
#  hosts: ["localhost:5044"]