Graylog处理docker容器的多行日志之过程记录

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Graylog处理docker容器的多行日志之过程记录相关的知识,希望对你有一定的参考价值。

docker容器虽然支持gelf日志驱动,却不支持合并多行日志为1个message,详情见 log driver should support multiline · Issue #22920 · moby/moby · GitHub

,这导致在graylog查看java应用的报错日志时非常不方便。

 

解决思路:用logstash处理后再发给graylog。

1、docker安装logstash

将 /usr/share/logstash/conf.d/ 目录映射出来,方便编辑配置文件

mkdir -p  /opt/logstash/conf.d/

vi /opt/logstash/logstash.yml

logstash.yml内容如下:

path.config: /usr/share/logstash/conf.d/*.conf
path.logs: /var/log/logstash

 

vi /opt/logstash/conf.d/test.conf

input 
    file
        path => "/usr/share/logstash/conf.d/test.log"
        start_position => "beginning"
        type=>"runtimelog"
                codec=> multiline 
                        pattern => "^%TIMESTAMP_ISO8601 "
                        negate => true
                        what => "previous"
                
	


filter 

output 
	stdout 
        	codec => rubydebug      
	


 

docker run -d -p 5044:5044 -p 5045:5045 -p 12200:12200/udp --name logstash -v /opt/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml -v /opt/logstash/conf.d/:/usr/share/logstash/conf.d/ logstash:7.16.1

 

2、进入容器内安装插件

logstash-plugin install logstash-output-gelf
logstash-plugin install logstash-input-gelf

 安装完插件再添加相关conf

vi /opt/logstash/conf.d/app.conf

input 
    gelf 
	port =>12200
	host => "0.0.0.0"
        codec => multiline 
            pattern => "^%TIMESTAMP_ISO8601 "
            negate => true
            what => "previous"
        
    


filter 

output 
    gelf 
	host => "172.17.0.1"
        port => 12201
        protocol => "UDP"
    


测试结果:input类型为file时,multiline编码正常,input类型为gelf时,无效...

参考链接:docker - logstash-5.x gelf input multiline codec doesn\'t work - Stack Overflow

 

既然logstash行不通,换成fluent-bit试试:

mkdir -p /opt/fluent-bit/

vi /opt/fluent-bit/fluent-bit.conf

[INPUT]
    name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M
    #Multiline	      On
    #Parser_Firstline  multiline_pattern

[OUTPUT]
    Name                    gelf
    Match                   *
    Host                    172.17.0.1
    Port                    12201
    Mode                    udp
    Gelf_Short_Message_Key  log
docker run -d --name fluent -p 24224:24224 -p 24224:24224/udp -v /opt/fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent/fluent-bit:1.8

很遗憾,fluent-bit的input类型为forward时,也不支持Multiline处理... unknown configuration property \'Multiline\'. The following properties are allowed: unix_path, buffer_chunk_size, and buffer_max_size.

参考链接:logging - Does Fluent Bit Input plugin "forward" support multi-line logs processing? - Stack Overflow

以上是关于Graylog处理docker容器的多行日志之过程记录的主要内容,如果未能解决你的问题,请参考以下文章

基于Graylog的容器日志监控

部署 Graylog 日志系统 - 每天5分钟玩转 Docker 容器技术(92)

Graylog2实现Docker容器日志收集

部署 Graylog 日志系统 - 每天5分钟玩转 Docker 容器技术(92)

如何用 Graylog 管理日志?- 每天5分钟玩转 Docker 容器技术(93)

docker容器显示graylog启动成功但无法访问