logstash json和rubydebug

Posted 运维工匠实战(如果发现有错误请大家把正确的方法发送给我,方便

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了logstash json和rubydebug相关的知识,希望对你有一定的参考价值。

查看一下agent端的shipper的配置:

# cat logstash_test2.shipper.conf 
input { 
    file { 
        path => ["/apps/logstash/conf/test/test2_log.txt"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
     }
 }
output { 
    stdout { 
        #codec => rubydebug
        codec => json
     }
 }
#这个测试主要是看输出的格式为json的

先简测一下刚配好的shipper:

# ./../bin/logstash -f logstash_test2.shipper.conf -t
Sending Logstashs logs to /apps/logstash/logs which is now configured via log4j2.properties
Configuration OK
[2016-12-08T18:14:27,771][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

可以看到没有报错,接下来启动logstash并指定刚才配置好的配置文件:

# ./../bin/logstash -f logstash_test2.shipper.conf -t
Sending Logstashs logs to /apps/logstash/logs which is now configured via log4j2.properties
Configuration OK
[2016-12-08T18:14:27,771][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[[email protected] conf]# ./../bin/logstash -f logstash_test2.shipper.conf 
Sending Logstashs logs to /apps/logstash/logs which is now configured via log4j2.properties
[2016-12-08T18:19:13,056][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-12-08T18:19:13,085][INFO ][logstash.pipeline        ] Pipeline main started
[2016-12-08T18:19:13,165][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.102Z","@version":"1","host":"ofs1","message":"haha------>","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.113Z","@version":"1","host":"ofs1","message":"haha------>2","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.118Z","@version":"1","host":"ofs1","message":"haha------>3","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.121Z","@version":"1","host":"ofs1","message":"haha------>3","tags":[]}

再看看所监控的log日志的内容:

# cat test/test2_log.txt 
haha------>
haha------>2
haha------>3
haha------>3

发现 这个shipper启动的时候会从头到尾,把配置文件全读一边(这种效里也是从配置文件中配置好的)

再看一下这个配置文件:

# cat logstash_test2.shipper.conf 
input { 
    file { 
        path => ["/apps/logstash/conf/test/test2_log.txt"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
     }
 }
output { 
    stdout { 
        #codec => rubydebug
        codec => json
     }
 }
#要点就是这行sincedb_path =>"/dev/null"了!该参数用来指定sincedb文件名,但是如果我们设置为/dev/null这个linux系统上特殊的空洞文件,
那么logstash每次重启进程的时候,尝试读取sincedb内容,都只会读到空洞,也就可以理解为前不有过运行记录,自然就从初始位置开始读取了!

下面往监控文件里写入内容时,会发生下面变化:

# echo "查看json格式是什么输出-------》">>test/test2_log.txt 

再看一下输出的内容:

# ./../bin/logstash -f logstash_test2.shipper.conf -t
Sending Logstashs logs to /apps/logstash/logs which is now configured via log4j2.properties
Configuration OK
[2016-12-08T18:14:27,771][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[[email protected] conf]# ./../bin/logstash -f logstash_test2.shipper.conf 
Sending Logstashs logs to /apps/logstash/logs which is now configured via log4j2.properties
[2016-12-08T18:19:13,056][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-12-08T18:19:13,085][INFO ][logstash.pipeline        ] Pipeline main started
[2016-12-08T18:19:13,165][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.102Z","@version":"1","host":"ofs1","message":"haha------>","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.113Z","@version":"1","host":"ofs1","message":"haha------>2","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.118Z","@version":"1","host":"ofs1","message":"haha------>3","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.121Z","@version":"1","host":"ofs1","message":"haha------>3","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T11:17:45.060Z","@version":"1","host":"ofs1","message":"查看json格式是什么输出-------》","tags":[]}

修改配置文件:

# cat logstash_test2.shipper.conf 
input { 
    file { 
        path => ["/apps/logstash/conf/test/test2_log.txt"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
     }
 }
output { 
    stdout { 
        codec => rubydebug #查看这种格式的日志输出
        #codec => json
     }
 }

查看日志:

# echo "查看rubydebug格式是什么输出-------》">>test/test2_log.txt 
# ./../bin/logstash -f logstash_test2.shipper.conf 
Sending Logstashs logs to /apps/logstash/logs which is now configured via log4j2.properties
[2016-12-08T19:22:37,214][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-12-08T19:22:37,260][INFO ][logstash.pipeline        ] Pipeline main started
[2016-12-08T19:22:37,338][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
{
          "path" => "/apps/logstash/conf/test/test2_log.txt",
    "@timestamp" => 2016-12-08T11:22:37.290Z,
      "@version" => "1",
          "host" => "ofs1",
       "message" => "haha------>",
          "tags" => []
}
{
          "path" => "/apps/logstash/conf/test/test2_log.txt",
    "@timestamp" => 2016-12-08T11:22:37.299Z,
      "@version" => "1",
          "host" => "ofs1",
       "message" => "haha------>2",
          "tags" => []
}
{
          "path" => "/apps/logstash/conf/test/test2_log.txt",
    "@timestamp" => 2016-12-08T11:22:37.301Z,
      "@version" => "1",
          "host" => "ofs1",
       "message" => "haha------>3",
          "tags" => []
}
{
          "path" => "/apps/logstash/conf/test/test2_log.txt",
    "@timestamp" => 2016-12-08T11:22:37.302Z,
      "@version" => "1",
          "host" => "ofs1",
       "message" => "haha------>3",
          "tags" => []
}
{
          "path" => "/apps/logstash/conf/test/test2_log.txt",
    "@timestamp" => 2016-12-08T11:22:37.303Z,
      "@version" => "1",
          "host" => "ofs1",
       "message" => "查看json格式是什么输出-------》",
          "tags" => []
}
{
          "path" => "/apps/logstash/conf/test/test2_log.txt",
    "@timestamp" => 2016-12-08T11:24:32.415Z,
      "@version" => "1",
          "host" => "ofs1",
       "message" => "查看rubydebug格式是什么输出-------》",
          "tags" => []
}

 


以上是关于logstash json和rubydebug的主要内容,如果未能解决你的问题,请参考以下文章

ELK学习笔记a

ELK 2 – 熟悉配置

windows logstash配置

logstash初体验

logstash收集syslog日志

logstash实战input插件syslog