Logstash收集nginx日志之使用grok过滤插件解析日志

Posted huangll99

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Logstash收集nginx日志之使用grok过滤插件解析日志相关的知识,希望对你有一定的参考价值。

grok作为一个logstash的过滤插件,支持根据模式解析文本日志行,拆成字段。

  • nginx日志的配置:
log_format  main  $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for";

 



  • logstash中grok的正则(添加在logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-xxx/patterns/grok-patterns文件中)为:
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}

 

logstash的配置为:

input {
    file {
        path     => ["/var/log/nginx/access.log"]
        type    => "nginxlog"
        start_position => "beginning"
    }
}

filter {  
    grok {  
      match => { "message" => "%{NGINXACCESS}" }
    }  
} 
output {
    stdout {
        codec    => rubydebug
    }
}

logstash的输出:

{
        "message" => "192.168.154.2 - - [30/Mar/2017:01:27:09 -0700] \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"",
       "@version" => "1",
     "@timestamp" => "2017-03-30T08:27:09.539Z",
           "path" => "/var/log/nginx/access.log",
           "host" => "spark4",
           "type" => "nginxlog",
      "remote_ip" => "192.168.154.2",
      "timestamp" => "30/Mar/2017:01:27:09 -0700",
         "method" => "GET",
        "request" => "/index.html",
    "httpversion" => "1.1",
         "status" => "304",
          "bytes" => "0",
        "referer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"",
       "xforward" => "\"-\""
}

 



以上是关于Logstash收集nginx日志之使用grok过滤插件解析日志的主要内容,如果未能解决你的问题,请参考以下文章

logstash实战filter插件之grok(收集apache日志)

logstash之grok

使用logstash收集javanginx系统等常见日志

logstash grok 分析 nginx access 日志

使用Logstash filter grok过滤日志文件

使用Logstash filter grok过滤日志文件