ELK：访问日志切割

Posted 2022-03-17 龙叔运维

        本文只针对ELK下对访问日志的分割进行讲解（如nginx的acc日志，IIS的W3C日志等等）

        本文只讲方向，具体的语法等知识点可以在官网或者百度查询

        本文以IIS日志为例，日志格式（列头）如下：

date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken

1、filebeat的dissect

优点：简单快捷，在简单应用场景中效率更高

缺点：日志切割场景单一，功能单一

实现：在配置文件中processors下面加上dissect 对日志进行切割

processors:

  - dissect:
      tokenizer: "%date %time %s-ip %cs-method %cs-uri-stem %cs-uri-query %s-port %cs-username %c-ip %cs(User-Agent) %cs(Referer) %sc-status %sc-substatus %sc-win32-status %time-taken"
      field: "message"
      target_prefix: "fields"

2、logstash的grok

优点：能灵活的切割各种日志，有很多功能加工日志

缺点：配置难度较dissect高出不少

实现：在logstash配置文件中input和output之间的filter中进行配置：

因为IIS的日志字段是可以自定义配置的，所以这里配置了多种解析配置

filter 
  grok 
    match => [
         "message","%TIMESTAMP_ISO8601:iis.access.time (?:-|%IPORHOST:destination.address) (?:-|%WORD:http.request.method) (?:-|%NOTSPACE:url.path) (?:-|%NOTSPACE:url.query) (?:-|%NUMBER:destination.port:long) (?:-|%NOTSPACE:user.name) (?:-|%IPORHOST:source.address) (?:-|%NOTSPACE:user_agent.original) (?:-|%NOTSPACE:http.request.referrer) (?:-|%NUMBER:http.response.status_code:long) (?:-|%NUMBER:iis.access.sub_status:long) (?:-|%NUMBER:iis.access.win32_status:long) (?:-|%NUMBER:temp.duration:long)",
         "message","%TIMESTAMP_ISO8601:iis.access.time (?:-|%NOTSPACE:iis.access.site_name) (?:-|%WORD:http.request.method) (?:-|%NOTSPACE:url.path) (?:-|%NOTSPACE:url.query) (?:-|%NUMBER:destination.port:long) (?:-|%NOTSPACE:user.name) (?:-|%IPORHOST:source.address) (?:-|%NOTSPACE:user_agent.original) (?:-|%NOTSPACE:iis.access.cookie) (?:-|%NOTSPACE:http.request.referrer) (?:-|%NOTSPACE:destination.domain) (?:-|%NUMBER:http.response.status_code:long) (?:-|%NUMBER:iis.access.sub_status:long) (?:-|%NUMBER:iis.access.win32_status:long) (?:-|%NUMBER:http.response.body.bytes:long) (?:-|%NUMBER:http.request.body.bytes:long) (?:-|%NUMBER:temp.duration:long)",
         "message","%TIMESTAMP_ISO8601:iis.access.time (?:-|%NOTSPACE:iis.access.site_name) (?:-|%NOTSPACE:iis.access.server_name) (?:-|%IPORHOST:destination.address) (?:-|%WORD:http.request.method) (?:-|%NOTSPACE:url.path) (?:-|%NOTSPACE:url.query) (?:-|%NUMBER:destination.port:long) (?:-|%NOTSPACE:user.name) (?:-|%IPORHOST:source.address) (?:-|HTTP/%NUMBER:http.version) (?:-|%NOTSPACE:user_agent.original) (?:-|%NOTSPACE:iis.access.cookie) (?:-|%NOTSPACE:http.request.referrer) (?:-|%NOTSPACE:destination.domain) (?:-|%NUMBER:http.response.status_code:long) (?:-|%NUMBER:iis.access.sub_status:long) (?:-|%NUMBER:iis.access.win32_status:long) (?:-|%NUMBER:http.response.body.bytes:long) (?:-|%NUMBER:http.request.body.bytes:long) (?:-|%NUMBER:temp.duration:long)",
         "message","%TIMESTAMP_ISO8601:iis.access.time \\\\[%IPORHOST:destination.address\\\\]\\\\(http://%IPORHOST:destination.address\\\\) (?:-|%WORD:http.request.method) (?:-|%NOTSPACE:url.path) (?:-|%NOTSPACE:url.query) (?:-|%NUMBER:destination.port:long) (?:-|%NOTSPACE:user.name) \\\\[%IPORHOST:source.address\\\\]\\\\(http://%IPORHOST:source.address\\\\) (?:-|%NOTSPACE:user_agent.original) (?:-|%NUMBER:http.response.status_code:long) (?:-|%NUMBER:iis.access.sub_status:long) (?:-|%NUMBER:iis.access.win32_status:long) (?:-|%NUMBER:temp.duration:long)",
         "message","%TIMESTAMP_ISO8601:iis.access.time (?:-|%IPORHOST:destination.address) (?:-|%WORD:http.request.method) (?:-|%NOTSPACE:url.path) (?:-|%NOTSPACE:url.query) (?:-|%NUMBER:destination.port:long) (?:-|%NOTSPACE:user.name) (?:-|%IPORHOST:source.address) (?:-|%NOTSPACE:user_agent.original) (?:-|%NUMBER:http.response.status_code:long) (?:-|%NUMBER:iis.access.sub_status:long) (?:-|%NUMBER:iis.access.win32_status:long) (?:-|%NUMBER:temp.duration:long)"
    ]
    tag_on_failure => ["fail_in_message"]
  

欢迎关注我的公众号：龙叔运维

持续分享运维经验