Geoip Logstash 过滤器

Posted 2023-02-23

【中文标题】Geoip Logstash 过滤器

【英文标题】：Geoip logstash filter

【发布时间】：2015-09-04 02:44:35

【问题描述】：

我有一个这样的配置文件：

input 
  file 
      path => "/home/kibana/Documents/external_noise.log"
      type => "external_noise"
      start_position => "beginning"
      sincedb_path => "/dev/null"
  

filter 

grok 
match =>  'message' => '%CISCOTIMESTAMP:timestamp %WORD:action%SPACE%DATA:logsource %DATA:interface %GREEDYDATA:kvpairs' 

     


kv   
source => "kvpairs"
field_split => ";"
value_split => ":"
remove_field => "kvpairs"

mutate 
    remove_field => [ "message" ]


geoip 

source => "src"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%[geoip][longitude]" ]
add_field => [ "[geoip][coordinates]", "%[geoip][latitude]" ]


 mutate 
convert => [ " [geoip][coordinates]", "float"]



date 
    match => [ "timestamp" , "MMM dd HH:mm:ss" ]
    target => "@timestamp"
    

if "_grokparsefailure" in [tags] 
    drop 
    




output 

stdout 
    codec => rubydebug
        
elasticsearch 
    action => "index"
    host => "localhost"
    index => "external-%+dd.MM.YYYY"
    workers => 1

我的示例日志文件如下：

Jan 1 22:54:17 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 70.77.116.190; dst: %DSTIP%; proto: tcp; product: ***-1 & FireWall-1; service: 445; s_port: 2612;
Jan 1 22:54:22 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 61.164.41.144; dst: %DSTIP%; proto: udp; product: ***-1 & FireWall-1; service: 5060; s_port: 5069;
Jan 1 22:54:23 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 69.55.245.136; dst: %DSTIP%; proto: tcp; product: ***-1 & FireWall-1; service: 445; s_port: 2970;
Jan 1 22:54:41 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 95.104.65.30; dst: %DSTIP%; proto: tcp; product: ***-1 & FireWall-1; service: 445; s_port: 2565;
Jan 1 22:54:43 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 222.186.24.11; dst: %DSTIP%; proto: tcp; product: ***-1 & FireWall-1; service: 2967; s_port: 6000;
Jan 1 22:54:54 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 74.204.108.202; dst: %DSTIP%; proto: udp; product: ***-1 & FireWall-1; service: 137; s_port: 53038;
Jan 1 22:55:10 drop   %LOGSOURCE% >eth1 rule: 7; rule_uid: C1336766-9489-4049-9817-50584D83A245; src: 71.111.186.26; dst: %DSTIP%; proto: tcp; product: ***-1 & FireWall-1; service: 445; s_port: 38548;

我尝试在 Kibana 上可视化我的 geoip，但它显示没有找到任何结果。我的geoip 配置有问题吗？我下载了数据库，对此没有任何问题。但似乎geoip 无法读取我存储ip 地址的src？另外，我扩展了字段表。我没有看到一些新的geoip 字段包含有关src IP 地址的信息被映射到真实地理位置。..

需要一些帮助的男孩

【参考方案1】：

您遇到的唯一问题是您的 kv 过滤器由于空格而无法正确拆分字段。

现在，当 logstash 解析您的日志时，您会收到如下事件：

      "@version" => "1",
    "@timestamp" => "2015-01-01T22:15:13.000Z",
          "host" => "iMac-de-Consulthys.local",
          "path" => "/home/kibana/Documents/external_noise.log",
          "type" => "external_noise",
     "timestamp" => "Jan 1 23:15:13",
        "action" => "drop",
     "logsource" => "%LOGSOURCE%",
     "interface" => ">eth1",
          " rule" => " 7",
      " rule_uid" => " C1336766-9489-4049-9817-50584D83A245",
           " src" => " 218.8.245.123",
           " dst" => " %DSTIP%",
         " proto" => " tcp",
       " product" => " ***-1&FireWall-1",
       " service" => " 2967",
        " s_port" => " 6000",

您会注意到kv 过滤器提取的所有字段的开头都有一个空格。这意味着geoip 过滤器找不到src 字段。

因此，您所要做的就是修改您的 kv 过滤器以修剪您的键和值，如下所示：

kv   
    source => "kvpairs"
    field_split => ";"
    value_split => ":"
    trim => "\s"                 <--- add this line
    trimkey => "\s"              <--- add this line
    remove_field => "kvpairs"

然后，您将使用正确创建的 geoip 字段获得不错的事件，如下所示：

      "@version" => "1",
    "@timestamp" => "2015-01-01T22:15:13.000Z",
          "host" => "iMac-de-Consulthys.local",
          "path" => "/home/kibana/Documents/external_noise.log",
          "type" => "external_noise",
     "timestamp" => "Jan 1 23:15:13",
        "action" => "drop",
     "logsource" => "%LOGSOURCE%",
     "interface" => ">eth1",
          "rule" => "7",
      "rule_uid" => "C1336766-9489-4049-9817-50584D83A245",
           "src" => "218.8.245.123",
           "dst" => "%DSTIP%",
         "proto" => "tcp",
       "product" => "***-1&FireWall-1",
       "service" => "2967",
        "s_port" => "6000",
         "geoip" => 
                      "ip" => "218.8.245.123",
           "country_code2" => "CN",
           "country_code3" => "CHN",
            "country_name" => "China",
          "continent_code" => "AS",
             "region_name" => "08",
               "city_name" => "Harbin",
                "latitude" => 45.75,
               "longitude" => 126.64999999999998,
                "timezone" => "Asia/Harbin",
        "real_region_name" => "Heilongjiang",
                "location" => [
            [0] 126.64999999999998,
            [1] 45.75
        ],
             "coordinates" => [
            [0] 126.64999999999998,
            [1] 45.75
        ]
    

【讨论】：

谢谢！！有效。我看到了“src”，我试图把我的来源作为“src”，但出现了错误。我真的很感激谢谢！！

很高兴它有帮助！最重要的是始终尽快修复您的数据，以防止垃圾侵入并导致问题。