logstash收集nginx日志

Posted 李建明180

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了logstash收集nginx日志相关的知识，希望对你有一定的参考价值。

创建模版

使用 devtools 创建模板，或者创建的索引以 logstash开头，确保location类型为geo_point。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
PUT _template/nginx
{
	"order": 0,
	"version": 60001,
	"index_patterns": [
		"nginx-*"
	],
	"settings": {
		"index": {
			"number_of_shards": "1",
			"refresh_interval": "5s"
		}
	},
	"mappings": {
		"properties": {
			"@timestamp": {
				"type": "date"
			},
			"geoip": {
				"dynamic": true,
				"properties": {
					"ip": {
						"type": "ip"
					},
					"latitude": {
						"type": "half_float"
					},
					"location": {
						"type": "geo_point"
					},
					"longitude": {
						"type": "half_float"
					}
				}
			},
			"body_bytes": {
				"type": "float"
			},
			"request_time": {
				"type": "float"
			},
			"response_code": {
				"type": "integer"
			},
			"@version": {
				"type": "keyword"
			}
		}
	},
	"aliases": {}
}

nginx 输出 json 格式日志

可以根据自己的需求，添加删除注释

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
log_format json   '{"@timestamp":"$time_iso8601",'
                       '"@source":"$server_addr",'
                       '"hostname":"$hostname",'
                       '"http_host":"$host",'
                       '"client":"$remote_addr",'
                       '"request_method":"$request_method",'
                       '"url":"$uri",'
                       '"http_version":"$server_protocol",'
                       '"scheme":"$scheme"'
                       '"response_code":"$status",'
                       '"body_bytes":"$body_bytes_sent",'
                       '"request_time":"$request_time",'
                       '"referer":"$http_referer",'
大专栏  logstash收集nginx日志pan class="line">                       '"ua":"$http_user_agent",'
                       #'"@version":"1",'
                       #'"args":"$args",'
                       #'"https":"$https",'
                       #'"http_cookie":"$http_cookie",'
                       #'"client_forward":"$http_x_forwarded_for",'
                       #'"host":"$server_addr",'
                       #'"upstream_host":"$upstream_addr"'
                       #'"upstream_status":"$upstream_status"'
                       #'"upstream_response_time":"$upstream_response_time"'
                       '}';

logstash 配置

1
2
3
4
5
6
7
8
9
10
11
12
filter {
   if ([fields][service] == "nginx-elastic-logs") {
       json {
           source => "message"
           remove_field => ["message","agent","ecs"]
       }
       geoip {
           source => "client"
           target => "geoip"
       }
   }
}

输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
"geoip" => {
           "region_name" => "Beijing",
                    "ip" => "123.58.107.118",
        "continent_code" => "AS",
          "country_name" => "China",
         "country_code3" => "CN",
              "location" => {
            "lon" => 116.3883,
            "lat" => 39.9289
        },
              "latitude" => 39.9289,
         "country_code2" => "CN",
           "region_code" => "BJ",
             "longitude" => 116.3883,
              "timezone" => "Asia/Shanghai"
    },

如果觉得信息太多，可以通过 fileds 选项选择自己需要的信息,city_name, continent_code, country_code2, country_code3, country_name, dma_code, ip, latitude, longitude, postal_code, region_name and timezone

修改后的配置

1
2
3
4
5
6
7
8
9
10
11
if ([fields][service] == "nginx-elastic-logs") {
    json {
        source => "message"
        remove_field => ["message","agent","ecs"]
    }
    geoip {
        source => "client"
        target => "geoip"
        fields => ["country_name","region_name","location","continent_code"]
    }
}

输出

1
2
3
4
5
6
7
8
9
"geoip" => {
        "continent_code" => "AS",
          "country_name" => "China",
           "region_name" => "Beijing",
              "location" => {
            "lon" => 116.3883,
            "lat" => 39.9289
        }
    },

以上是关于logstash收集nginx日志的主要内容，如果未能解决你的问题，请参考以下文章

logstash收集nginx日志

ELK之八----Logstash结合kafka收集系统日志和nginx日志

ELK之六-----logstash结合redis收集系统日志和nginx访问日志

Logstash收集nginx日志

Logstash收集nginx日志之使用grok过滤插件解析日志

Logstash收集nginx日志之使用grok过滤插件解析日志