初探ELK-logstash使用小结
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了初探ELK-logstash使用小结相关的知识，希望对你有一定的参考价值。
初探ELK-logstash使用小结
2016/9/12
【写在前言】
说起处理日志的手段，大家或多或少都听说过ELK（elasticsearch+logstash+kibana），怎么入门呢？咱们从一个小小的目标开始。
目标：收集nginx日志，集中展示。
不少人对 ELK 的第一印象，容易觉得它这个工具组合似乎挺难上手的，错！只需动手试试就知道啦！
目标分解：
1）熟悉 logstash 的安装和基本操作
2）熟悉 elasticsearch 的安装和基本操作，然后结合 logstash 使用
3）熟悉 kibana 的安装和基本操作，然后结合 elasticsearch 使用
本文主要是带你进入 logstash 的世界，其余内容请参考相关文章（elasticsearch使用小结，kibana使用小结）。
一、安装 1、jdk 和 环境变量 支持jdk-1.7以上，推荐jdk-1.8 在环境变量配置：JAVA_HOME 2、安装 有2种方式下载，推荐缓存rpm包到本地yum源 1）直接使用rpm wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.4.0.noarch.rpm 2）使用yum源 [[email protected] ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch [[email protected] ~]# vim /etc/yum.repos.d/logstash.repo [logstash-2.4] name=Logstash repository for 2.4.x packages baseurl=https://packages.elastic.co/logstash/2.4/centos gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 [[email protected] ~]# yum install logstash [[email protected] ~]# whereis logstash logstash: /etc/logstash /opt/logstash/bin/logstash /opt/logstash/bin/logstash.bat 二、使用 1、命令行测试 [[email protected] ~]# /opt/logstash/bin/logstash -e ‘input { stdin { } } output { stdout {} }‘ hi, let us go（输入） Settings: Default pipeline workers: 4 Pipeline main started 2016-09-12T02:42:59.110Z 0.0.0.0 hi, let us go（输出） why not TRY IT OUT（输入） 2016-09-12T02:43:11.904Z 0.0.0.0 why not TRY IT OUT（输出） （CTRL-D 退出） Pipeline main has been shutdown stopping pipeline {:id=>"main"} 2、使用配置文件 目的：从日志文件中读取数据，输出到另一个文件中来查看。 前提：已经配置了一个nginx服务，生成了以下日志文件： ]# ls /var/log/nginx/ access.log access_www.test.com_80.log error.log error_www.test.com_80.log 首先，我们尝试这样配置 logstash 来收集日志： [[email protected] ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} } } output { file { path => "/tmp/test.log" } } 上面使用到以下插件： file：日志数据的输入和输出 grok：来匹配标准的apache日志格式 【细节延伸】 显然，在3个环节）中，都有改进和调整的地方。 input：使用 filebeat filter：使用其他插件和规则 output：使用ES，redis等 具体请参考： https://www.elastic.co/guide/en/logstash/current/pipeline.html 3、测试配置文件： [[email protected] ~]# service logstash configtest Configuration OK 4、启动服务： [[email protected] ~]# service logstash start 5、测试请求nginx服务，然后观察输出的内容： [[email protected] ~]# cat /tmp/test.log 符合预期。 6、比较 去掉 filter 这一节，我们来对比一下 /tmp/test.log 收集到的内容的差异 【使用了 filter 的结果a】 {"message":"10.50.200.219 - - [12/Sep/2016:13:00:03 +0800] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\" \"-\" 0.000 \"-\" \"-\"","@version":"1","@timestamp":"2016-09-12T05:00:04.140Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"0.0.0.0","clientip":"10.50.200.219","ident":"-","auth":"-","timestamp":"12/Sep/2016:13:00:03 +0800","verb":"GET","request":"/","httpversion":"1.1","response":"200","bytes":"13","referrer":"\"-\"","agent":"\"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\""} 【未使用 filter 的结果b】 {"message":"10.50.200.219 - - [12/Sep/2016:13:07:49 +0800] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\" \"-\" 0.000 \"-\" \"-\"","@version":"1","@timestamp":"2016-09-12T05:07:49.917Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"0.0.0.0"} a的内容中，多出来的地方，正是使用了 grok 分析和结构化数据 --------------------------------------------------- Information Field Name ----------- ---------- IP Address clientip User ID ident User Authentication auth timestamp timestamp HTTP Verb verb Request body request HTTP Version httpversion HTTP Status Code response Bytes served bytes Referrer URL referrer User agent agent --------------------------------------------------- 7、改进 Logstash 默认自带了 apache 标准日志的 grok 正则: 如何使用自定义的日志格式呢？ 例如，默认的 nginx 日志是： log_format main ‘$remote_addr - $remote_user [$time_local] "$request" ‘ ‘$status $body_bytes_sent "$http_referer" ‘ ‘"$http_user_agent" "$http_x_forwarded_for"‘; 改成自定义的日志格式： log_format online ‘$remote_addr [$time_local] "$request" ‘ ‘"$http_content_type" "$request_body" "$http_referer" ‘ ‘$status $request_time $body_bytes_sent‘; 对应的数据： 【GET】# curl -H "Content-Type: text/html; charset=UTF-8" --referer ‘www.abc.com/this_is_a_referer‘ http://www.test.com/a/b/c.html?key1=value1 【结果】10.50.200.219 [12/Sep/2016:15:11:04 +0800] "GET /a/b/c.html?key1=value1 HTTP/1.1" "text/html; charset=UTF-8" "-" "www.abc.com/this_is_a_referer" 404 0.000 168 【POST】# curl -H "Content-Type: application/xml" -d "{"name": "Mark Lee" }" "http://www.test.com/start" 【结果】10.50.200.218 [12/Sep/2016:15:02:07 +0800] "POST /start HTTP/1.1" "application/xml" "-" "-" 404 0.000 168 尝试一下： [[email protected] ~]# mkdir -p /etc/logstash/patterns.d [[email protected] ~]# vim /etc/logstash/patterns.d/extra_patterns NGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-) 调整配置为： [[email protected] ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 } } filter { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } } output { file { path => "/tmp/test.log" } } [[email protected] ~]# service logstash restart 结果： {"message":"10.50.200.218 [12/Sep/2016:15:28:23 +0800] \"POST /start HTTP/1.1\" \"application/xml\" \"-\" \"-\" 404 0.000 168","@version":"1","@timestamp":"2016-09-12T07:28:24.007Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"0.0.0.0","clientip":"10.50.200.218","timestamp":"12/Sep/2016:15:28:23 +0800","verb":"POST","request":"/start","httpversion":"1.1","content_type":"\"application/xml\"","request_body":"\"-\"","response":"404","request_time":"0.000","bytes":"168"} {"message":"10.50.200.219 [12/Sep/2016:15:28:24 +0800] \"GET /a/b/c.html?key1=value1 HTTP/1.1\" \"text/html; charset=UTF-8\" \"-\" \"www.abc.com/this_is_a_referer\" 404 0.000 168","@version":"1","@timestamp":"2016-09-12T07:28:25.019Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"0.0.0.0","clientip":"10.50.200.219","timestamp":"12/Sep/2016:15:28:24 +0800","verb":"GET","request":"/a/b/c.html?key1=value1","httpversion":"1.1","content_type":"\"text/html; charset=UTF-8\"","request_body":"\"-\"","referrer":"\"www.abc.com/this_is_a_referer\"","response":"404","request_time":"0.000","bytes":"168"} 符合预期。 三、输出到 redis+elasticsearch+kibana 1、测试环境（已经部署了服务） 【客户端】10.50.200.49: logstash, nginx(www.test.com, www.work.com) 【服务端】10.50.200.220: logstash, redis, elasticsearch, kibana 【测试端】10.50.200.218, 10.50.200.219: curl 请求 nginx [[email protected] ~]# for i in `seq 1 5000`;do curl -H "Content-Type: application/xml" -d "{"name": "York vm218" }" "http://www.test.com/this_is_vm218";sleep 1s;done [[email protected] ~]# for i in `seq 1 5000`;do curl -H "Content-Type: text/html; charset=UTF-8" --referer ‘www.vm219.com/referer_here‘ http://www.test.com/a/b/c.html?key1=value1;sleep 1s;done hosts文件： 10.50.200.49 www.test.com 10.50.200.49 www.work.com 2、单域名场景 目的：将 www.test.com 的 access 日志收集起来集中展示 【客户端】 输入：file 输出：redis [[email protected] ~]# cat /etc/logstash/conf.d/nginx.conf input { file { type => "nginx_access" path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 } } filter { if[type] == "nginx_access" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } } } output { if[type] == "nginx_access" { redis { host => "10.50.200.220" data_type => "list" key => "logstash:redis:nginxaccess" } } } [[email protected] ~]# service logstash restart 【服务端】 输入：redis 输出：elasticsearch [[email protected] ~]# vim /etc/logstash/conf.d/redis.conf input { redis { host => ‘127.0.0.1‘ data_type => ‘list‘ port => "6379" key => ‘logstash:redis:nginxaccess‘ type => ‘redis-input‘ } } output { if[type] == "nginx_access" { elasticsearch { hosts => "127.0.0.1:9200" index => "nginxaccess-%{+YYYY.MM.dd}" } } } [[email protected] ~]# service logstash restart 可以通过命令行去观察 redis 的状态： [[email protected] ~]# redis-cli monitor 结果：符合预期。 3、多域名场景 目的：将 www.test.com 和 www.work.com 的 access 日志收集起来集中展示 【客户端】 输入：file 输出：redis [[email protected] ~]# cat /etc/logstash/conf.d/nginx.conf input { file { type => "nginx_access_www.test.com" path => "/var/log/nginx/access_www.test.com*.log" start_position => beginning ignore_older => 0 } file { type => "nginx_access_www.work.com" path => "/var/log/nginx/access_www.work.com*.log" start_position => beginning ignore_older => 0 } } filter { if[type] =~ "nginx_access" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } } } output { if[type] =~ "nginx_access" { redis { host => "10.50.200.220" data_type => "list" key => "logstash:redis:nginxaccess" } } } [[email protected] ~]# service logstash restart 【服务端】 输入：redis 输出：elasticsearch [[email protected] ~]# vim /etc/logstash/conf.d/redis.conf input { redis { host => ‘127.0.0.1‘ data_type => ‘list‘ port => "6379" key => ‘logstash:redis:nginxaccess‘ type => ‘redis-input‘ } } output { if[type] == "nginx_access_www.test.com" { elasticsearch { hosts => "127.0.0.1:9200" index => "nginxaccess-www.test.com-%{+YYYY.MM.dd}" } } else if[type] == "nginx_access_www.work.com" { elasticsearch { hosts => "127.0.0.1:9200" index => "nginxaccess-www.work.com-%{+YYYY.MM.dd}" } } } [[email protected] ~]# service logstash restart 当然了，要调整kibana的索引名称。 结果：符合预期。 四、小结 1、数据流向 ------------------------------------------------------------------ log_files -> logstash -> redis -> elasticsearch -> kibana ------------------------------------------------------------------ 2、TODO 1）filebeat 的使用 2）redis 是否被替换？ 3）Elasticsearch索引数据的清理 4）kibana的权限 5）ELK的性能和监控 ZYXW、参考 1、官网 https://www.elastic.co/guide/en/logstash/current/introduction.html https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html https://www.elastic.co/guide/en/logstash/current/installing-logstash.html https://www.elastic.co/guide/en/logstash/current/first-event.html https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html https://www.elastic.co/guide/en/logstash/current/pipeline.html https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html 2、ELK中文 http://kibana.logstash.es/content/ http://kibana.logstash.es/content/beats/file.html 3、用ELK搭建简单的日志收集分析系统 http://blog.csdn.net/lzw_2006/article/details/51280058
以上是关于初探ELK-logstash使用小结的主要内容，如果未能解决你的问题，请参考以下文章