logstash&Kibana杂记

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了logstash&Kibana杂记相关的知识，希望对你有一定的参考价值。

一、logstash基础

master1作为logstash_agent端，master1运行WEB网站。master2为logstash服务端，master3为Elasticsearch

1、安装logstash

1.1 yum安装

设置java环境变量：
[[email protected] ~]# vim /etc/profile.d/java.sh 

export JAVA_HOME=/usr

logstash已经被Elasticsearch收购，直接在ES官网下载即可

[[email protected] ~]# ls 
logstash-1.5.4-1.noarch.rpm

安装
[[email protected] ~]# yum install logstash-1.5.4-1.noarch.rpm

设置环境变量：
[[email protected] ~]# vim /etc/profile.d/logstash.sh

export PATH=/opt/logstash/bin:$PATH

重新加载：   
[[email protected] ~]# source /etc/profile.d/logstash.sh

1.2 创建配置文件

[[email protected] ~]# vim /etc/logstash/conf.d/sample.conf

input {
    stdin {}
}

output {
    stdout {
        codec   => rubydebug
    }
}

语法测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/sample.conf --configtest
Configuration OK

1.3 运行logstash

[[email protected] ~]# logstash -f /etc/logstash/conf.d/sample.conf
Logstash startup completed

测试：
Logstash startup completed
Hello Logstash
{
       "message" => "Hello Logstash",
      "@version" => "1",
    "@timestamp" => "2018-04-15T16:59:04.136Z",
          "host" => "master1.com"
}

2、示例（input、filter插件）

2.1 系统日志文件简单示例

[[email protected] ~]# vim /etc/logstash/conf.d/filesample.conf

input {
    file {
        path => ["/var/log/messages"]
        type => "system"
        start_position => "beginning"
    }
}

output {
    stdout {
        codec   => rubydebug
    }
}

语法测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/filesample.conf --configtest
Configuration OK

运行：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/filesample.conf

技术分享图片

file插件官网链接：
https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-file.html

结束：Ctrl+c

2.2 udp

master2安装 collectd，配置其network插件，向外发送数据。

[[email protected] ~]# yum install collectd

配置collectd
[[email protected] ~]# vim /etc/collectd.conf 

#定义主机名
Hostname    "master2.com"

#打开几个监控项
LoadPlugin cpu
LoadPlugin df
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network

定义发送到logstash端的监听端口
<Plugin network>
    <Server "10.201.106.131" "25826" >
    </Server>
</Plugin>

启动服务
[[email protected] ~]# systemctl start collectd.service
logstash端配置：
[[email protected] ~]# vim /etc/logstash/conf.d/udpsample.conf

input {
    udp {
        port    => 25826
        codec   => collectd {}
        type    => "collectd"
    }
}

output {
    stdout {
        codec   => rubydebug

语法检测：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/udpsample.conf --configtest
Configuration OK

启动：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/udpsample.conf
Logstash startup completed

技术分享图片

2.3 httpd

[[email protected] ~]# yum install httpd
[[email protected] ~]# systemctl start http

结构化文本数据
[[email protected] ~]# rpm -ql logstash | grep "patterns$"
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/mcollective-patterns

[[email protected] ~]# vim /etc/logstash/conf.d/groksample.conf

input {
    stdin {}
}

filter {
    grok {        match   => { "message" => "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
    }
}

output {
    stdout {
        codec   => rubydebug
    }
}

语法测试：
Configuration OK

运行测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/groksample.conf
Logstash startup completed
1.1.1.1 GET /index.html 30 0.23
{
       "message" => "1.1.1.1 GET /index.html 30 0.23",
      "@version" => "1",
    "@timestamp" => "2018-04-17T01:41:09.951Z",
          "host" => "master1.com",
      "clientip" => "1.1.1.1",
        "method" => "GET",
       "request" => "/index.html",
         "bytes" => "30",
      "duration" => "0.23"
}

2.4 apachelogs

[[email protected] ~]# vim /etc/logstash/conf.d/apachelogssample.conf

input {
    file {
        path    => ["/var/log/httpd/access_log"]
        type    => "apachelog"
        start_position  => "beginning"
    }
}

filter {
    grok {
        match   => { "message" => "%{COMBINEDAPACHELOG}" }
    }
}

output {
    stdout {
        codec   => rubydebug
    }
}

[[email protected] ~]# logstash -f /etc/logstash/conf.d/apachelogssample.conf --configtest
Configuration OK

运行测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/apachelogssample.conf

访问apache主页：http://10.201.106.131

技术分享图片

2.5 nginxlog

编辑pattern
[[email protected] ~]# vim /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns

# Nginx Logs
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request} (?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes} |-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}

安装启动nginx：
[[email protected] ~]# systemctl stop httpd.service
[[email protected] ~]# yum install nginx
[[email protected] ~]# systemctl start nginx.service

logstash配置：
[[email protected] ~]# cd /etc/logstash/conf.d/
[[email protected] conf.d]# cp apachelogssample.conf nginxlogsample.conf
[[email protected] conf.d]# vim nginxlogsample.conf 

input {
    file {
        path    => ["/var/log/nginx/access.log"]
        type    => "nginxlog"
        start_position  => "beginning"
    }
}

filter {
    grok {
        match   => { "message" => "%{NGINXACCESS}" }      
    }
}

output {
    stdout {
        codec   => rubydebug
    }
}

运行测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/nginxlogsample.conf 
Logstash startup completed

技术分享图片

3、output插件

3.1 redis存入数据

安装redis
[[email protected] ~]# yum install redis
配置
[[email protected] ~]# vim /etc/redis.conf
#修改其监听在0.0.0.0（监听本机所有IP）即可。
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bind 0.0.0.0

启动：
[[email protected] ~]# systemctl start redis.service

logstash配置：
[[email protected] ~]# cd /etc/logstash/conf.d/
[[email protected] conf.d]# cp nginxlogsample.conf nglogredissample.conf

[[email protected] conf.d]# vim nglogredissample.conf 

input {
    file {
        path    => ["/var/log/nginx/access.log"]
        type    => "nginxlog"
        start_position  => "beginning"
    }
}

filter {
    grok {
        match   => { "message" => "%{NGINXACCESS}" }
    }
}

output {
    redis {
        port    => "6379"
        host    => ["127.0.0.1"]
        data_type   => "list"
        key     => "logstash-%{type}"   
    }
}

语法测试：
[[email protected] conf.d]# logstash -f ./nglogredissample.conf --configtest
Configuration OK

运行测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/nglogredissample.conf
Logstash startup completed

再次访问nginx主页，http://10.201.106.131

查看redis：
[[email protected] ~]# redis-cli 
127.0.0.1:6379> LLEN logstash-nginxlog
(integer) 20

查看索引的第一个元素：
127.0.0.1:6379> LINDEX logstash-nginxlog 1
"{\"message\":\"10.201.106.1 - - [17/Apr/2018:13:51:38 +0800] \\\"GET /nginx-logo.png HTTP/1.1\\\" 200 368 \\\"http://10.201.106.131/\\\" \\\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36\\\" \\\"-\\\"\",\"@version\":\"1\",\"@timestamp\":\"2018-04-17T05:51:39.579Z\",\"host\":\"master1.com\",\"path\":\"/var/log/nginx/access.log\",\"type\":\"nginxlog\",\"clientip\":\"10.201.106.1\",\"remote_user\":\"-\",\"timestamp\":\"17/Apr/2018:13:51:38 +0800\",\"verb\":\"GET\",\"request\":\"/nginx-logo.png\",\"httpversion\":\"1.1\",\"response\":\"200\",\"bytes\":\"368\",\"referrer\":\"\\\"http://10.201.106.131/\\\"\",\"agent\":\"\\\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36\\\"\",\"http_x_forwarded_for\":\"\\\"-\\\"\"}"

3.2 redis读出数据到标准输出

master1为logstash-agent端
master2为logstash服务端
同步时间

master2的java环境变量设置
[[email protected] ~]# vim /etc/profile.d/java.sh

export JAVA_HOME=/usr

安装logstash
[[email protected] ~]# yum install logstash-1.5.4-1.noarch.rpm

定义环境变量：
[[email protected] ~]# vim /etc/profile.d/logstash.sh

export PATH=/opt/logstash/bin:$PATH

加载环境变量设置：
[[email protected] ~]# source /etc/profile.d/logstash.sh

配置接收redis数据
[[email protected] ~]# vim /etc/logstash/conf.d/server.conf

input {
    redis {
        port    => "6379"
        host    => "10.201.106.131"
        data_type   => "list"
        key     => "logstash-nginxlog"
    }
}

output {
    stdout {
        codec   => rubydebug
    }
}

语法测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/server.conf --configtest
Configuration OK

运行测试：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/server.conf
Logstash startup completed

技术分享图片

3.3 redis读出数据到Elasticsearch

3.3.1 Elasticsearch配置

技术分享图片

master3为Elasticsearch

使用系统自带JDK环境，并安装java扩展
[[email protected] ~]# yum install java-1.7.0-openjdk-devel

设置java环境变量：
[[email protected] ~]# vim /etc/profile.d/java.sh 

export JAVA_HOME=/usr

安装Elasticsearch：
[[email protected] ~]# yum install elasticsearch-1.7.2.noarch.rpm

配置：
[[email protected] ~]# vim /etc/elasticsearch/elasticsearch.yml 

cluster.name: loges
node.name: "master3.com"

启动：
[[email protected] ~]# systemctl daemon-reload
[[email protected] ~]# systemctl start elasticsearch

安装插件（方便查看状态）：
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i bigdesk -u file:///root/bigdesk-latest.zip
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
    - bigdesk

测试访问插件：

技术分享图片

3.3.2 Kibana（前端展示）

下载链接：https://www.elastic.co/downloads/past-releases

[[email protected] ~]# ls
kibana-4.1.2-linux-x64.tar.gz

解压到/usr/local
[[email protected] ~]# tar xf kibana-4.1.2-linux-x64.tar.gz -C /usr/local/
[[email protected] local]# ln -sv kibana-4.1.2-linux-x64 kibana
‘kibana’ -> ‘kibana-4.1.2-linux-x64’

配置：
[[email protected] config]# pwd
/usr/local/kibana/config
[[email protected] config]# vim kibana.yml

#修改其中一个节点的IP或者node名字，如果是本机直接localhost
elasticsearch_url: "http://10.201.106.133:9200"

运行（如需运行后台，在命令后门加&即可）：
[[email protected] ~]# /usr/local/kibana/bin/kibana

访问：
http://10.201.106.133:5601

技术分享图片

3.3.3 配置logstash输出到Elasticsearch

[[email protected] ~]# vim /etc/logstash/conf.d/server.conf 

input {
    redis {
        port    => "6379"
        host    => "10.201.106.131"
        data_type   => "list"
        key     => "logstash-nginxlog"
    }
}

output {
    elasticsearch {
        cluster => "loges"
        index   => "logstash-%{+YYYY.MM.dd}"
    }
}

语法测试（尽量使用java8）
[[email protected] ~]# logstash -f /etc/logstash/conf.d/server.conf --configtest
[2018-04-18 01:42:55.146]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Configuration OK

启动（会自动发现Elasticsearch节点）：
[[email protected] ~]# logstash -f /etc/logstash/conf.d/server.conf
[2018-04-18 01:44:19.274]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Apr 18, 2018 1:44:21 AM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-master2.com-2679-11622] version[1.7.0], pid[2679], build[929b973/2015-07-16T14:31:07Z]
Apr 18, 2018 1:44:21 AM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-master2.com-2679-11622] initializing ...
Apr 18, 2018 1:44:22 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [logstash-master2.com-2679-11622] loaded [], sites []
Apr 18, 2018 1:44:27 AM org.elasticsearch.bootstrap.Natives <clinit>
WARNING: JNA not found. native methods will be disabled.
Apr 18, 2018 1:44:29 AM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-master2.com-2679-11622] initialized
Apr 18, 2018 1:44:29 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-master2.com-2679-11622] starting ...
Apr 18, 2018 1:44:30 AM org.elasticsearch.transport.TransportService doStart
INFO: [logstash-master2.com-2679-11622] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.201.106.132:9300]}
Apr 18, 2018 1:44:30 AM org.elasticsearch.discovery.DiscoveryService doStart
INFO: [logstash-master2.com-2679-11622] loges/xZYxFmKDSu6ziX8wtt2TSQ
Apr 18, 2018 1:44:33 AM org.elasticsearch.cluster.service.InternalClusterService$UpdateTask run
INFO: [logstash-master2.com-2679-11622] detected_master [master3.com][89ejQ2cHQzC-RlTMCRnd3g][master3.com][inet[/10.201.106.133:9300]], added {[master3.com][89ejQ2cHQzC-RlTMCRnd3g][master3.com][inet[/10.201.106.133:9300]],}, reason: zen-disco-receive(from master [[master3.com][89ejQ2cHQzC-RlTMCRnd3g][master3.com][inet[/10.201.106.133:9300]]])
Apr 18, 2018 1:44:33 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-master2.com-2679-11622] started
Logstash startup completed

查看master3的Elasticsearch索引：
[[email protected] ~]# curl -XGET ‘localhost:9200/_cat/indices‘
yellow open .kibana             1 1 1 0 2.5kb 2.5kb 
yellow open logstash-2018.04.17 5 1 0 0  575b  575b 

查看索引上的文档
[[email protected] ~]# curl -XGET ‘localhost:9200/_search?pretty‘

3.3.4 配置Kibana

技术分享图片

技术分享图片

可进行搜索：

技术分享图片

3.3.5 服务后台运行

logstash：
/etc/logstash/conf.d将无用的配置文件清除后可通过守护进程启动。
service start logstash

kibana:
[[email protected] ~]# /usr/local/kibana/bin/kibana -l /var/log/kibina.log &

3.6

由于logstash较为重量级，agent侧可以使用lumberjack代替获取数据，减少对WEB服务器的资源占用 。

以上是关于logstash&Kibana杂记的主要内容，如果未能解决你的问题，请参考以下文章

logstash 多行合并

CentOS7??????Elasticsearch+ Logstash+kibana??????????????????????????????

elasticsearch+logstash+kibana部署

使用Logstash filter grok过滤日志文件

Logstash/Kibana GeoIP 不工作

用ElasticSearch,LogStash,Kibana搭建实时日志收集系统