ELK集中日志管理系统安装部署
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ELK集中日志管理系统安装部署相关的知识，希望对你有一定的参考价值。
一、简介
1.ELK介绍
ELK Stack 是 Elasticsearch、Logstash、Kibana 三个开源软件的组合。在实时数据检索和分析场合，三者通常是配合共用，而且又都先后归于 Elastic.co 公司名下，故有此简称。
ELK Stack 在最近两年迅速崛起，成为机器数据分析，或者说实时日志处理领域，开源界的第一选择。
ELK由三个组建构成：
Elasticsearch，负责数据的索引和存储
Logstash ，负责日志的采集和格式化
Kibana，负责前端统计的展示
大致的架构如下：
二、logstansh安装
1.同步时间
[[email protected] ~]# yum install -y ntpdate [[email protected] ~]# echo ‘*/5 * * * * * /usr/sbin/ntpdate us.pool.ntp.org‘ >> /var/spool/cron/root
2.JDK安装
[[email protected] ~]# yum install -y java-1.8.0 [[email protected] ~]# java -version openjdk version "1.8.0_101" OpenJDK Runtime Environment (build 1.8.0_101-b13) OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)
3.logstansh安装
[[email protected] ~]# wget https://download.elastic.co/logstash/logstash/logstash-2.3.4.tar.gz [[email protected] ~]# tar xf logstash-2.3.4.tar.gz [[email protected] ~]# mv logstash-2.3.4 /usr/local/ [[email protected] ~]# echo "PATH=$PATH:/usr/local/logstash-2.3.4/bin" >> /etc/profile [[email protected] ~]# source /etc/profile
4.新建 logstansh配置文件目录
[[email protected] ~]# mkdir /usr/local/logstash-2.3.4/conf
5.测试logstansh
[[email protected] ~]# logstash -e "input {stdin{}} output {stdout{}}" Settings: Default pipeline workers: 4 Pipeline main started
三、Redis安装
1.redis安装 [[email protected] ~]# wget http://download.redis.io/releases/redis-2.8.20.tar.gz [[email protected] ~]# yum install tcl gcc gcc-c++ -y [[email protected] ~]# tar xf redis-2.8.20.tar.gz [[email protected] ~]# mv redis-2.8.20 /usr/local/ [[email protected] ~]# cd /usr/local/redis-2.8.20/ [[email protected] redis-2.8.20]# make MALLOC=libc [[email protected] redis-2.8.20]# make install [[email protected] redis-2.8.20]# cd utils/ [[email protected] utils]# ./install_server.sh #选项默认，一直回车
2.查看redis的监控端口
[[email protected] utils]# netstat -tnlup | grep redis tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 3015/redis-server * tcp 0 0 :::6379 :::* LISTEN 3015/redis-server *
3.测试redis是否缓存数据
a.新建logstansh配置文件如下
[[email protected] ~]# cat /usr/local/logstash-2.3.4/conf/output_redis.conf input { stdin { } } #手动输入数据 output { stdout { codec => rubydebug } #页面debug信息 redis { host => ‘127.0.0.1‘ data_type => ‘list‘ key => ‘redis‘ } }
4.启动logstansh
[[email protected] ~]#logstash -f /usr/local/logstash-2.3.4/conf/output_redis.conf --verbose
5.查看redis中是否有数据
[[email protected] ~]# cd /usr/local/redis-2.8.20/src/ [[email protected] src]# ls adlist.c crc64.o lzfP.h rdb.o rio.o t_hash.o adlist.h db.c Makefile redisassert.h scripting.c t_list.c adlist.o db.o Makefile.dep redis-benchmark scripting.o t_list.o
四、elasticsearch安装
1.elasticsearch安装
[[email protected] ~]# wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.3.4/elasticsearch-2.3.4.zip [[email protected] ~]# unzip elasticsearch-2.3.4.zip [[email protected] ~]# mv elasticsearch-2.3.4 /usr/local/
修改elasticsearch配置文件
[[email protected] ~]# vim /usr/local/elasticsearch-2.3.4/config/elasticsearch.yml 把下面参数的注释去掉并改成服务器IP。这里只做简单安装，优化及集群后面再介绍 network.host： 192.168.16.177
2.elasticsearch启动
[[email protected] ~]# useradd elk [[email protected] ~]# su elk [[email protected] ~]$ chown -R elk.root /usr/local/elasticsearch-2.3.4/ [[email protected] ~]$ /usr/local/elasticsearch-2.3.4/bin/elasticsearch -d
查看是否启动
[[email protected] local]$ netstat -tnlup | grep java (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 ::ffff:192.168.16.177:9200 :::* LISTEN 2192/java tcp 0 0 ::ffff:192.168.16.177:9300 :::* LISTEN 2192/java
3、测试logstansh和elasticsearch是否能结合使用
新建logstansh配置文件elasticsearch.conf
[[email protected] conf]# cat /usr/local/logstash-2.3.4/conf/elasticsearch.conf input { stdin {} } #手动输入 output { elasticsearch { hosts => "192.168.16.177" } stdout { codec=> rubydebug } #页面debug信息 }
启动elasticsearch.conf配置文件
[[email protected] ~]#logstash -f /usr/local/logstash-2.3.4/conf/elasticsearch.conf --verbose
查看elasticsearch是否获取到了"hello elasticsearch"
[[email protected] ~]# curl http://192.168.16.177:9200/_search?pretty { "took" : 1, "timed_out" : false, "_shards" : { "total" : 0, "successful" : 0, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : 0.0, "hits" : [ ] } }
4、安装elasticsearch插件
elasticsearch有很多插件：http://www.searchtech.pro/elasticsearch-plugins
elasticsearch-head插件安装，若无法下载请至github下载，解压至/usr/local/elasticsearch-2.3.4/plugins/head目录中
[[email protected] lang-expression]# cd /usr/local/elasticsearch-2.3.4/bin/ [[email protected] bin]# ./plugin install mobz/elasticsearch-head
[[email protected] ~]# wget https://www.elastic.co/downloads/past-releases/kibana-4-5-2 [[email protected] ~]# tar xf kibana-4.5.2-linux-x64.tar.gz [[email protected] ~]# mv kibana-4.5.2-linux-x64 /usr/local/ [[email protected] ~]# vim /usr/local/kibana-4.5.2-linux-x64/config/kibana.yml 修改kibana配置文件，把下面这行改成elasticsearc的访问路径 elasticsearch.url: "http://192.168.16.177:9200"
[[email protected] ~]#sh /usr/local/kibana-4.5.2-linux-x64/bin/kibana &
六、配置客户端传输日志到ELK（本机测试）
1.server端的logstash.conf的配置
vim /usr/local/logstash-2.3.4/conf/redis_elasticserach.conf
input {
redis {
host => ‘192.168.16.177‘
data_type => ‘list‘
port => "6379"
key => ‘logstash:syslog_log‘
type => ‘redis-input‘
}
}
output {
elasticsearch {
hosts => "192.168.16.177"
index => "logstash-%{+YYYY.MM.dd}"
}
}2.client端的logstash.conf的配置
vim /usr/local/logstash-2.3.4/conf/logstash_redis.conf
input {
file {
path => "/var/log/messages"
start_position => beginning
sincedb_write_interval => 0
add_field => {"Host"=>"192.168.16.177"}
type => "SYSLOG_LOG"
}
}
output {
redis {
host => "192.168.16.177:6379"
data_type => "list"
key => "logstash:syslog_log"
}
}七、启动ELK各项服务
logstash -f /usr/local/logstash-2.3.4/conf/logstash_redis.conf & logstash -f /usr/local/logstash-2.3.4/conf/redis_elasticserach.conf & /usr/local/elasticsearch-2.3.4/bin/elasticsearch -d #elk用户启动 /usr/local/kibana-4.5.2-linux-x64/bin/kibana &
八、查看
http://192.168.16.177:9200/_plugin/head/ 点击数据浏览
http://192.168.16.177:5601/ 点击Discover
九、配置客户端传输日志到ELK
1.server端创建证书
[[email protected] ~]# cd /etc/pki/tls/ [[email protected] tls]# openssl req -subj ‘/CN=www.elk.com/‘ -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt [[email protected] tls]# scp certs/logstash-forwarder.crt 192.168.16.188:/etc/pki/tls/certs/ #在将logstash-forwarder.crt拷贝到client端
2.创建server端logstash.conf配置
echo "192.168.16.177 www.elk.com" >> /etc/hosts
vim /usr/local/logstash-2.3.4/conf/logstash.conf
input {
file {
type => "syslog"
path => [ "/var/log/pacloud/pacloud.log" ]
}
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
output {
stdout { codec=> rubydebug }
elasticsearch {hosts => "192.168.16.177:9200" }
}3.客户端安装
[[email protected] ~]# wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm [[email protected] ~]# yum localinstall -y logstash-forwarder-0.4.0-1.x86_64.rpm
#注意两个配置文件：
配置文件 /etc/logstash-forwarder.conf
日志目录 /var/log/logstash-forwarder
[[email protected] ~]# cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.bak [[email protected] ~]# echo "192.168.16.177 www.elk.com" >> /etc/hosts [[email protected] ~]# > /etc/logstash-forwarder.conf [[email protected] ~]# vim /etc/logstash-forwarder.conf { "network": { "servers": [ "www.elk.com:5000" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/pacloud/pacloud.log" ], "fields": { "type": "syslog" } }, { "paths": [ "其他路径的文件" ], "fields": { "type": "pacloud" } } ] }
注意：
一定要写域名，不能写server端的IP，因为写IP不能通过证书的认知
"ssl ca" 一定要正确写明路径
5.启动测试
服务端启动
logstash -f /usr/local/logstash-2.3.4/conf/logstash.conf & /usr/local/elasticsearch-2.3.4/bin/elasticsearch -d #elk用户启动 /usr/local/kibana-4.5.2-linux-x64/bin/kibana&
本文出自 “卫斯理” 博客，请务必保留此出处http://szk5043.blog.51cto.com/8456440/1854969
以上是关于ELK集中日志管理系统安装部署的主要内容，如果未能解决你的问题，请参考以下文章