ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台相关的知识,希望对你有一定的参考价值。

ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台

 

日志主要包括系统日志、应用程序日志和安全日志。系统运维和开发人员可以通过日志了解服务器软硬件信息、检查配置过程中的错误及错误发生的原因。经常分析日志可以了解服务器的负荷,性能安全性,从而及时采取措施纠正错误。

通常,日志被分散的储存不同的设备上。如果你管理数十上百台服务器,你还在使用依次登录每台机器的传统方法查阅日志。这样是不是感觉很繁琐和效率低下。当务之急我们使用集中化的日志管理,例如:开源的syslog,将所有服务器上的日志收集汇总。

集中化管理日志后,日志的统计和检索又成为一件比较麻烦的事情,一般我们使用grep、awk和wc等Linux命令能实现检索和统计,但是对于要求更高的查询、排序和统计等要求和庞大的机器数量依然使用这样的方法难免有点力不从心。

官方网站

https://www.elastic.co/

 ELK中文指南 http://kibana.logstash.es/content/index.html

技术分享

 

 

1、部署环境
[[email protected] ~]# cat /etc/redhat-release 
CentOS release 6.8 (Final)

关闭防火墙&Sellinux
http://blog.csdn.net/xiegh2014/article/details/53031781

配置yum源
http://blog.csdn.net/xiegh2014/article/details/53031894


两台服务器
节点1安装部署
主机hosts文件配置
[[email protected] ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.8.95           elk-node1
172.16.8.96           elk-node2

JAVA安装(安装JDK需要重启操作系统)
[[email protected] ~]# rpm -ivh jdk-8u111-linux-x64.rpm 

elasticsearch安装
[[email protected] ~]# rpm -ivh elasticsearch-5.1.1.rpm 
[[email protected] ~]# chkconfig --add elasticsearch

elasticsearch配置
[[email protected] ~]# chkconfig --add elasticsearch
[[email protected] ~]# mkdir -pv /data/elasticsearch/{data,logs}
mkdir: 已创建目录 "/data"
mkdir: 已创建目录 "/data/elasticsearch"
mkdir: 已创建目录 "/data/elasticsearch/data"
mkdir: 已创建目录 "/data/elasticsearch/logs"
[[email protected] ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch
[[email protected] ~]# grep -n ‘^[a-z]‘ /etc/elasticsearch/elasticsearch.yml 
[[email protected] ~]# vi /etc/elasticsearch/elasticsearch.yml 
[[email protected] ~]# grep -n ‘^[a-z]‘ /etc/elasticsearch/elasticsearch.yml 
17:cluster.name: app-elk
23:node.name: elk-node1
33:path.data: /data/elasticsearch/data
37:path.logs: /data/elasticsearch/logs
43:bootstrap.memory_lock: true
55:network.host: 0.0.0.0
59:http.port: 9200

修改elasticsearch的参数 # 换个集群的名字,免得跟别人的集群混在一起 cluster.name: es-5.0-test # 换个节点名字 node.name: node-101 # 修改一下ES的监听地址,这样别的机器也可以访问 network.host: 0.0.0.0 # 默认的就好 http.port: 9200 # 增加新的参数,这样head插件可以访问es http.cors.enabled: true http.cors.allow-origin: "*" [[email protected] ~]# vi /etc/security/limits.conf # allow user ‘elasticsearch‘ mlockall elasticsearch   soft     memlock         unlimited elasticsearch   hard     memlock         unlimited [[email protected] ~]# vi /etc/security/limits.conf *               soft     nofile          65536 *               hard     nofile          131072 *               soft     nproc           2048 *               hard     nproc           4096 [[email protected] ~]# vi /etc/security/limits.d/90-nproc.conf  修改如下内容: * soft nproc 1024 #修改为 * soft nproc 2048 [[email protected] ~]# vi /etc/sysctl.conf 添加下面配置: vm.max_map_count=655360 [[email protected] ~]# sysctl -p [[email protected] ~]# /etc/init.d/elasticsearch  restart http://172.16.8.95:9200/

技术分享

节点2安装部署
主机hosts文件配置
[[email protected] ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.8.95           elk-node1
172.16.8.96           elk-node2


JAVA安装(安装JDK需要重启操作系统)
[[email protected] ~]# rpm -ivh jdk-8u111-linux-x64.rpm 


elasticsearch安装
[[email protected] ~]# rpm -ivh elasticsearch-5.1.1.rpm 
[[email protected] ~]# chkconfig --add elasticsearch


elasticsearch配置
[[email protected] ~]# mkdir -pv /data/elasticsearch/{data,logs}
[[email protected] ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch
[[email protected] ~]# grep -n ‘^[a-z]‘ /etc/elasticsearch/elasticsearch.yml 
17:cluster.name: app-elk
23:node.name: elk-node2
33:path.data: /data/elasticsearch/data
37:path.logs: /data/elasticsearch/logs
43:bootstrap.memory_lock: true
55:network.host: 0.0.0.0
59:http.port: 9200


[[email protected] ~]# /etc/init.d/elasticsearch restart


报错信息1
[[email protected] ~]# tail -f /data/elasticsearch/logs/app-elk.log 
[2016-09-19T18:08:11,804][INFO ][o.e.t.TransportService   ] [elk-node2] publish_address {172.16.8.96:9300}, bound_addresses {[::]:9300}
[2016-09-19T18:08:11,825][INFO ][o.e.b.BootstrapCheck     ] [elk-node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-09-19T18:08:11,830][ERROR][o.e.b.Bootstrap          ] [elk-node2] node validation exception
bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2016-09-19T18:08:11,842][INFO ][o.e.n.Node               ] [elk-node2] stopping ...
[2016-09-19T18:08:11,896][INFO ][o.e.n.Node               ] [elk-node2] stopped
[2016-09-19T18:08:11,896][INFO ][o.e.n.Node               ] [elk-node2] closing ...
[2016-09-19T18:08:11,933][INFO ][o.e.n.Node               ] [elk-node2] closed


[[email protected] ~]# vi /etc/security/limits.conf
*               soft     nofile          65536
*               hard     nofile          131072
*               soft     nproc           2048
*               hard     nproc           4096
[[email protected] ~]# vi /etc/security/limits.d/90-nproc.conf 
修改如下内容:
* soft nproc 1024
#修改为
* soft nproc 2048


[[email protected] ~]# vi /etc/sysctl.conf
添加下面配置:
vm.max_map_count=655360
[[email protected] ~]# sysctl -p
[[email protected] ~]# /etc/init.d/elasticsearch  restart


报错信息2
[2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=无法分配内存
[2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2016-09-19T18:18:19,271][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user ‘elasticsearch‘ mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2016-09-19T18:18:19,271][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2016-09-19T18:18:20,000][INFO ][o.e.n.Node               ] [elk-node2] initializing ...
[2016-09-19T18:18:20,384][INFO ][o.e.e.NodeEnvironment    ] [elk-node2] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [39gb], net total_space [43.9gb], spins? [possibly], types [ext4]
[2016-09-19T18:18:20,385][INFO ][o.e.e.NodeEnvironment    ] [elk-node2] heap size [3.9gb], compressed ordinary object pointers [true]
[2016-09-19T18:18:20,391][INFO ][o.e.n.Node               ] [elk-node2] node name [elk-node2], node ID [KBLSr8zERri083vvtJBQhA]
[2016-09-19T18:18:20,405][INFO ][o.e.n.Node               ] [elk-node2] version[5.1.1], pid[25073], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-642.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
[2016-09-19T18:18:29,227][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [aggs-matrix-stats]
[2016-09-19T18:18:29,228][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [ingest-common]
[2016-09-19T18:18:29,228][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [lang-expression]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [lang-groovy]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [lang-mustache]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [lang-painless]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [percolator]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [reindex]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [transport-netty3]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService     ] [elk-node2] loaded module [transport-netty4]
[2016-09-19T18:18:29,231][INFO ][o.e.p.PluginsService     ] [elk-node2] no plugins loaded


[[email protected] ~]# vi /etc/security/limits.conf
# allow user ‘elasticsearch‘ mlockall
elasticsearch   soft     memlock         unlimited
elasticsearch   hard     memlock         unlimited


[[email protected] ~]# /etc/init.d/elasticsearch  restart


http://172.16.8.96:9200/

技术分享

以上是关于ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台的主要内容,如果未能解决你的问题,请参考以下文章

ELk之使用kibana展示访问IP地图

ElasticSearch——自定义模板

ELK===》ELK介绍Elasticsearch单节点部署Elasticsearch集群部署

企业运维之 ELK日志分析平台(Elasticsearch)

企业运维之 ELK日志分析平台(Elasticsearch)

批量搞机:分布式ELK平台Elasticsearch介绍Elasticsearch集群安装ES 插件的安装与使用