CentOS7下Elastic Stack 5.0日志分析系统搭建
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了CentOS7下Elastic Stack 5.0日志分析系统搭建相关的知识，希望对你有一定的参考价值。
一、概述
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个开源的用于收集,分析和存储日志的工具。
Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以汇总、分析和搜索重要数据日志。
Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent，是在被监控服务器上以客户端形式运行的数据收集器的统称，可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch，然后进行后续的数据分析活动。Beats由如下组成:
1.Packetbeat：是一个网络数据包分析器，用于监控、收集网络流量信息，
Packetbeat嗅探服务器之间的流量，解析应用层协议，并关联到消息的处理， 其支 持ICMP (v4 and v6)、DNS、HTTP、mysql、PostgreSQL、Redis、
MongoDB、Memcache等协议；
2. Filebeat：用于监控、收集服务器日志文件，其已取代 logstash forwarder；
3. Metricbeat：可定期获取外部系统的监控指标信息，其可以监控、收集
Apache、HAProxy、MongoDB、MySQL、nginx、PostgreSQL、
Redis、System、Zookeeper等服务；
4. Winlogbeat：用于监控、收集Windows系统的日志信息；
5. Create your own Beat：自定义beat ，如果上面的指标不能满足需求，elasticsarch鼓励开发者 使用go语言，扩展实现自定义的beats，只需要按照模板，实现监控的输入，日志，输出等即可。
Beats 将搜集到的数据发送到 Logstash，经 Logstash 解析、过滤后，将其发送到 Elasticsearch 存储，并由 Kibana 呈现给用户。
Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源，解决了 Logstash 在各服务器节点上占用系统资源高的问题。
Elastic Stack官方下载地址:https://www.elastic.co/downloads。
二、开源实时日志分析系统Elastic Stack 5.0部署:
A.安装依赖包JDK
①关闭防火墙
# cat /etc/selinux/config |grep -v "#" SELINUX=disabled ##关闭selinux SELINUXTYPE=targeted # systemctl stop firewalld ##关闭防火墙
②下载安装JDK
# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # yum install jre ##这里我只安装的是jre环境，一样可用 # java -version openjdk version "1.8.0_111" OpenJDK Runtime Environment (build 1.8.0_111-b15) OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode
B.安装Elasticsearch
①下载安装elasticsearch
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch # cat > /etc/yum.repos.d/elasticsearch.repo <<EOF [elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF # yum install elasticsearch # systemctl start elasticsearch # systemctl status elasticsearch # /usr/share/elasticsearch/bin/elasticsearch -V ##查看elasticsearch版本 Version: 5.0.1, Build: 080bb47/2016-11-11T22:08:49.812Z, JVM: 1.8.0_111
注:安装后出现如下问题，致使elasticsearch无法启动
# systemctl status elasticsearch OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should conf...CThreads=N OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error=‘Cannot a ...‘(errno=12) # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/hs_err_pid11084.log
解决:
配置elasticsearch下的jvm.options：
# vi /etc/elasticsearch/jvm.options -Xms4g ##启用如下两项 -Xmx4g ##-Xms2g ##关闭如下两项 ##-Xmx2g
②ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300。
# ss -tlnp |grep -E ‘9200|9300‘
③测试服务
# curl -X GET http://localhost:9200
{
"name" : "XVY0Ovb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA",
"version" : {
"number" : "5.0.1",
"build_hash" : "080bb47",
"build_date" : "2016-11-11T22:08:49.812Z",
"build_snapshot" : false,
"lucene_version" : "6.2.1"
},
"tagline" : "You Know, for Search"
}
也可以使用如下命令测试
# curl -i -XGET ‘localhost:9200/‘
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 327
{
"name" : "XVY0Ovb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA",
"version" : {
"number" : "5.0.1",
"build_hash" : "080bb47",
"build_date" : "2016-11-11T22:08:49.812Z",
"build_snapshot" : false,
"lucene_version" : "6.2.1"
},
"tagline" : "You Know, for Search"
}
C.安装Logstash
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch # cat > /etc/yum.repos.d/logstash.repo <<EOF [logstash-5.x] name=Elastic repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF # yum install logstash # systemctl start logstash # systemctl status logstash # /usr/share/logstash/bin/logstash -V ##查看logstash版本 logstash 5.0.1
D.安装Kibana
①安装Kibana
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch # cat > /etc/yum.repos.d/kibana.repo <<EOF [kibana-5.x] name=Kibana repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF # yum install kibana # systemctl start kibana # systemctl status kibana # /usr/share/kibana/bin/kibana -V ##查看kibana版本 5.0.1
②只需更改如下配置
# cat /etc/kibana/kibana.yml |grep -v "#" server.host: "192.168.147.128"
③检测服务
# ss -tlnp|grep 5601 ##Kibana默认进程名：node ，端口5601
E、Beats安装部署
a.安装部署Filebeat
①安装Filebeat
# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm # rpm -ivh filebeat-5.0.1-x86_64.rpm # systemctl start filebeat # systemctl status filebeat # filebeat.sh -version ##查看版本 filebeat version 5.0.1 (amd64), libbeat 5.0.1
②配置Filebeat
# cp /etc/filebeat # cp filebeat.yml filebeat.yml.bak # vi /etc/filebeat/filebeat.yml ##配置filebeat #============= Filebeat prospectors =============== filebeat.prospectors: - input_type: log paths: - /var/log/*.log #==================== Outputs ===================== #------------- Elasticsearch output --------------- output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] #---------------- Logstash output ----------------- output.logstash: # The Logstash hosts hosts: ["localhost:5043"] ##只需配置该处，其他默认即可 # filebeat.sh -configtest -e ##验证配置文件
③配置Logstash
# cat /etc/logstash/conf.d/logstash.conf
input {
beats {
port => "5043"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
# systemctl restart logstash ##重启logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit ##验证配置文件
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
# ss -tlnp|grep -E ‘5043|9600‘
④配置kibana
浏览器输入http://localhost:5601，配置filebeat的索引（只需输入filebeat-*即可）。
在第一个框里输入filebeat-*后稍等片刻，kibana会自动识别，OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后，最后就会呈现如下图所示:
我们再会过头新建logstash的索引，浏览器输入http://localhost:5601，点击左边栏的”Management”===>然后点击“index Patterns”===>
===>然后点击“Add New”===>
===>点击“Crete”按钮创建logstash索引，创建完成后即会展现如下图所示:
b.安装部署Packetbeat
①安装配置Packetbeat
# yum install libpcap # curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm # rpm -ivh packetbeat-5.0.1-x86_64.rpm # cat /etc/packetbeat/packetbeat.yml ==================== Network device =================== packetbeat.interfaces.device: any ##捕获所有消息发送或接收的网络接口 ======================== Flows ======================== packetbeat.flows: timeout: 30s period: 10s ================== Transaction protocols ============== ##如下是packetbeat默认支持的主要协议及端口 packetbeat.protocols.icmp: enabled: true packetbeat.protocols.amqp: ports: [5672] packetbeat.protocols.cassandra: ports: [9042] packetbeat.protocols.dns: ports: [53] include_authorities: true include_additionals: true packetbeat.protocols.http: ports: [80, 8080, 8000, 5000, 8002] packetbeat.protocols.memcache: ports: [11211] packetbeat.protocols.mysql: ports: [3306] packetbeat.protocols.pgsql: ports: [5432] packetbeat.protocols.redis: ports: [6379] packetbeat.protocols.thrift: ports: [9090] packetbeat.protocols.mongodb: ports: [27017] packetbeat.protocols.nfs: ports: [2049] ========================= General ========================= ========================= Outputs ========================= ------------------- Elasticsearch output ------------------ output.elasticsearch: hosts: ["localhost:9200"] --------------------- Logstash output --------------------- output.logstash: hosts: ["localhost:5043"] ##只需配置该处，其他默认即可 ============================= Logging =====================
②验证配置并启动packetbeat
# packetbeat.sh -version ##查看packetbeat版本 packetbeat version 5.0.1 (amd64), libbeat 5.0.1 # packetbeat.sh -configtest -e ##测试配置文件 ...... Config OK # systemctl start packetbeat # systemctl status packetbeat
③配置Kibana(新建packetbeat索引)
在http://localhost:5601下新建索引页面输入“packetbeat-*”，之后kibana会自动更新，在“Time-field name”下面的三个选项中选择“@timestamp”，最后点击“Create”创建即可。
创建完成后，kibana显示如下:
c.安装部署Metricbeat
①安装配置metricbeat
# curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm # rpm -ivh metricbeat-5.0.1-x86_64.rpm # cat /etc/metricbeat/metricbeat.yml ================= Modules configuration ================= metricbeat.modules: ---------------------- System Module --------------------- - module: system metricsets: - cpu - load - filesystem - fsstat - memory - network - process enabled: true period: 10s processes: [‘.*‘] ========================= General ======================= ========================= Outputs ======================= ------------------- Elasticsearch output ---------------- output.elasticsearch: hosts: ["localhost:9200"] --------------------- Logstash output ------------------- output.logstash: hosts: ["localhost:5043"] ##只需配置该项，其他默认即可 ======================= Logging =========================
②验证配置并启动metricbeat
# metricbeat.sh -version ##查看版本信息 metricbeat version 5.0.1 (amd64), libbeat 5.0.1 # metricbeat.sh -configtest -e ##验证配置文件 # systemctl start metricbeat # systemctl status metricbeat
③配置kibana(新建metricbeat索引)
在http://localhost:5601下新建索引页面输入“metricbeat-*”，之后kibana会自动更新，在“Time-field name”下面的选项中选择“@timestamp”，最后点击“Create”创建即可。
最后呈现如下图所示:
注1:
关于ELK Stack的一些查询语句:
①查询filebeat
# curl -XGET ‘http://localhost:9200/filebeat-*/_search?pretty‘
②查询packetbeat
# curl -XGET ‘http://localhost:9200/packetbeat-*/_search?pretty‘
③查询metricbeat
# curl -XGET ‘http://localhost:9200/metricbeat-*/_search?pretty‘
④查询集群健康度
# curl ‘localhost:9200/_cat/health?v‘
⑤查看节点列表
# curl ‘localhost:9200/_cat/nodes?v‘ ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 37 93 3 0.05 0.07 0.41 mdi * XVY0Ovb
⑥列出所有索引
# curl ‘localhost:9200/_cat/indices?v‘ health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open filebeat-2016.11.28 Mn4MzxdTRaCj9iseutcmqA 5 1 2 0 12kb 12kb yellow open filebeat-2016.11.29 iMrr710mT42mApxdV62k-A 5 1 159 0 65.9kb 65.9kb yellow open packetbeat-2016.11.29 wkTcIwD6RgiiCFwlWBIILA 5 1 5652 0 1.6mb 1.6mb yellow open customer NvxXLgHoREefJLRhot13Ug 5 1 0 0 800b 800b yellow open packetbeat-2016.11.28 Beoe07S7QB-dntNV4nxJNQ 5 1 2446 0 676.4kb 676.4kb yellow open test M7WbkYq2QNmeJ9NOyMfMZA 5 1 0 0 800b 800b yellow open logstash-2016.11.28 pcb_84ChSBe9A7VRd-SQNw 5 1 161 0 123.2kb 123.2kb yellow open metricbeat-2016.11.29 AmVeT1xCQGCnxlAFXUxhYw 5 1 94459 0 37.6mb 37.6mb yellow open logstash-2016.11.29 6PCKMYKCSVmPfdg-Sx2ARA 5 1 85772 0 20.3mb 20.3mb yellow open .kibana QYTg0I5KS-yc3d7GSey3Zw 1 1 5 0 102kb 102kb
注2:
如果搭建期间有什么不清楚或不了解的，建议去看官方文档，文档地址如下:
https://www.elastic.co/guide/index.html
本文出自 “记事本” 博客，请务必保留此出处http://wangzhijian.blog.51cto.com/6427016/1878636
以上是关于CentOS7下Elastic Stack 5.0日志分析系统搭建的主要内容，如果未能解决你的问题，请参考以下文章
Elastic Stack：Centos7安装Kibana7.7.1以及开机自启
Elastic Stack：Centos7安装Kibana7.7.1以及开机自启
Elasticsearch：在 Windows 下安装 Elastic Stack 8.x