ELK——01安装ELK
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ELK——01安装ELK相关的知识，希望对你有一定的参考价值。
参考技术A Centos 7.7因为搭建ELK需要Java环境，所以需要知道JAVA_HOME。
我最终在 /usr/lib/jvm/jre-1.8.0-openjdk 中知道了JDK，然后指定此路径为JAVA_HOME。
https://www.elastic.co/cn/downloads/
logstash官网
直接安装logstash官网方式安装。
和logstash一样，采用yum安装方式，直接看官网教程，不再赘述。
https://www.elastic.co/guide/en/kibana/7.9/rpm.html#rpm-repo
ELK也是cs架构，所以会有一个服务器用于接收存储并展示日志，此账号用于客户端。
一般而言，会为ELK专门创建一个账号。这个账号会用于所有ELK的相关连接和通信。
user_name改为用户名，your_pwd改为密码。
这是出于系统安全考虑设置的条件。由于ElasticSearch可以接收用户输入的脚本并且执行，为了系统安全考虑，
建议创建一个单独的用户用来运行ElasticSearch。
用curl 访问 http://localhost:9200 ，看是否能得到返回值。若得到返回值，证明elasticsearch服务启动成功。
测试可用性：打开redis客户端，随便设置一个值，并取出来。
yum 安装软件的原文件一版放置于 /etc/ 目录下，kibana也不例外，放在/etc/kibana
修改 /etc/kibana/kibana.yml 配置文件中的server.host 和 server.port，如下图所示。
然后再通过浏览器访问kibana的地址就可以了。
ELK测试
ELK 日志分析系统
ELK 日志分析系统
1.0 ELK 介绍
1.1 ELK 安装准备工作
1.2 es 安装
1.3 es配置
1.4 es测试
1.5 Kibana安装
1.6 logstash安装
1.7 logstash配置解析rsyslog文件
1.8 kibana查看日志
1.9 nginx日志收集
2.0 beats采集日志
1.0 ELK 介绍
官网https://www.elastic.co/cn/
中文指南https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details
ELK Stack （5.0版本之后）? Elastic Stack == （ELK Stack + Beats）
ELK Stack包含：ElasticSearch、Logstash、Kibana
ElasticSearch是一个搜索引擎，用来搜索、分析、存储日志。它是分布式的，也就是说可以横向扩容，可以自动发现，索引自动分片，总之很强大。文档https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html
Logstash用来采集日志，把日志解析为json格式交给ElasticSearch。
Kibana是一个数据可视化组件，把处理后的结果通过web界面展示
Beats在这里是一个轻量级日志采集器，其实Beats家族有5个成员
早期的ELK架构中使用Logstash收集、解析日志，但是Logstash对内存、cpu、io等资源消耗比较高。相比
Logstash，Beats所占系统的CPU和内存几乎可以忽略不计
x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，是收费的
1.1 ELK 安装准备工作
环境准备：192.168.137.30、192.168.137.40、192.168.137.45
// 三台机器均安装Elaelasticsearch(后续简称es)、jdk8，设置hosts
主节点：
192.168.137.30
数据节点：
192.168.137.40、192.168.137.45
所有节点均安装jdk环境
yum install -y java-1.8.0-openjdk
1.2 es 安装
官方文档 https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html
三台机器均都要执行以下操作
[[email protected] ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[[email protected] ~]# cat /etc/yum.repos.d/elastic.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[[email protected] ~]# yum install -y elasticsearch
或者方法安装
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
[[email protected] ~]# rpm -ivh elasticsearch-6.0.0.rpm
1.3 es配置
elasticsearch配置文件/etc/elasticsearch和/etc/sysconfig/elasticsearch
参考https://www.elastic.co/guide/en/elasticsearch/reference/6.0/rpm.html
主节点：192.168.137.30编辑配置文件
[[email protected] ~]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: linux-node3.com
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: linux-node3.com
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
node.master: true
node.data: false
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.137.30
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["192.168.137.30", "192.168.137.40", "192.168.137.45"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
[[email protected] ~]#
同理修改数据节点配置：
[[email protected] ~]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: linux-node3.com
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: linux-node4.com
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
node.master: false
node.data: true
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.137.40
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["192.168.137.30", "192.168.137.40", "192.168.137.45"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
[[email protected] ~]#
三台均启动服务
[[email protected] ~]# systemctl start elasticsearch
[[email protected] ~]# ps -aux |grep elasticsearch
elastic+ 3140 23.7 45.9 1482312 459248 ? Ssl 14:29 0:00 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root 3168 5.0 0.0 112680 716 pts/1 S+ 14:29 0:00 grep --color=auto elasticsearch
[[email protected] ~]#
[[email protected] ~]# netstat -lntnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 966/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1095/master
tcp 0 0 192.168.137.30:27017 0.0.0.0:* LISTEN 1006/mongod
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1006/mongod
tcp6 0 0 192.168.137.30:9200 :::* LISTEN 1422/java
tcp6 0 0 :::8080 :::* LISTEN 1185/java
tcp6 0 0 :::80 :::* LISTEN 961/httpd
tcp6 0 0 192.168.137.30:9300 :::* LISTEN 1422/java
tcp6 0 0 :::22 :::* LISTEN 966/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1095/master
[[email protected] ~]#
// 依次启动数据节点。端口分别为9200,9300
[[email protected] ~]# ss -ltnp |grep -E ‘9200|9300‘
LISTEN 0 128 ::ffff:192.168.137.45:9200 :::* users:(("java",pid=2758,fd=118))
LISTEN 0 128 ::ffff:192.168.137.45:9300 :::* users:(("java",pid=2758,fd=108))
[[email protected] ~]#
[[email protected] ~]# ss -ltnp |grep -E ‘9200|9300‘
LISTEN 0 128 ::ffff:192.168.137.40:9200 :::* users:(("java",pid=3257,fd=119))
LISTEN 0 128 ::ffff:192.168.137.40:9300 :::* users:(("java",pid=3257,fd=110))
1.4 es测试
健康检查
[[email protected] ~]# curl ‘192.168.137.30:9200/_cluster/health?pretty‘
{
"cluster_name" : "linux-node3.com",
"status" : "green", //健康状态
"timed_out" : false,
"number_of_nodes" : 3, //3个节点
"number_of_data_nodes" : 2, //2个数据节点
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[[email protected] ~]#
集群详细信息
[[email protected] ~]# curl ‘192.168.137.30:9200/_cluster/state?pretty‘
{
"cluster_name" : "linux-node3.com",
"compressed_size_in_bytes" : 355,
"version" : 5,
"state_uuid" : "RBH5dvstTyqgHVVSdfNi_Q",
"master_node" : "d1yLa9f9RfSPwUXPvm_lqQ",
"blocks" : { },
"nodes" : {
"d1yLa9f9RfSPwUXPvm_lqQ" : {
"name" : "linux-node3.com",
"ephemeral_id" : "DGs6lBiaQvaJlmyasez-TA",
"transport_address" : "192.168.137.30:9300",
"attributes" : { }
},
"pyOddTkYRN6fRjWjb-ehBw" : {
"name" : "linux-05.com",
"ephemeral_id" : "X8oa-yozSxqVmb6Dp2fhAQ",
"transport_address" : "192.168.137.45:9300",
"attributes" : { }
},
"mf7rEM3oScqEOqNFniEJfA" : {
"name" : "linux-node4.com",
"ephemeral_id" : "eZ4jATDJRDyv3rmnup3zfg",
"transport_address" : "192.168.137.40:9300",
"attributes" : { }
}
},
"metadata" : {
"cluster_uuid" : "3_2FFY-XTPexeDEZ6MXR1Q",
"templates" : { },
"indices" : { },
"index-graveyard" : {
"tombstones" : [ ]
}
},
"routing_table" : {
"indices" : { }
},
"routing_nodes" : {
"unassigned" : [ ],
"nodes" : {
"mf7rEM3oScqEOqNFniEJfA" : [ ],
"pyOddTkYRN6fRjWjb-ehBw" : [ ]
}
},
"restore" : {
"snapshots" : [ ]
},
"snapshots" : {
"snapshots" : [ ]
},
"snapshot_deletions" : {
"snapshot_deletions" : [ ]
}
}
[[email protected] ~]#
1.5 Kibana安装
主节点安装：kibana
[[email protected] ~]# yum install -y kibana //会很慢
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
[[email protected] ~]# rpm -ivh kibana-6.0.0-x86_64.rpm
准备中... ################################# [100%]
正在升级/安装...
1:kibana-6.0.0-1 ################################# [100%]
[[email protected] ~]#
[[email protected] ~]# grep -v "^#" /etc/kibana/kibana.yml
server.port: 5601 //监听端口
server.host: "192.168.137.30"
elasticsearch.url: "http://192.168.137.30:9200" //es访问地址
logging.dest: /var/log/kibana.log
[[email protected] ~]#
[[email protected] log]# touch kibana.log
[[email protected] log]# chmod 777 kibana.log
[[email protected] log]# systemctl restart kibana
[[email protected] log]# ps aux | grep kibana
kibana 1626 39.5 11.6 1121852 116968 ? Ssl 17:09 0:04 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root 1638 0.0 0.0 112680 980 pts/0 R+ 17:10 0:00 grep --color=auto kibana
[[email protected] log]# netstat -lntnp | grep nod
tcp 0 0 192.168.137.30:5601 0.0.0.0:* LISTEN 1626/node
[[email protected] log]#
浏览器里访问 http://192.168.137.30:5601
1.6 logstash安装
数据节点安装： logstash
[[email protected] ~]# yum install -y logstash 或者使用
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
rpm -ivh logstash-6.0.0.rpm
[[email protected] ~]# rpm -ivh logstash-6.0.0.rpm
准备中... ################################# [100%]
正在升级/安装...
1:logstash-1:6.0.0-1 ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Successfully created system startup script for Logstash
1.7 logstash配置解析rsyslog文件
[[email protected] ~]# cat /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
stdout {
codec => rubydebug
}
}
[[email protected] ~]#
检测配置文件是否有错
[[email protected] ~]# cd /usr/share/logstash/bin
[[email protected] bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash‘s logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
启动logstash
vim /etc/rsyslog.conf//在#### RULES下面增加一行
*.* @@192.168.137.40:10514
[[email protected] ~]# systemctl restart rsyslog
[[email protected] ~]# netstat -lnp |grep 10514
tcp6 0 0 :::10514 :::* LISTEN 3708/java
udp 0 0 0.0.0.0:10514 0.0.0.0:* 3708/java
[[email protected] bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash‘s logs to /var/log/logstash which is now configured via log4j2.properties
{
"severity" => 6,
"program" => "rsyslogd",
"message" => "[origin software="rsyslogd" swVersion="7.4.7" x-pid="3768" x-info="http://www.rsyslog.com"] start
",
"type" => "system-syslog",
"priority" => 46,
"logsource" => "linux-node4",
"@timestamp" => 2017-12-14T10:07:03.000Z,
"@version" => "1",
"host" => "192.168.137.40",
"facility" => 5,
"severity_label" => "Informational",
"timestamp" => "Dec 14 18:07:03",
"facility_label" => "syslogd"
}
{
"severity" => 6,
"program" => "systemd",
"message" => "Stopping System Logging Service...
",
"type" => "system-syslog",
"priority" => 30,
"logsource" => "linux-node4",
"@timestamp" => 2017-12-14T10:07:03.000Z,
"@version" => "1",
"host" => "192.168.137.40",
"facility" => 3,
"severity_label" => "Informational",
"timestamp" => "Dec 14 18:07:03",
"facility_label" => "system"
}
{
"severity" => 6,
"program" => "systemd",
"message" => "Starting System Logging Service...
",
"type" => "system-syslog",
"priority" => 30,
"logsource" => "linux-node4",
"@timestamp" => 2017-12-14T10:07:03.000Z,
"@version" => "1",
"host" => "192.168.137.40",
"facility" => 3,
"severity_label" => "Informational",
"timestamp" => "Dec 14 18:07:03",
"facility_label" => "system"
}
{
"severity" => 6,
"program" => "systemd",
"message" => "Started System Logging Service.
",
"type" => "system-syslog",
"priority" => 30,
"logsource" => "linux-node4",
"@timestamp" => 2017-12-14T10:07:03.000Z,
"@version" => "1",
"host" => "192.168.137.40",
"facility" => 3,
"severity_label" => "Informational",
"timestamp" => "Dec 14 18:07:03",
"facility_label" => "system"
}
{
"severity" => 5,
"pid" => "654",
"program" => "polkitd",
"message" => "Unregistered Authentication Agent for unix-process:3761:2437779 (system bus name :1.47, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8) (disconnected from bus)
",
"type" => "system-syslog",
"priority" => 85,
"logsource" => "linux-node4",
"@timestamp" => 2017-12-14T10:07:03.000Z,
"@version" => "1",
"host" => "192.168.137.40",
"facility" => 10,
"severity_label" => "Notice",
"timestamp" => "Dec 14 18:07:03",
"facility_label" => "security/authorization"
}
// 注：启动后不能敲命令，屏幕上查看到日志输出
1.8 kibana查看日志
数据节点配置日志收集：
启动logstash
[[email protected] ~]# cat /etc/logstash/conf.d/syslog.conf
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.137.30:9200"]
index => "system-syslog-%{+YYYY.MM}"
}
}
[[email protected] ~]# chown -R logstash /var/lib/logstash
[[email protected] ~]# systemctl start logstash
[[email protected] ~]#
[[email protected] ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 963/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1238/master
tcp6 0 0 192.168.137.40:9200 :::* LISTEN 13890/java
tcp6 0 0 :::10514 :::* LISTEN 14164/java
tcp6 0 0 192.168.137.40:9300 :::* LISTEN 13890/java
tcp6 0 0 :::22 :::* LISTEN 963/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1238/master
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 14164/java
// 默认是127.0.0.1：9600,修改
[[email protected] ~]# grep -v "^#" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
http.host: "192.168.137.40"
path.logs: /var/log/logstash
[[email protected] ~]#
[[email protected] ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 965/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1224/master
tcp6 0 0 192.168.137.40:9200 :::* LISTEN 2215/java
tcp6 0 0 :::10514 :::* LISTEN 5450/java
tcp6 0 0 192.168.137.40:9300 :::* LISTEN 2215/java
tcp6 0 0 :::22 :::* LISTEN 965/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1224/master
tcp6 0 0 192.168.137.40:9600 :::* LISTEN 5450/java
主节点查看索引信息
[[email protected] ~]# curl ‘192.168.137.30:9200/_cat/indices?v‘ //可以获取索引信息
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana Z7JVUVlLSRSu5xqySplQ5w 1 1 1 0 6.9kb 3.4kb
green open system-syslog-2017.12 c9ZmYijTTYSMLMIESb3N4Q 5 1 1 0 24.5kb 12.2kb
[[email protected] ~]#
[[email protected] ~]# curl -XGET ‘192.168.137.30:9200/indexname?pretty‘
{ // 获指定索引详细信息
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "indexname",
"index_uuid" : "_na_",
"index" : "indexname"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "indexname",
"index_uuid" : "_na_",
"index" : "indexname"
},
"status" : 404
}
[[email protected] ~]#
curl -XDELETE ‘localhost:9200/logstash-xxx-*‘ 可以删除指定索引
浏览器访问192.168.137.40:5601，到kibana配置索引
左侧点击“Managerment”-> “Index Patterns”-> “Create Index Pattern”
Index pattern这里需要根据前面curl查询到的索引名字来写，否则下面的按钮是无法点击
输入：system-syslog-2017.12 或者system-syslog-*
1.9 nginx日志收集
[[email protected] ~]# cat /etc/logstash/conf.d/nginx.conf
input {
file {
path => "/tmp/elk_access.log"
start_position => "beginning"
type => "nginx"
}
}
filter {
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} [%{HTTPDATE:timestamp}] "(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.137.40:9200"]
index => "nginx-test-%{+YYYY.MM.dd}"
}
}
[[email protected] ~]# cd /usr/share/logstash/bin
[[email protected] bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash‘s logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
//没有nginx的需要安装nginx
[[email protected] ~]# yum -y install nginx
[[email protected] ~]# cat /etc/nginx/conf.d/elk.conf
server {
listen 80;
server_name elk.linux.com;
location / {
proxy_pass http://192.168.137.30:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2;
}
[[email protected] ~]#
配置日志
vim /etc/nginx/nginx.conf//增加如下内容
log_format main2 ‘$http_host $remote_addr - $remote_user [$time_local] "$request" ‘
‘$status $body_bytes_sent "$http_referer" ‘
‘"$http_user_agent" "$upstream_addr" $request_time‘;
[[email protected] ~]# systemctl start nginx
[[email protected] ~]# ps -ef | grep nginx
root 2916 1 0 14:48 ? 00:00:00 nginx: master process /usr/sbin/nginx
nginx 2917 2916 0 14:48 ? 00:00:00 nginx: worker process
root 2919 2732 0 14:48 pts/4 00:00:00 grep --color=auto nginx
[[email protected] ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2916/nginx: master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 972/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1247/master
tcp6 0 0 :::80 :::* LISTEN 2916/nginx: master
tcp6 0 0 192.168.137.40:9200 :::* LISTEN 2214/java
tcp6 0 0 :::10514 :::* LISTEN 2304/java
tcp6 0 0 192.168.137.40:9300 :::* LISTEN 2214/java
tcp6 0 0 :::22 :::* LISTEN 972/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1247/master
tcp6 0 0 192.168.137.40:9600 :::* LISTEN 2304/java
[[email protected] ~]#
绑定hosts 192.168.137.40 elk.linux.com
浏览器访问，检查是否有日志产生
[[email protected] ~]# systemctl restart logstash
主节点查看获取的索引
[[email protected] ~]# curl ‘192.168.137.30:9200/_cat/indices?v‘
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana Z7JVUVlLSRSu5xqySplQ5w 1 1 2 0 20.4kb 10.2kb
green open system-syslog-2017.12 c9ZmYijTTYSMLMIESb3N4Q 5 1 127 0 723.8kb 317.6kb
green open nginx-test-2017.12.18 w3j3J-wXT6eXzaVf6ycmBg 5 1 20 0 42.7kb 466b
[[email protected] ~]#
// 检查是否有nginx-test开头的索引生成
如果有，才能到kibana里去配置该索引
左侧点击“Managerment”-> “Index Patterns”-> “Create Index Pattern”
Index pattern这里写nginx-test-*
之后点击左侧的Discover
2.0 beats采集日志
官网：https://www.elastic.co/cn/products/beats
优点：可扩展，支持自定义构建
数据节点：linux-05.com
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
[[email protected] ~]# rpm -ivh filebeat-6.0.0-x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:filebeat-6.0.0-1 ################################# [100%]
[[email protected] ~]#
编辑配置文件
[[email protected] ~]# grep -v "^#" /etc/filebeat/filebeat.yml | grep -v "#" |grep -v "^$"
filebeat.prospectors:
- type: log
paths:
- /var/log/messages
output.console:
enable: true
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml //可以在屏幕上看到对应的日志信息
[[email protected] ~]# /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
^C[[email protected] ~]# /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
{"@timestamp":"2017-12-18T07:32:01.785Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"prospector":{"type":"log"},"beat":{"name":"linux-05.com","hostname":"linux-05.com","version":"6.0.0"},"message":"Dec 18 12:30:01 linux-05 systemd: Started Session 6 of user root.","source":"/var/log/messages","offset":66}
{"@timestamp":"2017-12-18T07:32:01.785Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"offset":133,"message":"Dec 18 12:30:01 linux-05 systemd: Starting Session 6 of user root.","prospector":{"type":"log"},"beat":{"version":"6.0.0","name":"linux-05.com","hostname":"linux-05.com"},"source":"/var/log/messages"}...........
再编辑配置文件
vim /etc/filebeat/filebeat.yml //增加或者更改
filebeat.prospectors:
- input_type: log
paths:
- /var/log/messages
output.elasticsearch:
hosts: ["192.168.137.30:9200"]
[[email protected] ~]# systemctl start filebeat
[[email protected] ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 933/sshd
tcp 0 0 192.168.137.45:27017 0.0.0.0:* LISTEN 1061/mongod
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1061/mongod
tcp6 0 0 :::3306 :::* LISTEN 1454/mysqld
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 192.168.137.45:9200 :::* LISTEN 888/java
tcp6 0 0 192.168.137.45:9300 :::* LISTEN 888/java
tcp6 0 0 :::22 :::* LISTEN 933/sshd
[[email protected] ~]# ps aux |grep filebeat
root 5123 0.0 1.2 277436 12324 ? Ssl 15:45 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebea
root 5140 0.0 0.0 112652 964 pts/3 R+ 15:46 0:00 grep --color=auto filebeat
[[email protected] ~]#
拓展
x-pack 收费，免费 http://www.jianshu.com/p/a49d93212eca
https://www.elastic.co/subscriptions
Elastic stack演进 http://70data.net/1505.html
基于kafka和elasticsearch，linkedin构建实时日志分析系统 http://t.cn/RYffDoE
使用redis http://blog.lishiming.net/?p=463
ELK+Filebeat+Kafka+ZooKeeper 构建海量日志分析平台 https://www.cnblogs.com/delgyd/p/elk.html
http://www.jianshu.com/p/d65aed756587
以上是关于ELK——01安装ELK的主要内容，如果未能解决你的问题，请参考以下文章
Centos6搭建ELK - 01 安装Elasticsearch7.13
ELK实战，Linux版docker安装ElasticSearchES-headLogstashKiabana入门，无坑详细图解