ES+Logstash

Posted 2020-09-15

Logstash日志收集实践

先了解以下几个基本概念:
logstash收集日志基本流程: input-->codec-->filter-->codec-->output
1.input:从哪里收集日志。
2.filter:发出去前进行过滤
3.output:输出至Elasticsearch或Redis消息队列
4.codec:输出至前台，方便边实践边测试
5.数据量不大日志按照月来进行收集

#通常使用rubydebug方式前台输出展示以及测试
[[email protected] ～]# /opt/logstash/bin/logstash -e ‘input { stdin {} } output { stdout{codec => rubydebug} }‘
hello #输入
{
"message" => "hello",
"@version" => "1",
"@timestamp" => "2016-12-10T08:16:36.354Z",
"host" => "linux-Tcat1.com"
}

Logstash安装

Logstash需要Java环境，所以直接使用yum安装。

1.安装java

[[email protected] ~]# yum install java
[[email protected] ~]# java -version
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

2.下载并安装GPG key

[[email protected] ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

3.添加logstash的yum仓库

#添加logstash的yum仓库
[[email protected] ~]# cat /etc/yum.repos.d/logstash.repo
[logstash-2.3]
name=Logstash repository for 2.3.x packages
baseurl=https://packages.elastic.co/logstash/2.3/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

安装Logstash

[[email protected] ~]# yum install -y logstash
声明:如果需要前台查看测试结果，在output加入如下:

stdout {
codec => "rubydebug"
}
#执行命令：
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/syslog.conf
#执行完毕，将文件放置/etc/logstash/conf.d目录,logstash会自动读取相关配置文件
如果无法读取，可将/etc/init.d/logstash里USER和GROUP修改为root