ELK部署logstash

Posted 2023-03-05 LK丶旋律

Logstash 是一个开源的数据收集引擎，可以水平伸缩，而且 logstash 整个 ELK
当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指
定的且可以是多个不同目的地。

https://github.com/elastic/logstash
https://baike.baidu.com/item/Ruby/11419 #基于 ruby 开发

安装jdk8

[root@logstash1 ~]# apt install openjdk-8-jdk -y

使用dpkg安装

[root@logstash1 src]# dpkg -i logstash-7.12.1-amd64.deb

启动

[root@logstash1 src]# systemctl start logstash.service

查看状态

[root@logstash1 src]# systemctl status logstash.service

测试

[root@logstash1 src]# /usr/share/logstash/bin/logstash -e 'input  stdin  output  stdout codec => rubydebug '

123 # 手动输入
# 返回数据，即可

          "host" => "logstash1.example.local",
       "message" => "123",
      "@version" => "1",
    "@timestamp" => 2021-08-24T06:48:28.756Z

测试输出到 elasticsearch：

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -e 'input  stdin  output  elasticsearch hosts => ["172.31.2.101:9200"] index => "mytest-%+YYYY.MM.dd"                       '

收集日志

通过 logstash 收集日志

收集单个系统日志并输出至文件

[root@logstash1 ~]# vim /etc/logstash/conf.d/system.conf

input 
    stdin 
       type => stdin
   

output 
    file 
       path => "/tmp/linux104.log"
   

执行测试语法

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t

重启

[root@logstash1 ~]# systemctl restart logstash.service

或者执行

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf

查看

[root@logstash1 ~]# ll /tmp/linux104.log

-rw-r--r-- 1 root root 121 Aug 24 16:40 /tmp/linux104.log

写入es

[root@logstash1 ~]# vim /etc/logstash/conf.d/system.con

input 
    file 
       path => "/var/log/syslog"
   


output 
    elasticsearch 
       hosts => ["172.31.2.102:9200","172.31.2.103:9200"]
       index => "long-system-syslog-%+YYYY.MM.dd"
   

重启

[root@logstash1 ~]# systemctl restart logstash.service

或者执行

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf

把收集多个日志一起写入es集群

[root@logstash1 ~]# cat /etc/logstash/conf.d/es-muber-test.conf

input 
   file 
       path => "/var/log/syslog"
       start_position => "beginning"
       stat_interval => 3
       type => "syslog"
   

   file 
     path => "/var/log/bootstrap.log"
     start_position => "beginning"
     stat_interval => 3
     type => "bootstrap"
   


output 
    if [type] == "syslog"
    elasticsearch 
       hosts => ["172.31.2.101:9200","172.31.2.102:9200"]
       index => "long-system-syslog-%+YYYY.MM.dd"
   

   if [type] == "bootstrap"
     elasticsearch 
        hosts => ["172.31.2.101:9200"]
        index => "long-bootstrap-log-%+YYYY.MM.dd"
   

重启

[root@logstash1 ~]# systemctl restart logstash.service

或者执行

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/es-muber-test.conf

把index索引添加到kibana