ELKStack之消息队列

Posted 2020-09-25 minseo

redis消息队列

安装redis

yum -y install redis

修改配置文件

修改ip

后台运行

启动

systemctl start redis

查看

lsof -i:6379

连接

redis-cli -h 10.13.85.9

 

cd /etc/logstash/conf.d/

vim redis.conf

input{
    stdin {}
}

output{
    redis{
    	host => "10.13.85.9"
	port => "6379"
	db => "6"
	data_type => "list"
	key => "demo"
    }
}

启动

/opt/logstash/bin/logstash -f redis.conf

另外开一个窗口启动redis连接

redis-cli -h 10.13.85.9

select 6

验证可以写一个收集apache日志的配置文件

vim apache.conf

input{
    file{
        path => "/var/log/httpd/access_log"
        start_position => "beginning"
    }
}
output{
    redis{
        host => "10.13.85.9"
        port => "6379"
        db => "6"
        data_type => "list"
        key => "apache-accesslog"
    }
}

启动

/opt/logstash/bin/logstash -f apache.conf

查看最好一行

 

生产中可以在另外一台服务器启动一个logstash收集redis里面的数据

在另外服务器上面

vim /etc/logstash/conf.d/indexer.conf

input{
    redis{
    	host => "10.13.85.9"
	port => "6379"
	db => "6"
	data_type => "list"
	key => "demo"
    }

}

output{
    stdout{
    	codec => rubydebug
    }
}

启动如果成功了加filter处理apache

 

input{
    redis{
    	host => "10.13.85.9"
	port => "6379"
	db => "6"
	data_type => "list"
	key => "demo"
    }
}

filter{
    grok{
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }

}


output{
    elasticsearch{
    	hosts => ["10.13.85.9:9200"]
	index => "apache-accesslog-%{+YYYY.MM.dd}"
    }
}

启动

/opt/logstash/bin/logstash  -f indexer.conf

作业：消息队列kafka