ELK Kafka json to elk

Posted 2020-07-04 MOSS

Logstash配置

   

input {

kafka {

zk_connect => "127.0.0.1:2181"

topic_id => "cluster"

codec => plain

reset_beginning => false

consumer_threads => 5

decorate_events => true

}

}

   

   

output {

if [type]=="cluster3" or [type]=="cluster2" or [type]=="clusterjson"

{

elasticsearch {

hosts => ["localhost:9200"]

index => "test-kafka-%{type}-%{+YYYY-MM}"

}

}

   

stdout { codec => rubydebug }

}

   

Server.properties 主要内容

broker.id=0

   

############################# Socket Server Settings #############################

   

listeners=PLAINTEXT://:9092

   

# The port the socket server listens on

port=9092

   

   

Server-1.properties 主要内容

   

broker.id=1

   

############################# Socket Server Settings #############################

   

listeners=PLAINTEXT://:9093

   

# The port the socket server listens on

port=9093

   

   

启动kafka,server.properties和server-1.properties

E:\\soft\\elk\\kafka_2.11-0.9.0.1>bin\\windows\\kafka-server-start.bat config\\custom\\server-1.properties

   

创建topic,并查看状态

   

   

查找端口占用情况，并删除一个节点

   

   

查看 topic状态

   

   

   

生产者输入json数据，如下

   

{"Name":"BULK 7","Data":7,"CreateTime":"2016-04-05T10:16:22Z","type":"cluster3"}

   

Kibana 4显示如下