linux鍗曟満閮ㄧ讲kafka(filebeat+elk缁勫悎)
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了linux鍗曟満閮ㄧ讲kafka(filebeat+elk缁勫悎)相关的知识，希望对你有一定的参考价值。
鏍囩锛?a href='http://www.mamicode.com/so/1/plain' title='plain'>plain log local 鍚姩 none yml off grep -v event
filebeat+elk缁勫悎涔媖afka鍗曟満閮ㄧ讲
鍑嗗锛?/p>
kafka涓嬭浇閾炬帴鍦板潃锛歨ttp://kafka.apache.org/downloads.html
鍦ㄨ繖閲屼笅杞絢afka_2.12-2.10.0.0.tgz锛坘afka鍜寊ookeeper閮界敤鍚屼竴涓寘閲岀殑锛夈€?/p>
涓€銆佸畨瑁呭拰閰嶇疆jdk(涓嬭浇jdk锛岄厤缃幆澧冨嵆鍙?
JAVA_HOME=/opt/jdk1.8.0_131
CLASSPATH=.:$JAVA_HOME/lib.tools.jar
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) Server VM (build 25.131-b11, mixed mode)
鎴栬€呭湪bin/kafka-run-class.sh鎸囧畾kafka jdk 鐜鍙橀噺
vi bin/kafka-run-class.sh
JAVA_HOME=/opt/jdk1.8.0_131
浜屻€佸畨瑁?/strong>Kafka
1銆?/strong>瀹夎glibc
# yum -y install glibc.i686
2銆佽В鍘媖afka_2.12-2.10.0.0.tgz
鍏堥厤缃?/strong>zookeeper
$cd kafka_2.12-2.10.0.0
$vi config/zookeeper.properties
dataDir=/data/soft/kafka/data
dataLogDir=/data/soft/kafka/log
clientPort=2181
maxClientCnxns=100
tickTime=2000
initLimit=10
閰嶇疆鍚庣洿鎺ュ惎鍔▃ookeeper:
$bin/zookeeper-server-start.sh config/zookeeper.properties
濡傛灉娌℃湁鎶ラ敊,鍙互杞悗鍙板惎鍔?
$nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
鍐嶉厤缃?/strong>kafka
$ vi config/server.properties
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://server20.srv:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/log/kafka
num.partitions=2
num.recovery.threads.per.data.dir=1
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
鍚姩kafka锛?/span>
$ bin/kafka-server-start.sh config/server.properties
濡傛灉娌℃湁鎶ラ敊,鍙互杞悗鍙板惎鍔細
$nohup bin/kafka-server-start.sh config/server.properties &
妫€鏌ュ惎鍔ㄦ儏鍐碉細榛樿寮€鍚殑绔彛涓?181锛坺ookeeper锛夊拰9202锛坘afka锛夈€?/p>
3銆佹祴璇?/strong>kafka锛?/strong>
(1)銆佸垱寤簍opic
$bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
(2)銆佹煡鐪嬪垱寤虹殑topic
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test
(3)銆佺敓浜ф秷鎭祴璇曪紙妯℃嫙瀹㈡埛绔彂閫佹秷鎭級
$bin/kafka-console-producer.sh --broker-list 192.168.53.20:9092 --topic test
> ..hello world.. #杈撳叆鍐呭鍥炶溅
>
(4)銆佹秷璐规秷鎭祴璇曪紙妯℃嫙瀹㈡埛绔帴鏀朵俊鎭級
$bin/kafka-console-consumer.sh --bootstrap-server 192.168.53.20:9202 --topic test --from-beginning
..hello world.. #濡傛灉鑳芥甯告帴鏀跺埌淇℃伅璇存槑kafka閮ㄧ讲姝ｅ父
(5)銆佸垹闄opic
$bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
瀹屾垚浠ヤ笂浠ｈ〃kafka鍗曞寘鏈哄畨瑁呮垚鍔熴€?/p>
涓夈€侀厤缃?/strong>filebeat
filebeat.yml鏂囦欢娣诲姞閰嶇疆淇℃伅锛屾敞閲婃帀鍘熸潵鐨刲ogstash output銆?/p>
#------------------- Kafka output ---------------------
output.kafka:
hosts: ["server20.srv:9092"]
topic: 鈥榢afka_logstash鈥?/span>
鍥涖€侀厤缃?/strong>logstash
logstash.conf鏂囦欢娣诲姞閰嶇疆淇℃伅锛屾敞閲婃帀鍘熸潵inputbeats...銆?/p>
input
kafka
codec => "json"
bootstrap_servers => "server20.srv:9092"
topics => ["kafka_logstash"]
group_id => "kafka-consumer-group"
decorate_events => true
auto_offset_reset => "latest"
鍦╨ogstash鏈嶅姟鍣ㄤ笂閰嶇疆濂絢afka璁块棶鍦板潃锛?/p>
$ cat /etc/hosts
122.9.10.106 server20.srv 8bet-kafka
浜斻€?/strong>kafka鐩稿叧閰嶇疆鏂囦欢鍙傝€?/strong>
$ cat config/server.properties | egrep -v 鈥榐$|#鈥?/p>
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://server20.srv:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/log/kafka
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
$cat config/zookeeper.properties | egrep -v 鈥榐$|#鈥?/p>
dataDir=/data/soft/kafka/data
dataLogDir=/data/soft/kafka/zookeeper_log
clientPort=2181
maxClientCnxns=100
tickTime=2000
initLimit=10
$cat config/producer.properties | egrep -v 鈥榐$|#鈥?/p>
bootstrap.servers=localhost:9092
compression.type=none
$cat config/consumer.properties | egrep -v 鈥榐$|#鈥?/p>
bootstrap.servers=localhost:9092
group.id=kafka-consumer-group
鍏€侀厤缃畬鍚庢祴璇曟秷璐规秷鎭繛閫氾紝濡傛灉鎺ュ彈姝ｅ父锛屽垯鎴愬姛
$bin/kafka-console-consumer.sh --bootstrap-server server20.srv:9202 --topic test --from-beginning
以上是关于linux鍗曟満閮ㄧ讲kafka(filebeat+elk缁勫悎)的主要内容，如果未能解决你的问题，请参考以下文章