kafka-伪分布部署

Posted 2021-09-21 星河scorpion

安装kafka伪集群模式
命令参考:https://www.orchome.com/454
主题分区配置参考@KafkaListener https://www.orchome.com/6805
advertised.listeners参数的重要性(外部访问局域网kafka):https://blog.csdn.net/lsr40/article/details/84135959
https://segmentfault.com/a/1190000020715650

advertised.listeners配置的是外网ip与端口

1.开启zookeeper
前台运行:bin/zookeeper-server-start.sh config/zookeeper.properties
后台运行:nohup ./zookeeper-server-start.sh ../config/zookeeper.properties > zookeeper-run.log 2>&1 &

2.关闭zookeeper
bin/zookeeper-server-stop.sh

3.开启kafka
前台运行:bin/kafka-server-start.sh config/server.properties
后台运行:nohup ./kafka-server-start.sh ../config/server.properties > kafka-0-run.log 2>&1 &
nohup ./kafka-server-start.sh ../config/server1.properties > kafka-1-run.log 2>&1 &
nohup ./kafka-server-start.sh ../config/server2.properties > kafka-2-run.log 2>&1 &

伪分布模式:
config/server1.properties:
    broker.id=0
    listeners=PLAINTEXT://192.168.10.130:9092
    log.dirs=kafka-logs
    zookeeper.connect=localhost:2181
    
config/server-1.properties: 
    broker.id=1
    listeners=PLAINTEXT://192.168.10.130:9093
    log.dirs=kafka-logs-1
    zookeeper.connect=localhost:2181

config/server-2.properties: 
    broker.id=2
    listeners=PLAINTEXT://192.168.10.130:9094
    log.dirs=kafka-logs-2
    zookeeper.connect=localhost:2181

4.创建topic
./kafka-topics.sh --create --zookeeper localhost:2181 --config max.message.bytes=12800000 --config flush.messages=1 --replication-factor 1 --partitions 1 --topic test

5.查看搜索topic
./kafka-topics.sh --list --zookeeper localhost:2181

6.删除主题
在server.properties中增加设置，默认未开启
delete.topic.enable=true

删除主题命令
/bin/kafka-topics --delete --topic test --zookeeper localhost:2181

4.关闭kafka
bin/kafka-server-stop.sh

canal 设置是否json开启
canal.mq.flatMessage =false //是否为flat json格式对象

canal-kafka 数据乱码解决方案
https://www.cnblogs.com/createweb/p/10523780.html

github:
https://github.com/alibaba/canal/tree/master/client/src/main/java/com/alibaba/otter/canal/client/kafka