kafka查看消费了多少条数据
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了kafka查看消费了多少条数据相关的知识,希望对你有一定的参考价值。
参考技术A 前面应该还有个数据生产者,比如flume. flume负责生产数据,发送至kafka。 spark streaming作为消费者,实时的从kafka中获取数据进行计算。 计算结果保存至redis,供实时推荐使用。 flume+kafka+spark+redis是实时数据收集与计算的一套经典架构...查看storm消费了多少kafka的数据
参考技术A 基于0.8.0版本。##查看topic分布情况kafka-list-topic.shbin/kafka-list-topic.sh --zookeeper 192.168.197.170:2181,192.168.197.171:2181 (列出所有topic的分区情况)bin/kafka-list-topic.sh --zookeeper 192.168.197.170:2181,192.168.197.171:2181 --topic test (查看test的分区情况)其实kafka-list-topic.sh里面就一句 exec $(dirname $0)/kafka-run-class.sh kafka.admin.ListTopicCommand $@实际是通过kafka-run-class.sh脚本执行的包kafka.admin下面的类##创建TOPIC kafka-create-topic.shbin/kafka-create-topic.sh --replica 2 --partition 8 --topic test --zookeeper 192.168.197.170:2181,192.168.197.171:2181创建名为test的topic, 8个分区分别存放数据,数据备份总共2份bin/kafka-create-topic.sh --replica 1 --partition 1 --topic test2 --zookeeper 192.168.197.170:2181,192.168.197.171:2181结果 topic: test2 partition: 0 leader: 170 replicas: 170 isr: 170##重新分配分区kafka-reassign-partitions.sh这个命令可以分区指定到想要的--broker-list上bin/kafka-reassign-partitions.sh --topics-to-move-json-file topics-to-move.json --broker-list "171" --zookeeper 192.168.197.170:2181,192.168.197.171:2181 --execute cat topic-to-move.json"topics": ["topic": "test2"], "version":1##为Topic增加 partition数目kafka-add-partitions.shbin/kafka-add-partitions.sh --topic test --partition 2 --zookeeper 192.168.197.170:2181,192.168.197.171:2181 (为topic test增加2个分区)##控制台接收消息bin/kafka-console-consumer.sh --zookeeper 192.168.197.170:2181,192.168.197.171:2181 --from-beginning --topic test##控制台发送消息bin/kafka-console-producer.sh --broker-list 192.168.197.170:9092,192.168.197.171: 9092 --topic test ##手动均衡topic, kafka-preferred-replica-election.shbin/kafka-preferred-replica-election.sh --zookeeper 192.168.197.170:2181,192.168.197.171:2181 --path-to-json-file preferred-click.jsoncat preferred-click.json "partitions": [ "topic": "click", "partition": 0, "topic": "click", "partition": 1, "topic": "click", "partition": 2, "topic": "click", "partition": 3, "topic": "click", "partition": 4, "topic": "click", "partition": 5, "topic": "click", "partition": 6, "topic": "click", "partition": 7, "topic": "play", "partition": 0, "topic": "play", "partition": 1, "topic": "play", "partition": 2, "topic": "play", "partition": 3, "topic": "play", "partition": 4, "topic": "play", "partition": 5, "topic": "play", "partition": 6, "topic": "play", "partition": 7 ]##删除topic,慎用,只会删除zookeeper中的元数据,消息文件须手动删除bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic test666 --zookeeper 192.168.197.170:2181 ,192.168.197.171:2181以上是关于kafka查看消费了多少条数据的主要内容,如果未能解决你的问题,请参考以下文章