Apache Kafka - Quick Start on Windows
Posted 银雪飞马
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Apache Kafka - Quick Start on Windows相关的知识,希望对你有一定的参考价值。
在这篇文章中,我将要介绍如何搭建和使用Apache Kafka在windows环境。在开始之前,简要介绍一下Kafka,然后再进行实践。
Apache Kafka
Kafka是分布式的发布-订阅消息的解决方案。相比于传统的消息系统,Kafka快速,可扩展,耐用。想象一下传统的发布-订阅消息系统,producers产生/写消息到topic中,另一边,consumers从topic中消费/读消息。Kafka的topic可以在多个服务器之间分区(partition)和复制(replicate)。
可以得到更多细节信息从Kafka官网。
我参考了这篇博客(http://blog.cloudera.com/blog/2014/09/apache-kafka-for-beginners/)。它简单并很好的解释了Kafka是什么。这两张图片也取自同一篇博客。
"Messages are simply byte arrays and the developers can use them to store any object in any format – with String, JSON, and Avro the most common. It is possible to attach a key to each message, in which case the producer guarantees that all messages with the same key will arrive to the same partition. When consuming from a topic, it is possible to configure a consumer group with multiple consumers. Each consumer in a consumer group will read messages from a unique subset of partitions in each topic they subscribe to, so each message is delivered to one consumer in the group, and all messages with the same key arrive at the same consumer."
“信息只不过是简单的字节数组,开发人员可以用它们来存储任何对象用任何格式--String,JSON,Avro是最常用的。可以给每个消息附上一个键,这样producer可以保证拥有相同键的消息到达相同的分区。当从一个topic消费信息时,可以配置一个消费组拥有多个消费者。在消费组里的每个消费者从订阅的topic的partition中读取唯一的一段消息,所以每个消息交付给组里的一个consumer,而且拥有相同键的所有消息到达同一个consumer。”
"What makes Kafka unique is that Kafka treats each topic partition as a log (an ordered set of messages). Each message in a partition is assigned a unique offset. Kafka does not attempt to track which messages were read by each consumer and only retain unread messages; rather, Kafka retains all messages for a set amount of time, and consumers are responsible to track their location in each log. Consequently, Kafka can support a large number of consumers and retain large amounts of data with very little overhead."
“使Kafka独特的是Kafka把每个topic分区当做一条日志来处理(一组有序的消息)。在一个分区当中的每一条消息被分配一个唯一的偏移量。Kafka并不试图追踪哪些消息被consumer读取,而是保留未被读取的消息;而且,Kafka保留了所有消息的时间设定量,consumer负责追踪在每一个log中他们的位置。因此,Kafka可以支持众多的消费者,保留大量的数据,只用了非常小的开销。”
现在你要问了,“怎么在Windows上设置Kafka环境?”。不必着急,我通过简单的几步来引导你。
log.dirs=<kafka_dir>\\kafka-logs
dataDir=<kafka_dir>\\zookeeper-data
<kafka_dir>\\bin\\windows\\zookeeper-server-start.bat ..\\..\\config\\zookeeper.properties
<kafka_dir>\\bin\\windows\\kafka-server-start.bat ..\\..\\config\\server.properties
<kafka_dir>\\bin\\windows\\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic
Created topic "mytopic".
<kafka_dir>\\bin\\windows\\kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
<kafka_dir>\\bin\\windows\\kafka-console-consumer.bat --zookeeper localhost:2181 --topic mytopic
以上是关于Apache Kafka - Quick Start on Windows的主要内容,如果未能解决你的问题,请参考以下文章