Apache Kafka

Posted 2021-01-03 yvhqbat

Apache Kafka 是一个分布式数据流平台。

Apache Kafka

• http://kafka.apache.org/documentation

• kafka 基本概念

• Apache Kafka在大型应用中的20项最佳实践

• 分布式发布订阅消息系统 Kafka 架构设计

• Kafka 在InfoQ上的内容

• 消息中间件 Kafka介绍及安装部署

• Kafka学习之路

• Docker wurstmeister/kafka

1. GETTING STARTED

1.1 Introduction 简介

Apache Kafka® is a distributed streaming platform. What exactly does that mean?
Apache Kafka 是一个分布式数据流平台。什么意思呢？

A streaming platform has three key capabilities:
一个数据流平台具备3个主要功能：

1. Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
2. Store streams of records in a fault-tolerant durable way.
3. Process streams of records as they occur.

Kafka is generally used for two broad classes of applications:
Kafka 通常用于2中应用：

1. Building real-time streaming data pipelines that reliably get data between systems or applications
2. Building real-time streaming applications that transform or react to the streams of data

To understand how Kafka does these things, let\'s dive in and explore Kafka\'s capabilities from the bottom up.

First a few concepts:
首先是几个概念：

1. Kafka is run as a cluster on one or more servers that can span multiple datacenters.
2. The Kafka cluster stores streams of records in categories called topics.
3. Each record consists of a key, a value, and a timestamp.

Kafka has four core APIs:
Kafka 有四个核心APIs:

1. The Producer API allows an application to publish a stream of records to one or more Kafka topics.
2. The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
3. The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
4. The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.

In Kafka the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in many languages.

• librdkafka - the Apache Kafka C/C++ client library
• Sarama - an MIT-licensed Go client library for Apache Kafka version 0.8 (and later)