5 Kafka Streams exactly once semantic

Posted 2021-02-12 crazy-chinese

About Exactly Once Semantic

Since Kafka 0.11 version Kafka support Exactly Once Semantic, which means, that Kafka can give you the grarantee that each messege can be exactly once consumed/produced to/from Kafka.

There are a lot of engineering details, which I wouldn‘t explain here... In this section I want to keep it simple.

Since 0.11 version there is a new implementation for this Exactly Once Semantic, which is idempotent !

It means if a producer send the same messeage twice or retries, Kafka will make sure to only keep one copy of it !

For example in a typically process for Producer:

• producer send a messeage to Kafka
• Kafka receive the messege
• and send back a Acknowledgement to Producer
• Producer commit Offset (done)

in step two, if Kafka just receive the messege and then lost the network connection, Kafka will not be able to send back the Acknowledgement. In this case Producer will send the messege again by retry. Then will Kafka receive the this messege at this moment twoice. But with Idempotent it will only keep one copy of them.

How to do exactly once in Kafka Streams

Very simple, only add one more line of code in configuration property file:

PS: with this feature it could slow down Kafka a little bit.. but it‘s fine :-)

You can also fine tune the setting using commit.interval.ms