Kafka Server写数据的时候报错org.apache.kafka.common.errors.RecordTooLargeException

Posted 2020-08-05

向Kafka中输入数据，抛异常org.apache.kafka.common.errors.RecordTooLargeException

官网两个参数描述如下：

message.max.bytes

The maximum size of message that the server can receive

int

1000012

[0,...]

high

fetch.message.max.bytes

1024 * 1024

The number of byes of messages to attempt to fetch for each topic-partition in each fetch request. These bytes will be read into memory for each partition, so this helps control the memory used by the consumer. The fetch request size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch.

message.max.bytes：server能接受消息体的最大值。

fetch.message.max.bytes：consumer从partition中获取消息体放入内存中，这个参数控制conusmer所用的内存大小。如果message.max.bytes大于fetch.message.max.bytes，就会导致consumer分配的内存放不下一个message。

因此，在server.properties中添加两个配置项

#broker能接收消息的最大字节数
message.max.bytes=20000000
#broker可复制的消息的最大字节数
replica.fetch.max.bytes=20485760

如果不想修改配置文件，可以采用修改topic配置的方法，与server配置项message.max.bytes对应的topic配置项是max.message.bytes

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic
    --config max.message.bytes=128000

本文出自 “巧克力黑” 博客，请务必保留此出处http://10120275.blog.51cto.com/10110275/1844461