storm trident 一个batch多大

Posted 2020-11-08 brainstorm

You can increase the batch size by changing "tridentKafkaConfig.fetchSizeBytes" property.

Also, batch size is related with number of brokers and number of partitions.

For example, if you have 2 brokers and 3 partitions for each broker that means the total count of partition is 6.

y this way, the batch size equals to tridentKafkaConfig.fetchSizeBytes X total partition count.

if we assume that the tridentKafkaConfig.fetchSizeBytes is 1024X1024, the batch size equals to 6 MB.(3x2x1024x1024)bytes

相当于从kafka里面是批量读取的，然后形成storm中的一个batch
而且spout里面有一个buffer，应该也算是一个缓存，将正在处理的一部分数据，存在内存中，直到处理完