pulsar的Kop压力测试

Posted sqtce

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了pulsar的Kop压力测试相关的知识,希望对你有一定的参考价值。

背景

kop服务器配置:cpu: 32C 内存: 128G  硬盘:3.7T (8台机器)

zk服务配置:cpu: 4C 内存: 8G 硬盘:40G (3台)

部署:zk部署在3台机器上,kop分别部署在8台机器上

目的:对Pulsar集群的kop进行性能压力测试,

 

总结

1,10万tps,消息体1k,最大tps11万 (40和80分区一样)

2,100万tps,消息体1k,最大tps30万(80分区)

3,10万tps,消息体1.5k,最大tps22万(40分区)

4,100万tps,消息体1.5k,最大tps21万(80分区)

5,10万tps,消息体2k,最大tps17万(40分区))

6,100万tps,消息体2k,最大tps17万(80分区)

 

命令注解:

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 100000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
# --nurn-records:用来指定发送消息的总条数  # --record-size:用来设置每条消息的字节数 
# --throughput:用来进行限流控制,当设定的值小于0不限流,当设定的值大于0时,当发送的吞吐量大于该值时就会被阻塞一段时间。 
# --producer-props:参数用来指定生产者的配置,可同时指定多组配置,各组配置之间以空格分隔,与producer-props 参数对应的还有一个producer-config参数,它用来指定生产者的配置文件


100000 records sent, 111856.823266 records/sec (106.67 MB/sec), 133.69 ms avg latency, 366.00 ms max latency, 133 ms 50th, 188 ms 95th, 207 ms 99th, 217 ms 99.9th.
#  records sent:表示测试时发送的消息总数
#  records sec:表示以每秒发迭的消息数来统计吞吐量,括号中 MB/sec 表示以每秒发送的消息大小来统计吞吐量,
#  avg latency: 表示消息处理的平均耗时
#  max latency:表示消息处理的最大耗时 
#  50th,95th,99t,99.9th: 分别表示 50% 95 99% 99.9% 的消息处理耗时。
一,10万tps,消息体1k

1,40个分区 tps11万

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 100000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
100000 records sent, 111856.823266 records/sec (106.67 MB/sec), 133.69 ms avg latency, 366.00 ms max latency, 133 ms 50th, 188 ms 95th, 207 ms 99th, 217 ms 99.9th.

2,80个分区 tps11万

bin/kafka-producer-perf-test.sh --topic a-topic --num-records 100000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
100000 records sent, 115473.441109 records/sec (110.12 MB/sec), 119.06 ms avg latency, 417.00 ms max latency, 118 ms 50th, 172 ms 95th, 190 ms 99th, 202 ms 99.9th.
二,100万tps,消息体1k

1,40个分区tps29.6万

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 296735.905045 records/sec (282.99 MB/sec), 93.82 ms avg latency, 369.00 ms max latency, 94 ms 50th, 163 ms 95th, 192 ms 99th, 218 ms 99.9th.

2,80个分区tps30万

 bin/kafka-producer-perf-test.sh --topic a-topic --num-records 1000000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 306748.466258 records/sec (292.54 MB/sec), 91.04 ms avg latency, 390.00 ms max latency, 87 ms 50th, 185 ms 95th, 230 ms 99th, 260 ms 99.9th.

3,160个分区tps29万

bin/kafka-producer-perf-test.sh --topic b-topic --num-records 1000000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 292740.046838 records/sec (279.18 MB/sec), 95.75 ms avg latency, 413.00 ms max latency, 86 ms 50th, 232 ms 95th, 302 ms 99th, 367 ms 99.9th.

 

4,240个分区tps27.7万

bin/kafka-producer-perf-test.sh --topic c-topic --num-records 1000000 --record-size 1000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 277392.510402 records/sec (264.54 MB/sec), 100.46 ms avg latency, 612.00 ms max latency, 29 ms 50th, 370 ms 95th, 487 ms 99th, 562 ms 99.9th.

 

三,10万tps,消息体1.5k

 

1,40个分区tps22万

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1500  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 224618.149146 records/sec (321.32 MB/sec), 81.58 ms avg latency, 364.00 ms max latency, 78 ms 50th, 161 ms 95th, 188 ms 99th, 205 ms 99.9th.

2,80个分区tps9万

 bin/kafka-producer-perf-test.sh --topic a-topic --num-records 100000 --record-size 1500  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
100000 records sent, 93632.958801 records/sec (133.94 MB/sec), 122.44 ms avg latency, 408.00 ms max latency, 116 ms 50th, 217 ms 95th, 249 ms 99th, 280 ms 99.9th.
四,100万tps,消息体1.5k

1,40个分区tps21万

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1500  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 210304.942166 records/sec (300.84 MB/sec), 87.28 ms avg latency, 366.00 ms max latency, 77 ms 50th, 216 ms 95th, 274 ms 99th, 322 ms 99.9th.

2,80个分区 tps22万

bin/kafka-producer-perf-test.sh --topic a-topic --num-records 1000000 --record-size 1500  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 221043.324492 records/sec (316.21 MB/sec), 82.24 ms avg latency, 418.00 ms max latency, 78 ms 50th, 171 ms 95th, 212 ms 99th, 249 ms 99.9th.

3,160个分区tps21万

bin/kafka-producer-perf-test.sh --topic b-topic --num-records 1000000 --record-size 1500  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
1000000 records sent, 217959.895379 records/sec (311.79 MB/sec), 83.91 ms avg latency, 527.00 ms max latency, 56 ms 50th, 232 ms 95th, 306 ms 99th, 481 ms 99.9th.

 

4,240个分区tps21万

bin/kafka-producer-perf-test.sh --topic c-topic --num-records 1000000 --record-size 1500  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092

1000000 records sent, 215146.299484 records/sec (307.77 MB/sec), 83.36 ms avg latency, 693.00 ms max latency, 21 ms 50th, 391 ms 95th, 526 ms 99th, 629 ms 99.9th.

 

五,10万tps,消息体2k

 

1,40个分区 tps17.7万

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 2000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
867234 records sent, 173412.1 records/sec (330.76 MB/sec), 85.5 ms avg latency, 365.0 ms max latency.
1000000 records sent, 176678.445230 records/sec (336.99 MB/sec), 84.94 ms avg latency, 365.00 ms max latency, 80 ms 50th, 163 ms 95th, 213 ms 99th, 268 ms 99.9th.

2,80个分区tps8万

 bin/kafka-producer-perf-test.sh --topic a-topic --num-records 100000 --record-size 2000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
100000 records sent, 82508.250825 records/sec (157.37 MB/sec), 117.56 ms avg latency, 410.00 ms max latency, 119 ms 50th, 180 ms 95th, 207 ms 99th, 229 ms 99.9th.
六,100万tps,消息体2k

1,40个分区tps17万

bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 2000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092

848922 records sent, 169750.4 records/sec (323.77 MB/sec), 87.3 ms avg latency, 355.0 ms max latency.
1000000 records sent, 174125.021766 records/sec (332.12 MB/sec), 86.40 ms avg latency, 355.00 ms max latency, 86 ms 50th, 185 ms 95th, 209 ms 99th, 241 ms 99.9th.

2,80个分区tps18万

 bin/kafka-producer-perf-test.sh --topic a-topic --num-records 1000000 --record-size 2000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
882894 records sent, 176578.8 records/sec (336.80 MB/sec), 84.2 ms avg latency, 386.0 ms max latency.
1000000 records sent, 181093.806592 records/sec (345.41 MB/sec), 82.92 ms avg latency, 386.00 ms max latency, 73 ms 50th, 182 ms 95th, 219 ms 99th, 277 ms 99.9th.

3,160个分区tps18万

 bin/kafka-producer-perf-test.sh --topic b-topic --num-records 1000000 --record-size 2000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092
890341 records sent, 178068.2 records/sec (339.64 MB/sec), 82.2 ms avg latency, 539.0 ms max latency.
1000000 records sent, 180929.980098 records/sec (345.10 MB/sec), 82.08 ms avg latency, 539.00 ms max latency, 51 ms 50th, 250 ms 95th, 358 ms 99th, 477 ms 99.9th.

 

4,240个分区tps17万

bin/kafka-producer-perf-test.sh --topic c-topic --num-records 1000000 --record-size 2000  --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092

840520 records sent, 168003.2 records/sec (320.44 MB/sec), 86.3 ms avg latency, 749.0 ms max latency.
1000000 records sent, 170183.798502 records/sec (324.60 MB/sec), 87.21 ms avg latency, 753.00 ms max latency, 20 ms 50th, 383 ms 95th, 543 ms 99th, 678 ms 99.9th.

 

 

 

以上是关于pulsar的Kop压力测试的主要内容,如果未能解决你的问题,请参考以下文章

腾讯宣布开源 RoP:Apache Pulsar 支持原生 RocketMQ 协议

腾讯宣布开源 RoP:Apache Pulsar 支持原生 RocketMQ 协议

[pulsar] pulsar go sdk 测试

03_Apache Pulsar的Local与分布式集群构建Pulsar的分布式集群模式Pulsar的分布式集群模式构建启动测试

Pulsar 中的意外积压大小

修改开放消息以测试 Pulsar 的自定义独立安装时面临的问题