pulsar的Kop压力测试
Posted sqtce
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了pulsar的Kop压力测试相关的知识,希望对你有一定的参考价值。
背景kop服务器配置:cpu: 32C 内存: 128G 硬盘:3.7T (8台机器)
zk服务配置:cpu: 4C 内存: 8G 硬盘:40G (3台)
部署:zk部署在3台机器上,kop分别部署在8台机器上
目的:对Pulsar集群的kop进行性能压力测试,
总结
1,10万tps,消息体1k,最大tps11万 (40和80分区一样)
2,100万tps,消息体1k,最大tps30万(80分区)
3,10万tps,消息体1.5k,最大tps22万(40分区)
4,100万tps,消息体1.5k,最大tps21万(80分区)
5,10万tps,消息体2k,最大tps17万(40分区))
6,100万tps,消息体2k,最大tps17万(80分区)
命令注解:
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 100000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 # --nurn-records:用来指定发送消息的总条数 # --record-size:用来设置每条消息的字节数 # --throughput:用来进行限流控制,当设定的值小于0不限流,当设定的值大于0时,当发送的吞吐量大于该值时就会被阻塞一段时间。 # --producer-props:参数用来指定生产者的配置,可同时指定多组配置,各组配置之间以空格分隔,与producer-props 参数对应的还有一个producer-config参数,它用来指定生产者的配置文件 100000 records sent, 111856.823266 records/sec (106.67 MB/sec), 133.69 ms avg latency, 366.00 ms max latency, 133 ms 50th, 188 ms 95th, 207 ms 99th, 217 ms 99.9th. # records sent:表示测试时发送的消息总数 # records sec:表示以每秒发迭的消息数来统计吞吐量,括号中 MB/sec 表示以每秒发送的消息大小来统计吞吐量, # avg latency: 表示消息处理的平均耗时 # max latency:表示消息处理的最大耗时 # 50th,95th,99t,99.9th: 分别表示 50% 95 99% 99.9% 的消息处理耗时。一,10万tps,消息体1k
1,40个分区 tps11万
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 100000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 100000 records sent, 111856.823266 records/sec (106.67 MB/sec), 133.69 ms avg latency, 366.00 ms max latency, 133 ms 50th, 188 ms 95th, 207 ms 99th, 217 ms 99.9th.
2,80个分区 tps11万
bin/kafka-producer-perf-test.sh --topic a-topic --num-records 100000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 100000 records sent, 115473.441109 records/sec (110.12 MB/sec), 119.06 ms avg latency, 417.00 ms max latency, 118 ms 50th, 172 ms 95th, 190 ms 99th, 202 ms 99.9th.二,100万tps,消息体1k
1,40个分区tps29.6万
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 296735.905045 records/sec (282.99 MB/sec), 93.82 ms avg latency, 369.00 ms max latency, 94 ms 50th, 163 ms 95th, 192 ms 99th, 218 ms 99.9th.
2,80个分区tps30万
bin/kafka-producer-perf-test.sh --topic a-topic --num-records 1000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 306748.466258 records/sec (292.54 MB/sec), 91.04 ms avg latency, 390.00 ms max latency, 87 ms 50th, 185 ms 95th, 230 ms 99th, 260 ms 99.9th.
3,160个分区tps29万
bin/kafka-producer-perf-test.sh --topic b-topic --num-records 1000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 292740.046838 records/sec (279.18 MB/sec), 95.75 ms avg latency, 413.00 ms max latency, 86 ms 50th, 232 ms 95th, 302 ms 99th, 367 ms 99.9th.
4,240个分区tps27.7万
bin/kafka-producer-perf-test.sh --topic c-topic --num-records 1000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 277392.510402 records/sec (264.54 MB/sec), 100.46 ms avg latency, 612.00 ms max latency, 29 ms 50th, 370 ms 95th, 487 ms 99th, 562 ms 99.9th.
三,10万tps,消息体1.5k
1,40个分区tps22万
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1500 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 224618.149146 records/sec (321.32 MB/sec), 81.58 ms avg latency, 364.00 ms max latency, 78 ms 50th, 161 ms 95th, 188 ms 99th, 205 ms 99.9th.
2,80个分区tps9万
bin/kafka-producer-perf-test.sh --topic a-topic --num-records 100000 --record-size 1500 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 100000 records sent, 93632.958801 records/sec (133.94 MB/sec), 122.44 ms avg latency, 408.00 ms max latency, 116 ms 50th, 217 ms 95th, 249 ms 99th, 280 ms 99.9th.四,100万tps,消息体1.5k
1,40个分区tps21万
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1500 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 210304.942166 records/sec (300.84 MB/sec), 87.28 ms avg latency, 366.00 ms max latency, 77 ms 50th, 216 ms 95th, 274 ms 99th, 322 ms 99.9th.
2,80个分区 tps22万
bin/kafka-producer-perf-test.sh --topic a-topic --num-records 1000000 --record-size 1500 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 221043.324492 records/sec (316.21 MB/sec), 82.24 ms avg latency, 418.00 ms max latency, 78 ms 50th, 171 ms 95th, 212 ms 99th, 249 ms 99.9th.
3,160个分区tps21万
bin/kafka-producer-perf-test.sh --topic b-topic --num-records 1000000 --record-size 1500 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 217959.895379 records/sec (311.79 MB/sec), 83.91 ms avg latency, 527.00 ms max latency, 56 ms 50th, 232 ms 95th, 306 ms 99th, 481 ms 99.9th.
4,240个分区tps21万
bin/kafka-producer-perf-test.sh --topic c-topic --num-records 1000000 --record-size 1500 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 1000000 records sent, 215146.299484 records/sec (307.77 MB/sec), 83.36 ms avg latency, 693.00 ms max latency, 21 ms 50th, 391 ms 95th, 526 ms 99th, 629 ms 99.9th.
五,10万tps,消息体2k
1,40个分区 tps17.7万
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 2000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 867234 records sent, 173412.1 records/sec (330.76 MB/sec), 85.5 ms avg latency, 365.0 ms max latency. 1000000 records sent, 176678.445230 records/sec (336.99 MB/sec), 84.94 ms avg latency, 365.00 ms max latency, 80 ms 50th, 163 ms 95th, 213 ms 99th, 268 ms 99.9th.
2,80个分区tps8万
bin/kafka-producer-perf-test.sh --topic a-topic --num-records 100000 --record-size 2000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 100000 records sent, 82508.250825 records/sec (157.37 MB/sec), 117.56 ms avg latency, 410.00 ms max latency, 119 ms 50th, 180 ms 95th, 207 ms 99th, 229 ms 99.9th.六,100万tps,消息体2k
1,40个分区tps17万
bin/kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 2000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 848922 records sent, 169750.4 records/sec (323.77 MB/sec), 87.3 ms avg latency, 355.0 ms max latency. 1000000 records sent, 174125.021766 records/sec (332.12 MB/sec), 86.40 ms avg latency, 355.00 ms max latency, 86 ms 50th, 185 ms 95th, 209 ms 99th, 241 ms 99.9th.
2,80个分区tps18万
bin/kafka-producer-perf-test.sh --topic a-topic --num-records 1000000 --record-size 2000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 882894 records sent, 176578.8 records/sec (336.80 MB/sec), 84.2 ms avg latency, 386.0 ms max latency. 1000000 records sent, 181093.806592 records/sec (345.41 MB/sec), 82.92 ms avg latency, 386.00 ms max latency, 73 ms 50th, 182 ms 95th, 219 ms 99th, 277 ms 99.9th.
3,160个分区tps18万
bin/kafka-producer-perf-test.sh --topic b-topic --num-records 1000000 --record-size 2000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 890341 records sent, 178068.2 records/sec (339.64 MB/sec), 82.2 ms avg latency, 539.0 ms max latency. 1000000 records sent, 180929.980098 records/sec (345.10 MB/sec), 82.08 ms avg latency, 539.00 ms max latency, 51 ms 50th, 250 ms 95th, 358 ms 99th, 477 ms 99.9th.
4,240个分区tps17万
bin/kafka-producer-perf-test.sh --topic c-topic --num-records 1000000 --record-size 2000 --throughput -1 --producer-props bootstrap.servers=10.11.25.81:9092 840520 records sent, 168003.2 records/sec (320.44 MB/sec), 86.3 ms avg latency, 749.0 ms max latency. 1000000 records sent, 170183.798502 records/sec (324.60 MB/sec), 87.21 ms avg latency, 753.00 ms max latency, 20 ms 50th, 383 ms 95th, 543 ms 99th, 678 ms 99.9th.
以上是关于pulsar的Kop压力测试的主要内容,如果未能解决你的问题,请参考以下文章
腾讯宣布开源 RoP:Apache Pulsar 支持原生 RocketMQ 协议
腾讯宣布开源 RoP:Apache Pulsar 支持原生 RocketMQ 协议
03_Apache Pulsar的Local与分布式集群构建Pulsar的分布式集群模式Pulsar的分布式集群模式构建启动测试