Python组装jmx并调用JMeter执行压测

Posted 2023-03-02 一个处女座的测试

JMeter可以通过os命令调用Python脚本，Python同样可以通过系统命令调用JMeter执行压测

Python调用JMeter

1. 首先要安装JMeter，官方下载地址
   解压并配置配置环境路径或建立软连，使得在命令输入jmeter便可以执行，如

Copyunzip apache-jmeter-5.3.zip
mv apache-jmeter-5.3 /usr/loca/jmeter
ln -s /usr/local/jmeter/bin/jmeter /usr/bin/jmeter
ln -s /usr/local/jmeter/bin/jmeter-server /usr/bin/jmeter-server

2. 打开JMeter并设计一个测试计划保存为testplan.jmx

3. 使用Python调用JMeter压测并生成报告
   Python中可以使用os.system()或supprocess.Popen()调用系统命令，前者实时显示在屏幕上，后者可以获取到屏幕输出信息。
   使用Python调用JMeter运行及生成报告的命令如下。

Copyimport subprocess

jmx_file = 'testplan.jmx'  # jmx文件路径
result_file = 'result.jtl'  # 
log_file = 'run.log'
report_dir = 'report'

run_cmd = f'jmeter -n -t jmx_file -l result_file -j log_file'  # 无界面运行JMeter压测命令
report_cmd = f'jmeter -g result_file -o report_dir'  # 生成html报告命令

# 不需要获取屏幕输出是，可以使用os.system()
# os.system(run_cmd)
# os.system(report_cmd)

# 需要获取屏幕输出是，可以使用subprocess.Popen()
p1 = subprocess.Popen(run_cmd, shell=True, stdout=subprocess.PIPE)
print(p1.stdout.read().decode('utf-8'))
p2 = subprocess.Popen(report_cmd, shell=True, stdout=subprocess.PIPE)
print(p2.stdout.read().decode('utf-8'))

组装jmx

每一测试计划为一个jmx文件，jmx实际上是xml格式的，包含一些JMeter自定义的格式规范。

常用的组件有：

• : 测试计划

• : 线程组

• : CSV数据文件

• : HTTP请求

• : HTTP请求头管理器

• : Cookies管理器

• : DNS缓存管理器

• : 监听器(包括查看结果树、聚合报告等)

• ： 响应断言

• <io.github.ningyu.jmeter.plugin.dubbo.sample.DubboSample></io.github.ningyu.jmeter.plugin.dubbo.sample.DubboSample>: 三方Dubbo请求插件

Dubbo插件jmeter-plugins-dubbo下载链接

jmx中，如果一个组件有子组件，格式为

Copy<ThreadGroup 组件基本属性>
      ...线程组配置
</ThreadGroup><hashTree>
     ...内部子组件
</hashTree>
···
如果不包含子组件，则后面接一个<hashTree/>  单标签直接结束，例如：
```xml
<CSVDataSet>
    ... CSV数据组件配置
</CSVDataSet><hashTree/>

详细的格式可以自己使用JMeter创建一个测试计划，使用文本文件打开jmx文件查看。

使用Python组装jmx文件的方式有两种，一种是固定模板的数据渲染，一种类似JMeter的逐个组件添加，第一种比较简单。

通过分析jmx文件中的变量，我们使用jinja2模板语法，将其中的变量进行参数化，对需要判断和循环的变量设置if和for循环。

Jinja2中文文档

假设我们的测试计划结构为：

Copy测试计划
    DNS缓存管理器
    Cookies管理器
    CSV文件(多个)
    ...
    聚合报告
    线程组(多个)
        CSV文件(多个)
        HTTP请求(或Dubbo请求)
            HTTP请求头管理器
            CSV文件(多个)
            响应断言
            察看结果树

将jmx中的关键数据抽取并组合，我使用的完整数据格式如下：

data.yaml

Copytest_plan_name:测试计划comments:测试计划描述hosts:-name:las.secoo.comaddress:112.126.120.128cookies:clear_each_iteration:'true'csv_files:- 'name':'数据文件1', 'path':'data.csv', 'varnames':'a,b', 'delimiter':','
  - 'name':'数据文件2', 'path':'data.csv', 'varnames':'c,d', 'delimiter':','
thread_groups:-thread_group_name:线程组1comments:线程组1描述enabled:'true'num_threads:50loops:-1ramp_time:0scheduler:'true'duration:30delay:''http_samples:-request_name:HTTP-GETenabled:'true'csv_files:- 'name':'数据文件4', 'path':'data.csv', 'varnames':'a,b', 'delimiter':','
        request:protocol:httpsdomain:httpbin.orgport:''encoding:''path:/getmethod:GETconnect_timeout:''response_timeout:''params: 'a':1, 'b':2
          headers: 'token':'aaaaaa', 'x-text':'bbbbb'
          follow_redirects:'false'auto_redirects:'false'use_keepalive:'false'validate:-test_field:response_data# response_data响应文本 response_code响应代码response_message响应信息response_headers# 请求头request_headers  sample_label URL样本 response_data_as_document文档(文本) 请求数据request_datatest_type:2# 2 包括 1匹配 8 相等  16字符串  否+4 或者+32strings: ['a', 'b']
      -request_name:HTTP-POSTenabled:'true'request:protocol:httpsdomain:httpbin.orgport:''encoding:''path:/postmethod:POSTdata: 'c':3, 'd':4
          follow_redirects:'false'auto_redirects:'false'use_keepalive:'false'connect_timeout:''response_timeout:''-request_name:HTTP-JSONenabled:'true'request:protocol:httpsdomain:httpbin.orgport:''encoding:''path:/postmethod:POSTconnect_timeout:''response_timeout:''raw_data:'"e": 5, "f": 6'follow_redirects:'false'auto_redirects:'false'use_keepalive:'false'-thread_group_name:线程组2comments:线程组2描述enabled:'false'num_threads:50loops:-1ramp_time:0scheduler:'true'duration:30delay:''csv_files:- 'name':'数据文件3', 'path':'data.csv', 'varnames':'a,b','delimiter':'\\t'
    dubbo_samples:-request_name:查询运费接口-dubboenabled:'true'registry:type:zookeepergroup:''address:'zk-mall1.secoolocal.com:5181?backup=zk-mall2.secoolocal.com:5181,zk-mall3.secoolocal.com:5181'dubbo:timeout:100retires:0group:''connections:100load_balance:randomcluster:failfastservice:'com.secoo.business.config.rpc.service.BusinessConfigStoreService'method:queryFreightheaders:Content-Type:'application/json'params:-type:java.util.Listvalue:$freight_wareHouseId_sendAreaId-type:java.lang.Stringvalue:110100-type:java.util.Listvalue:$freight_wareHouseId_sendAreaIdvalidate:-test_field:response_data# response_data响应文本 response_code响应代码response_message响应信息response_headerstest_type:16# 2 包括 1匹配 8 相等  16字符串  否+4 或者+32strings: ['"code": 0']

对应的模板文件tpl.xml代码如下：

Copy<?xml version="1.0" encoding="UTF-8"?><jmeterTestPlanversion="1.2"properties="5.0"jmeter="5.3"><hashTree><TestPlanguiclass="TestPlanGui"testclass="TestPlan"testname="test_plan_name"enabled="true"><stringPropname="TestPlan.comments">comments</stringProp><boolPropname="TestPlan.functional_mode">false</boolProp><boolPropname="TestPlan.tearDown_on_shutdown">true</boolProp><boolPropname="TestPlan.serialize_threadgroups">false</boolProp><elementPropname="TestPlan.user_defined_variables"elementType="Arguments"guiclass="ArgumentsPanel"testclass="Arguments"testname="用户定义的变量"enabled="true"><collectionPropname="Arguments.arguments"/></elementProp><stringPropname="TestPlan.user_define_classpath"></stringProp></TestPlan><hashTree>% if hosts %
      <DNSCacheManagerguiclass="DNSCachePanel"testclass="DNSCacheManager"testname="DNS缓存管理器"enabled="true"><collectionPropname="DNSCacheManager.servers"/><collectionPropname="DNSCacheManager.hosts">% for host in hosts %
          <elementPropname="las.secoo.com"elementType="StaticHost"><stringPropname="StaticHost.Name">host.name</stringProp><stringPropname="StaticHost.Address">host.address</stringProp></elementProp>% endfor %
        </collectionProp><boolPropname="DNSCacheManager.clearEachIteration">false</boolProp><boolPropname="DNSCacheManager.isCustomResolver">true</boolProp></DNSCacheManager><hashTree/>% endif % % if cookies %
      <CookieManagerguiclass="CookiePanel"testclass="CookieManager"testname="HTTP Cookie管理器"enabled="true"><collectionPropname="CookieManager.cookies"/><boolPropname="CookieManager.clearEachIteration">cookies.clear_each_iteration</boolProp></CookieManager><hashTree/>% endif % % if csv_files %% for csv_file in csv_files %
      <CSVDataSetguiclass="TestBeanGUI"testclass="CSVDataSet"testname="csv_file.name"enabled="true"><stringPropname="filename">dat/csv_file.path</stringProp><stringPropname="fileEncoding">UTF-8</stringProp><stringPropname="variableNames">csv_file.varnames</stringProp><boolPropname="ignoreFirstLine">true</boolProp><stringPropname="delimiter">csv_file.delimiter</stringProp><boolPropname="quotedData">false</boolProp><boolPropname="recycle">true</boolProp><boolPropname="stopThread">false</boolProp><stringPropname="shareMode">shareMode.group</stringProp></CSVDataSet><hashTree/>% endfor %% endif %
      <ResultCollectorguiclass="StatVisualizer"testclass="ResultCollector"testname="聚合报告"enabled="true"><boolPropname="ResultCollector.error_logging">false</boolProp><objProp><name>saveConfig</name><valueclass="SampleSaveConfiguration"><time>true</time><latency>true</latency><timestamp>true</timestamp><success>true</success><label>true</label><code>true</code><message>true</message><threadName>true</threadName><dataType>true</dataType><encoding>false</encoding><assertions>true</assertions><subresults>true</subresults><responseData>false</responseData><samplerData>false</samplerData><xml>false</xml><fieldNames>true</fieldNames><responseHeaders>false</responseHeaders><requestHeaders>false</requestHeaders><responseDataOnError>true</responseDataOnError><saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage><assertionsResultsToSave>0</assertionsResultsToSave><bytes>true</bytes><sentBytes>true</sentBytes><threadCounts>true</threadCounts><idleTime>true</idleTime></value></objProp><stringPropname="filename"></stringProp></ResultCollector><hashTree/>% for thread_group in thread_groups %
      <ThreadGroupguiclass="ThreadGroupGui"testclass="ThreadGroup"testname="thread_group.thread_group_name"enabled="thread_group.enabled"><stringPropname="TestPlan.comments">thread_group.comments</stringProp><stringPropname="ThreadGroup.on_sample_error">continue</stringProp><elementPropname="ThreadGroup.main_controller"elementType="LoopController"guiclass="LoopControlPanel"testclass="LoopController"testname="循环控制器"enabled="true"><boolPropname="LoopController.continue_forever">false</boolProp><intPropname="LoopController.loops">thread_group.loops</intProp></elementProp><stringPropname="ThreadGroup.num_threads">thread_group.num_threads</stringProp><stringPropname="ThreadGroup.ramp_time">thread_group.ramp_time</stringProp><boolPropname="ThreadGroup.scheduler">thread_group.scheduler</boolProp><stringPropname="ThreadGroup.duration">thread_group.duration</stringProp><stringPropname="ThreadGroup.delay">thread_group.delay</stringProp><boolPropname="ThreadGroup.same_user_on_next_iteration">false</boolProp></ThreadGroup><hashTree>% if thread_group.csv_files %% for csv_file in thread_group.csv_files %
        <CSVDataSetguiclass="TestBeanGUI"testclass="CSVDataSet"testname="csv_file.name"enabled="true"><stringPropname="filename">dat/csv_file.path</stringProp><stringPropname="fileEncoding">UTF-8</stringProp><stringPropname="variableNames">csv_file.varnames</stringProp><boolPropname="ignoreFirstLine">true</boolProp><stringPropname="delimiter">csv_file.delimiter</stringProp><boolPropname="quotedData">false</boolProp><boolPropname="recycle">true</boolProp><boolPropname="stopThread">false</boolProp><stringPropname="shareMode">shareMode.group</stringProp></CSVDataSet><hashTree/>% endfor %% endif % % if thread_group.http_samples %% for http_sample in thread_group.http_samples %
        <HTTPSamplerProxyguiclass="HttpTestSampleGui"testclass="HTTPSamplerProxy"testname="http_sample.request_name"enabled="http_sample.enabled"><elementPropname="HTTPsampler.Arguments"elementType="Arguments"guiclass="HTTPArgumentsPanel"testclass="Arguments"testname="用户定义的变量"enabled="true"><collectionPropname="Arguments.arguments">% if http_sample.request.params %% for name, value in http_sample.request.params.items() %
              <elementPropname="name"elementType="HTTPArgument"><boolPropname="HTTPArgument.always_encode">false</boolProp><stringPropname="Argument.value">value</stringProp><stringPropname="Argument.metadata">=</stringProp><boolPropname="HTTPArgument.use_equals">true</boolProp><stringPropname="Argument.name">name</stringProp></elementProp>% endfor %% endif % % if http_sample.request.data %% for name, value in http_sample.request.data.items() %
              <elementPropname="name"elementType="HTTPArgument"><boolPropname="HTTPArgument.always_encode">false</boolProp><stringPropname="Argument.value">value</stringProp><stringPropname="Argument.metadata">=</stringProp><boolPropname="HTTPArgument.use_equals">true</boolProp><stringPropname="Argument.name">name</stringProp></elementProp>% endfor %% endif % % if http_sample.request.raw_data %
              <elementPropname=""elementType="HTTPArgument"><boolPropname="HTTPArgument.always_encode">false</boolProp><stringPropname="Argument.value">http_sample.request.raw_data</stringProp><stringPropname="Argument.metadata">=</stringProp></elementProp>% endif %
            </collectionProp></elementProp><stringPropname="HTTPSampler.domain">http_sample.request.domain</stringProp><stringPropname="HTTPSampler.port">http_sample.request.port</stringProp><stringPropname="HTTPSampler.protocol">http_sample.request.protocol</stringProp><stringPropname="HTTPSampler.contentEncoding">http_sample.request.encoding</stringProp><stringPropname="HTTPSampler.path">http_sample.request.path</stringProp><stringPropname="HTTPSampler.method">http_sample.request.method</stringProp><boolPropname="HTTPSampler.follow_redirects">http_sample.request.follow_redirects</boolProp><boolPropname="HTTPSampler.auto_redirects">http_sample.request.auto_redirects</boolProp><boolPropname="HTTPSampler.use_keepalive">http_sample.request.use_keepalive</boolProp><boolPropname="HTTPSampler.DO_MULTIPART_POST">false</boolProp><stringPropname="HTTPSampler.embedded_url_re"></stringProp><stringPropname="HTTPSampler.connect_timeout">http_sample.request.connect_timeout</stringProp><stringPropname="HTTPSampler.response_timeout">http_sample.request.response_timeout</stringProp></HTTPSamplerProxy><hashTree>% if http_sample.request.headers %
          <HeaderManagerguiclass="HeaderPanel"testclass="HeaderManager"testname="HTTP信息头管理器"enabled="true"><collectionPropname="HeaderManager.headers">% for name, value in http_sample.request.headers.items() %
              <elementPropname=""elementType="Header"><stringPropname="Header.name">name</stringProp><stringPropname="Header.value">value</stringProp></elementProp>% endfor %
            </collectionProp></HeaderManager><hashTree/>% endif % % if http_sample.csv_files %% for csv_file in http_sample.csv_files %
          <CSVDataSetguiclass="TestBeanGUI"testclass="CSVDataSet"testname="CSV 数据文件设置"enabled="true"><stringPropname="delimiter">csv_file.delimiter</stringProp><stringPropname="fileEncoding">UTF_8</stringProp><stringPropname="filename">dat/csv_file.path</stringProp><boolPropname="ignoreFirstLine">true</boolProp><boolPropname="quotedData">false</boolProp><boolPropname="recycle">true</boolProp><stringPropname="shareMode">shareMode.group</stringProp><boolPropname="stopThread">false</boolProp><stringPropname="variableNames">csv_file.varnames</stringProp></CSVDataSet><hashTree/>% endfor %% endif % % if http_sample.validate %% for assertion in http_sample.validate %
          <ResponseAssertionguiclass="AssertionGui"testclass="ResponseAssertion"testname="响应断言"enabled="true"><collectionPropname="Asserion.test_strings">% if assertion.strings %% for string in assertion.strings %
              <stringPropname="97">string</stringProp>% endfor %% endif %
            </collectionProp><stringPropname="Assertion.custom_message"></stringProp><stringPropname="Assertion.test_field">Assertion.assertion.test_field</stringProp><boolPropname="Assertion.assume_success">false</boolProp><intPropname="Assertion.test_type">assertion.test_type</intProp></ResponseAssertion><hashTree/>% endfor %% endif %
          <ResultCollectorguiclass="ViewResultsFullVisualizer"testclass="ResultCollector"testname="察看结果树"enabled="true"><boolPropname="ResultCollector.error_logging">false</boolProp><objProp><name>saveConfig</name><valueclass="SampleSaveConfiguration"><time>true</time><latency>true</latency><timestamp>true</timestamp><success>true</success><label>true</label><code>true</code><message>true</message><threadName>true</threadName><dataType>true</dataType><encoding>false</encoding><assertions>true</assertions><subresults>true</subresults><responseData>false</responseData><samplerData>false</samplerData><xml>false</xml><fieldNames>true</fieldNames><responseHeaders>false</responseHeaders><requestHeaders>false</requestHeaders><responseDataOnError>false</responseDataOnError><saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage><assertionsResultsToSave>0</assertionsResultsToSave><bytes>true</bytes><sentBytes>true</sentBytes><url>true</url><threadCounts>true</threadCounts><idleTime>true</idleTime><connectTime>true</connectTime></value></objProp><stringPropname="filename"></stringProp></ResultCollector><hashTree/></hashTree>% endfor %% endif % % if thread_group.dubbo_samples % % for dubbo_sample in thread_group.dubbo_samples %
        <io.github.ningyu.jmeter.plugin.dubbo.sample.DubboSampleguiclass="io.github.ningyu.jmeter.plugin.dubbo.gui.DubboSampleGui"testclass="io.github.ningyu.jmeter.plugin.dubbo.sample.DubboSample"testname="dubbo_sample.request_name"enabled="dubbo_sample.enabled"><stringPropname="FIELD_DUBBO_REGISTRY_PROTOCOL">dubbo_sample.registry.type</stringProp><stringPropname="FIELD_DUBBO_REGISTRY_GROUP">dubbo_sample.registry.group</stringProp><stringPropname="FIELD_DUBBO_RPC_PROTOCOL">dubbo://</stringProp><stringPropname="FIELD_DUBBO_ADDRESS">dubbo_sample.registry.address</stringProp><stringPropname="FIELD_DUBBO_TIMEOUT">dubbo_sample.dubbo.timeout</stringProp><stringPropname="FIELD_DUBBO_VERSION"></stringProp><stringPropname="FIELD_DUBBO_RETRIES">dubbo_sample.dubbo.retries</stringProp><stringPropname="FIELD_DUBBO_GROUP">dubbo_sample.dubbo.group</stringProp><stringPropname="FIELD_DUBBO_CONNECTIONS">dubbo_sample.dubbo.connections</stringProp><stringPropname="FIELD_DUBBO_LOADBALANCE">dubbo_sample.dubbo.load_balance</stringProp><stringPropname="FIELD_DUBBO_ASYNC">sync</stringProp><stringPropname="FIELD_DUBBO_CLUSTER">dubbo_sample.dubbo.cluster</stringProp><stringPropname="FIELD_DUBBO_INTERFACE">dubbo_sample.dubbo.service</stringProp><stringPropname="FIELD_DUBBO_METHOD">dubbo_sample.dubbo.method</stringProp><intPropname="FIELD_DUBBO_METHOD_ARGS_SIZE">1</intProp>% for param in dubbo_sample.dubbo.params %
            <stringPropname="FIELD_DUBBO_METHOD_ARGS_PARAM_TYPE1">param.type</stringProp><stringPropname="FIELD_DUBBO_METHOD_ARGS_PARAM_VALUE1">param.value</stringProp>% endfor %
          <intPropname="FIELD_DUBBO_ATTACHMENT_ARGS_SIZE">0</intProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_PROTOCOL"></stringProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_GROUP"></stringProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_NAMESPACE"></stringProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_USER_NAME"></stringProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_PASSWORD"></stringProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_ADDRESS"></stringProp><stringPropname="FIELD_DUBBO_CONFIG_CENTER_TIMEOUT"></stringProp><stringPropname="FIELD_DUBBO_REGISTRY_USER_NAME"></stringProp><stringPropname="FIELD_DUBBO_REGISTRY_PASSWORD"></stringProp><stringPropname="FIELD_DUBBO_REGISTRY_TIMEOUT"></stringProp></io.github.ningyu.jmeter.plugin.dubbo.sample.DubboSample><hashTree>% if dubbo_sample.dubbo.headers %
          <HeaderManagerguiclass="HeaderPanel"testclass="HeaderManager"testname="HTTP信息头管理器"enabled="true"><collectionPropname="HeaderManager.headers">% for name, value in dubbo_sample.dubbo.headers.items() %
              <elementPropname=""elementType="Header"><stringPropname="Header.name">name</stringProp><stringPropname="Header.value">value</stringProp></elementProp>% endfor %
            </collectionProp></HeaderManager><hashTree/>% endif % % if dubbo_sample.validate % % for assertion in dubbo_sample.validate %
          <ResponseAssertionguiclass="AssertionGui"testclass="ResponseAssertion"testname="响应断言"enabled="true"><collectionPropname="Asserion.test_strings">% if assertion.strings %% for string in assertion.strings %
              <stringPropname="97">string</stringProp>% endfor %% endif %
            </collectionProp><stringPropname="Assertion.custom_message"></stringProp><stringPropname="Assertion.test_field">Assertion.assertion.test_field</stringProp><boolPropname="Assertion.assume_success">false</boolProp><intPropname="Assertion.test_type">assertion.test_type</intProp></ResponseAssertion>% endfor % % endif %
          <hashTree/>% endfor %% endif % % endfor %
      </hashTree></hashTree></hashTree></jmeterTestPlan>

组装出类似data.yaml格式的数据，并使用jinja2渲染模板即可得到完整的jmx文件

pip install pyyaml jinja2

Copyimport yaml
import jinja2

# 组装或读取数据withopen('data.yaml', encoding='utf-8') as f:
    data = yaml.safe_load(f)

 # 读取模板withopen('tpl.xml', encoding='utf-8') as f:
    tpl = f.read()

# 渲染模板生成jmx
jmx = jinja2.Template(tpl).render(data)
withopen(jmx_file, 'w', encoding='utf-8') as f:
    f.write(jmx)

后计

在实际项目中，还涉及数据文件的拷贝，节点环境的部署，脚本的分发，报告的下载等等，可以使用paramiko或者fabric或ansible完成，压测节点数据分发的服务管理。

如有不懂还要咨询下方小卡片，博主也希望和志同道合的测试人员一起学习进步

在适当的年龄，选择适当的岗位，尽量去发挥好自己的优势。

我的自动化测试开发之路，一路走来都离不每个阶段的计划，因为自己喜欢规划和总结，

测试开发视频教程、学习笔记领取传送门！！！