centos7 安装esrally 踩坑过程

Posted 2022-10-10 bohu83

背景：

为了压测下新ES集群性能，使用es官网推荐的Esrally。因为网络差异， 按照官网的步骤来，一步一个坑那基本上就是下载很慢，半天过去爬不出来。

所以那个简洁的命令有很多依赖前置条件。不然就是装不上。

pip安装方式：

需要依赖python3,很多大佬用了Python3.8的版本。我偷个懒，也是给自己挖了坑。

依赖条件：

<1>. 安装Python3.5及以上版本，系统默认可能是2.x版本,

<2>. 安装git1.9及以上版本

<3>. JDK 1.8+ (下载解压、配置环境变量即可)

对于很多后端同学，jdk解压配置不用说了，网上很多。

我基本上安装过程参照了干货：Elasticsearch 压测工具之 esrally的安装与使用（上） - 腾讯云开发者社区-腾讯云

区别是我的python 版本是3.7.14。到了pip3 install esrally就不行了

我是root账户，提示：

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

 所以，采用了虚拟环境来替代

　yum install python-virtualenv

python3 -m venv xp-env

这种最后的xp-env 自己起名字就好

加载虚拟环境

source xp-env/bin/activate

我为啥要单独测试bz2.

进行安装：pip install esrally

这个过程有点长。

不能有warning，执行esrally

之前安装完就报了 

Traceback (most recent call last):
  File "/usr/local/src/venv/bin/esrally", line 5, in <module>
    from esrally.rally import main
  File "/usr/local/src/venv/lib/python3.7/site-packages/esrally/rally.py", line 27, in <module>
    from esrally import PROGRAM_NAME, BANNER, SKULL, check_python_version, doc_link, telemetry
  File "/usr/local/src/venv/lib/python3.7/site-packages/esrally/telemetry.py", line 25, in <module>
    from esrally import metrics, time, exceptions
  File "/usr/local/src/venv/lib/python3.7/site-packages/esrally/metrics.py", line 33, in <module>
    from esrally import time, exceptions, config, version, paths
  File "/usr/local/src/venv/lib/python3.7/site-packages/esrally/config.py", line 27, in <module>
    from esrally.utils import io, git, console, convert
  File "/usr/local/src/venv/lib/python3.7/site-packages/esrally/utils/io.py", line 18, in <module>
    import bz2
  File "/root/training/Python-3.7.14/lib/python3.7/bz2.py", line 19, in <module>
    from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'

尝试使用pip3安装不行

1. pip3 install bz2

改为下载源码进行安装

wget http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar -zxf  bzip2-1.0.6.tar.gz 
 
cd bzip2-1.0.6  
 
make -f  Makefile-libbz2_so 
 
make && make install

还得再次安装Python，这个很耗时，所以很麻烦。

得验证到上面import bz2 不报错，下面这种不行

Python 3.7.14 (default, Oct  9 2022, 10:52:02) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bz2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/training/Python-3.7.14/lib/python3.7/bz2.py", line 19, in <module>
    from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'

解决了bz2的问题后，有报了别的错

Traceback (most recent call last):
  File "/usr/local/xp-env/bin/esrally", line 5, in <module>
    from esrally.rally import main
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/rally.py", line 28, in <module>
    from esrally import version, actor, config, paths, racecontrol, reporter, metrics, track, chart_generator, exceptions, \\
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/racecontrol.py", line 26, in <module>
    from esrally import actor, config, doc_link, driver, exceptions, mechanic, metrics, reporter, track, PROGRAM_NAME
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/driver/__init__.py", line 19, in <module>
    from .driver import DriverActor, PrepareBenchmark, PreparationComplete, StartBenchmark, BenchmarkComplete, TaskFinished
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/driver/driver.py", line 27, in <module>
    from esrally import actor, config, exceptions, metrics, track, client, paths, PROGRAM_NAME, telemetry
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/track/__init__.py", line 18, in <module>
    from .loader import (
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/track/loader.py", line 26, in <module>
    import jinja2
  File "/usr/local/xp-env/lib/python3.7/site-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/usr/local/xp-env/lib/python3.7/site-packages/jinja2/environment.py", line 15, in <module>
    from jinja2 import nodes
  File "/usr/local/xp-env/lib/python3.7/site-packages/jinja2/nodes.py", line 19, in <module>
    from jinja2.utils import Markup
  File "/usr/local/xp-env/lib/python3.7/site-packages/jinja2/utils.py", line 642, in <module>
    from markupsafe import Markup, escape, soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/xp-env/lib/python3.7/site-packages/markupsafe/__init__.py)

这个，看了大佬的办法不一样，我试了pip install --upgrade aws-sam-cli 

还有一些其他错误，可能python版本不同不太一样

Traceback (most recent call last):
  File "/usr/local/xp-env/bin/esrally", line 5, in <module>
    from esrally.rally import main
  File "/usr/local/xp-env/lib/python3.7/site-packages/esrally/__init__.py", line 24, in <module>
    __version__ = pkg_resources.require("esrally")[0].version
  File "/usr/local/xp-env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 909, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/xp-env/lib/python3.7/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (boto3 1.24.89 (/usr/local/xp-env/lib/python3.7/site-packages), Requirement.parse('boto3==1.10.32'), 'esrally')

这种，就是重新pip装,指定的版本就行。

最后，终于跑起来了