同步MySQL数据到ES神器mysqlmom介绍
Posted 运维之美
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了同步MySQL数据到ES神器mysqlmom介绍相关的知识，希望对你有一定的参考价值。
一、mysqlmom简单介绍:
一个同步Mysql数据到Elasticsearch的工具,特色是支持分析 binlog 做实时增量同步，以及支持编写自定义逻辑处理数据后再同步到ES
纯 Python 编写，运行 mysqlsmom 的唯三要求:
python2.7
redis
Mysql 配置 binlog-format=row
它是除了阿里canal+MQ或者kakfa 读取MySQL Binlog文件然后再写入到ES.或者logstash直接从mysql同步数据到ElasticSearch之后的又一个同步MySQL数据到ES的方式。
二、演示环境:
利用MySQLmom程序同步MySQL数据到ES整个部署环境介绍:
3台阿里ECS服务器centos7.6 x86_64位最小化安装
3台服务器默认的python版本号:Python 2.7.5
tidb06 172.16.0.247 部署elasticsearch-5.0.0和elasticsearch-6.2.4(采用的是二进制安装包)
tidb05 172.16.0.246 部署 mysqlsmom工具和redis-3.2.8(redis服务存放MySQL服务的pos位置点和binlog文件名称，是后面做增量同步MySQL数据到ES服务用到)
tidb04 172.16.0.197 部署MySQL5.7.22(MySQL服务没开启gtid,binlog-format的格式为row,)
阿里云安全组要把对应的端口放开，允许内网访问es和mysql安装部署此处省略
单节点ES安装可以参考下面的文档:
https://blog.51cto.com/wujianwei/2674658
mysqlsmom官方安装文档:
https://github.com/m358807551/mysqlsmom
官方提供的使用文档:
https://mysqlsmom.readthedocs.io/en/latest/
三、安装部署：
cd /data/soft/
git clone https://github.com/m358807551/mysqlsmom.git
[root@tidb05 mysqlsmom]# ls
docs mysqlsmom README.md README_OLD.md setup.py
pip install mysqlsmom
[root@tidb05 mysqlsmom]# pip install mysqlsmom
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won\'t be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/
Collecting mysqlsmom
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/e7/ed/3327e96cf31747a98248c833496d9d56349f444c8d5f7168b5bb9aff0925/mysqlsmom-0.1.7-py2-none-any.whl
Collecting idna==2.7 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
|████████████████████████████████| 61kB 4.9MB/s
Collecting asn1crypto==0.24.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
|████████████████████████████████| 102kB 8.0MB/s
Collecting six==1.11.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting Pygments==2.2.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl (841kB)
|████████████████████████████████| 849kB 8.7MB/s
Collecting wcwidth==0.1.7 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Collecting blinker==1.4 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/1b/51/e2a9f3b757eb802f61dc1f2b09c8c99f6eb01cf06416c0671253536517b6/blinker-1.4.tar.gz (111kB)
|████████████████████████████████| 112kB 65.1MB/s
Collecting decorator==4.3.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Collecting pytz==2018.5 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl (510kB)
|████████████████████████████████| 512kB 8.3MB/s
Collecting ipython-genutils==0.2.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting pathlib2==2.3.2 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/66/a7/9f8d84f31728d78beade9b1271ccbfb290c41c1e4dc13dbd4997ad594dcd/pathlib2-2.3.2-py2.py3-none-any.whl
Collecting tzlocal==1.5.1 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz
Collecting scandir==1.7 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/13/bb/e541b74230bbf7a20a3949a2ee6631be299378a784f5445aa5d0047c192b/scandir-1.7.tar.gz
Collecting pycparser==2.18 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/8c/2d/aad7f16146f4197a11f8e91fb81df177adcc2073d36a17b1491fd09df6ed/pycparser-2.18.tar.gz (245kB)
|████████████████████████████████| 256kB 63.1MB/s
Collecting yolk==0.4.3 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/2b/c0/73510e50668b70f3fad25328c58d7a29a300a2e452058dfdcb7c7538af7b/yolk-0.4.3.tar.gz (86kB)
|████████████████████████████████| 92kB 2.0MB/s
Collecting enum34==1.1.6 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Requirement already satisfied: chardet==3.0.4 in /usr/lib/python2.7/site-packages (from mysqlsmom) (3.0.4)
Collecting funcsigs==1.0.2 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting elasticsearch==2.4.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/3f/be/980d79da0fbb14acc29e3c9b724287cbc24be2cb1ede47f2c0eecba60809/elasticsearch-2.4.0-py2.py3-none-any.whl (54kB)
|████████████████████████████████| 61kB 1.8MB/s
Collecting prompt-toolkit==1.0.15 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/d1/b0/1a6c262da35c779dd79550137aa7c298a424987240a28792ec5ccf48f848/prompt_toolkit-1.0.15-py2-none-any.whl (247kB)
|████████████████████████████████| 256kB 63.2MB/s
Collecting backports.shutil-get-terminal-size==1.0.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/7d/cd/1750d6c35fe86d35f8562091737907f234b78fdffab42b29c72b1dd861f4/backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl
Collecting ipython==5.0.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/eb/b6/3ed54fe22c284108d9676029b7cdca0ce94bcb8dc24487af0c88d57c53a1/ipython-5.0.0-py2.py3-none-any.whl (743kB)
|████████████████████████████████| 747kB 29.8MB/s
Collecting peewee==3.5.2 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/33/07/d4c742d88998fb448e18be41dc25e748376b7f69359d37fedaee09e66617/peewee-3.5.2.tar.gz (1.9MB)
|████████████████████████████████| 1.9MB 86kB/s
Collecting simplegeneric==0.8.1 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip
Collecting pickleshare==0.7.4 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/9f/17/daa142fc9be6b76f26f24eeeb9a138940671490b91cb5587393f297c8317/pickleshare-0.7.4-py2.py3-none-any.whl
Collecting cryptography==2.2.2 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/dd/c2/3a5bfefb25690725824ade71e6b65449f0a9f4b29702cce10560f786ebf6/cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
|████████████████████████████████| 2.2MB 2.4MB/s
Collecting futures==3.2.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting urllib3==1.23 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
|████████████████████████████████| 143kB 66.9MB/s
Collecting PyMySQL==0.9.2 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/a7/7d/682c4a7da195a678047c8f1c51bb7682aaedee1dca7547883c3993ca9282/PyMySQL-0.9.2-py2.py3-none-any.whl (47kB)
|████████████████████████████████| 51kB 6.5MB/s
Collecting ipaddress==1.0.22 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting mysql-replication==0.18 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/ff/c3/51cb2bdd50468d0bf37c2ab674bed3fd5939b3ab680c47470ea3a900b721/mysql-replication-0.18.tar.gz
Collecting requests==2.19.1 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
|████████████████████████████████| 92kB 40.7MB/s
Collecting cffi==1.11.5 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)
|████████████████████████████████| 409kB 576kB/s
Collecting traitlets==4.3.2 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl (74kB)
|████████████████████████████████| 81kB 62.8MB/s
Collecting click==6.7 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)
|████████████████████████████████| 71kB 58.9MB/s
Collecting ptyprocess==0.6.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
Collecting certifi==2018.4.16 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
|████████████████████████████████| 153kB 64.3MB/s
Collecting pexpect==4.6.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl (57kB)
|████████████████████████████████| 61kB 55.9MB/s
Collecting redis==2.10.6 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/3b/f6/7a76333cf0b9251ecf49efff635015171843d9b977e4ffcf59f9c4428052/redis-2.10.6-py2.py3-none-any.whl (64kB)
|████████████████████████████████| 71kB 52.3MB/s
Collecting APScheduler==3.5.1 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/60/08/fbd377c75a4531d8ef55a65fe3fc316a1cada9b552e290f10705461807fb/APScheduler-3.5.1-py2.py3-none-any.whl (60kB)
|████████████████████████████████| 61kB 38.4MB/s
Collecting appnope==0.1.0 (from mysqlsmom)
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/87/a9/7985e6a53402f294c8f0e8eff3151a83f1fb901fa92909bb3ff29b4d22af/appnope-0.1.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from yolk==0.4.3->mysqlsmom) (36.4.0)
Building wheels for collected packages: blinker, tzlocal, scandir, pycparser, yolk, peewee, simplegeneric, mysql-replication
Building wheel for blinker (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/4d/0f/13/a39e525325104948e5ca166374d581961da53d7e8e6d7568ab
Building wheel for tzlocal (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/8b/02/9e/c2975e524c35630d636e8ee36c50f1802c1e2d5f55808f4eaa
Building wheel for scandir (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/43/5a/3a/e0601dc9620b1d57d0abf16ad1621245a5223ee90828eadb52
Building wheel for pycparser (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/73/a0/a2/1d71209495d2e50df76397c4402abdec4f113ac801c112d108
Building wheel for yolk (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/fc/3f/8f/6a0fe676b8c82fbe6ecf4081469fbc72c9ea11a283bcc01546
Building wheel for peewee (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/bc/0b/b7/44e1baafcd11ab6251d5e26f2948822960adf2dd0a85156be3
Building wheel for simplegeneric (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/5e/9c/8b/73a7fb1e4b0ffeb65c9c4341211d09854c86691e746d1af66b
Building wheel for mysql-replication (setup.py) ... done
Stored in directory: /root/.cache/pip/wheels/66/c7/4c/98d1b7d1ad1dc0ea6081ae3324446306e20b2bb65e238ccc40
Successfully built blinker tzlocal scandir pycparser yolk peewee simplegeneric mysql-replication
Installing collected packages: idna, asn1crypto, six, Pygments, wcwidth, blinker, decorator, pytz, ipython-genutils, scandir, pathlib2, tzlocal, pycparser, yolk, enum34, funcsigs, urllib3, elasticsearch, prompt-toolkit, backports.shutil-get-terminal-size, ptyprocess, pexpect, traitlets, simplegeneric, pickleshare, ipython, peewee, cffi, ipaddress, cryptography, futures, PyMySQL, mysql-replication, certifi, requests, click, redis, APScheduler, appnope, mysqlsmom
Found existing installation: idna 2.8
Uninstalling idna-2.8:
Successfully uninstalled idna-2.8
Found existing installation: decorator 3.4.0
Uninstalling decorator-3.4.0:
Successfully uninstalled decorator-3.4.0
Found existing installation: urllib3 1.25.3
Uninstalling urllib3-1.25.3:
Successfully uninstalled urllib3-1.25.3
Found existing installation: certifi 2019.6.16
Uninstalling certifi-2019.6.16:
Successfully uninstalled certifi-2019.6.16
Found existing installation: requests 2.22.0
Uninstalling requests-2.22.0:
Successfully uninstalled requests-2.22.0
Successfully installed APScheduler-3.5.1 PyMySQL-0.9.2 Pygments-2.2.0 appnope-0.1.0 asn1crypto-0.24.0 backports.shutil-get-terminal-size-1.0.0 blinker-1.4 certifi-2018.4.16 cffi-1.11.5 click-6.7 cryptography-2.2.2 decorator-4.3.0 elasticsearch-2.4.0 enum34-1.1.6 funcsigs-1.0.2 futures-3.2.0 idna-2.7 ipaddress-1.0.22 ipython-5.0.0 ipython-genutils-0.2.0 mysql-replication-0.18 mysqlsmom-0.1.7 pathlib2-2.3.2 peewee-3.5.2 pexpect-4.6.0 pickleshare-0.7.4 prompt-toolkit-1.0.15 ptyprocess-0.6.0 pycparser-2.18 pytz-2018.5 redis-2.10.6 requests-2.19.1 scandir-1.7 simplegeneric-0.8.1 six-1.11.0 traitlets-4.3.2 tzlocal-1.5.1 urllib3-1.23 wcwidth-0.1.7 yolk-0.4.3
WARNING: You are using pip version 19.1.1, however version 20.3.4 is available.
You should consider upgrading via the \'pip install --upgrade pip\' command.
[root@tidb05 mysqlsmom]# 说明：然后指定 elasticsearch 版本（默认支持2.4），支持其它版本请运行（将5.4换成需要的elasticsearch版本）
[root@tidb05 mysqlsmom]# pip install --upgrade elasticsearch==5.0
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won\'t be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/
Collecting elasticsearch==5.0
Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/f3/4a/446018515cbaecaa63465e3af003a41ce6d3c4deb6cbe751b73e1aff1dad/elasticsearch-5.0.0-py2.py3-none-any.whl (57kB)
|████████████████████████████████| 61kB 1.7MB/s
Requirement already satisfied, skipping upgrade: urllib3<2.0,>=1.8 in /usr/lib/python2.7/site-packages (from elasticsearch==5.0) (1.23)
ERROR: mysqlsmom 0.1.7 has requirement elasticsearch==2.4.0, but you\'ll have elasticsearch 5.0.0 which is incompatible.
Installing collected packages: elasticsearch
Found existing installation: elasticsearch 2.4.0
Uninstalling elasticsearch-2.4.0:
Successfully uninstalled elasticsearch-2.4.0
Successfully installed elasticsearch-5.0.0
WARNING: You are using pip version 19.1.1, however version 20.3.4 is available.
You should consider upgrading via the \'pip install --upgrade pip\' command.
[root@tidb05 mysqlsmom]# echo $?
0安装部署到此完成。是不是灰常的简单呢，后面会介绍全量同步MySQL表数据到ES服务
以上是关于同步MySQL数据到ES神器mysqlmom介绍的主要内容，如果未能解决你的问题，请参考以下文章