Mysql + canal + zookeeper环境搭建
Posted 何国秀_xue
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Mysql + canal + zookeeper环境搭建相关的知识，希望对你有一定的参考价值。
mysql + canal + zookeeper环境搭建
一、mysql集群搭建
1. mysql基本环境
操作系统： Linux version 2.6.32-431.el6.x86_64
数据库：MySQL Community Server 5.7.20
主节点IP：10.60.81.157
主节点IP：10.60.81.158
从节点IP：10.60.81.159
2. 安装mysql
2.1．官网下载MySQL
mysql-5.7.20-1.el6.x86_64.rpm-bundle.tar
2.2. 三个节点都安装：
2.2.1．三个节点查看是否安装mysql
rpm -qa | grep mysql
rpm -e --nodeps mysql-libs-5.1.66-2.el6_3.x86_64 （有则删除）
2.2.2. 三个节点都安装mysql
tar –xvf mysql-5.7.20-1.el6.x86_64.rpm-bundle.tar
rpm -ivh mysql-community-common-5.7.20-1.el6.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.20-1.el6.x86_64.rpm
rpm -ivh mysql-community-client-5.7.20-1.el6.x86_64.rpm
rpm -ivh mysql-community-server-5.7.20-1.el6.x86_64.rpm
2.3．设置密码
2.3.1. 编辑配置文件my.cnf
[mysqld]
server-id=157 #####这里不同
log-bin=mysql-bin ####开启二进制日志
log-slave-updates
binlog-format=ROW
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 1M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
#skip-grant-tables=1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
2.3.2. 启动mysql
service mysqld start
2.3.3. 获取MySQL的临时密码
如果安装的是RPM包，则默认是/var/log/mysqld.log
2.3.4. 登陆并修改密码
使用默认的密码登陆:
mysql -uroot -p
alter user \'root\'@\'localhost\' identified by \'Mysql@123456\';
flush privileges;
3. 创建同步数据用户及授权用户
3.1．授权其他机器登陆
GRANT ALL PRIVILEGES ON *.* TO \'root\'@\'%\' IDENTIFIED BY \'Mysql@123456\' WITH GRANT OPTION;
flush privileges;
3.2. 创建同步用户
grant replication slave on *.* to \'mytest\'@\'10.60.81.%\' identified by \'mytest@123\';
4. mysql主主配置
4.1. 配置两台主机及启动
查看两台主机（10.60.81.157、10.60.81.158）配置文件（my.cnf）除了server-id不一样，其他几乎一模一样。配置文件写好后，我们把两台服务器上的mysql服务器启动起来
4.2. 登录10.60.81.157的mysql中查看master状态
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000002 | 1290 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
4.3. 登录10.60.81.158的msyql中，把10.60.81.157配置成自己的主
Change master to master_host=\'10.60.81.157\',master_port=3306,master_user=\'mytest\',master_password=\'mytest@123\',master_log_file=\'mysql-bin.000002\',master_log_pos=1290;
同上，查看10.60.81.158，然后登录10.60.81.157,把10.60.81.158，配置成自己的主，然后分别在两台机器的mysql中，启动slave
4.4. 启动10.60.81.158状态
mysql> show slave status \\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.60.81.158
Master_User: mytest
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000003
Read_Master_Log_Pos: 1290
Relay_Log_File: sz280317-relay-bin.000009
Relay_Log_Pos: 367
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 1290
Relay_Log_Space: 743
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 158
Master_UUID: b805346c-54e0-11e8-baa8-005056a42d94
Master_Info_File: /var/lib/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
可以看到 Slave_IO_Running: Yes
Slave_SQL_Running: Yes
然后在看10.60.81.158的状态：
mysql> show slave status \\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.60.81.157
Master_User: mytest
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000005
Read_Master_Log_Pos: 717
Relay_Log_File: sz280318-relay-bin.000010
Relay_Log_Pos: 6027
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 717
Relay_Log_Space: 7402
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 157
Master_UUID: 1412180f-54c9-11e8-9e3e-005056a436f6
Master_Info_File: /var/lib/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
5．mysql主从配置
查看两台机器（10.60.81.157、10.60.81.159），把10.60.81.157作主，10.60.81.159作从，与主主配置唯一不同是，不要在主机配置从机为自己的主即可。
二、zookeeper集群搭建
1. zookeeper基本环境
操作系统： Linux version 2.6.32-431.el6.x86_64
Zookeeper: zookeeper-3.4.12.tar.gz
三台机器：10.60.81.160、10.60.81.161、10.60.81.162
2. zookeeper 安装与配置
2.1．官网下载zookeeper
zookeeper-3.4.12.tar.gz
2.2. 上传到远程节点
scp zookeeper.tar.gz root@10.60.81.161
scp zookeeper.tar.gz root@10.60.81.161
scp zookeeper.tar.gz root@10.60.81.162
2.3. 登录远程节点（其他两个节点的擦作大同小异）
ssh root@192.168.213.44
2.4. 解压包，配置，在cnof 目录新建zoo.cfg
sz280320:cloudera-scm@/hgx/zookeeper-3.4.12/conf>cat zoo.cfg
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/hgx/servers/zookeeper/data
dataLogDir=/hgx/servers/zookeeper/logs
clientPort=4180
server.60=10.60.81.160:2888:3888
server.61=10.60.81.161:2888:3888
server.62=10.60.81.162:2888:3888
在目录上述dataDir的目录中创建myid, 将上述的三个数，server.X(我这里的X是60，61，62),中的X值写入myid文件，三台节点的myid这是不一样的。
3、zookeeper 启动与运行
3.1. 三个节点都要启动
zookeeper/bin/zkServer.sh start
3.2、zookeeper状态检查
sz280320:cloudera-scm@/hgx/zookeeper-3.4.12/bin>./zkCli.sh -server 10.60.81.160:4180
Connecting to 10.60.81.160:4180
2018-05-21 11:26:37,346 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-05-21 11:26:37,355 [myid:] - INFO [main:Environment@100] - Client environment:host.name=sz280320
2018-05-21 11:26:37,355 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_151
2018-05-21 11:26:37,361 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-05-21 11:26:37,361 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_151/jre
2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/hgx/zookeeper-3.4.12/bin/../build/classes:/hgx/zookeeper-3.4.12/bin/../build/lib/*.jar:/hgx/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/hgx/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/hgx/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/hgx/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/hgx/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/hgx/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/hgx/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/hgx/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/hgx/zookeeper-3.4.12/bin/../conf:
2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2018-05-21 11:26:37,363 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2018-05-21 11:26:37,363 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2018-05-21 11:26:37,363 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64
2018-05-21 11:26:37,364 [myid:] - INFO [main:Environment@100] - Client environment:user.name=cloudera-scm
2018-05-21 11:26:37,364 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/cloudera-scm
2018-05-21 11:26:37,364 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/hgx/zookeeper-3.4.12/bin
2018-05-21 11:26:37,368 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=10.60.81.160:4180 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29
Welcome to ZooKeeper!
2018-05-21 11:26:37,430 [myid:] - INFO [main-SendThread(10.60.81.160:4180):ClientCnxn$SendThread@1028] - Opening socket connection to server 10.60.81.160/10.60.81.160:4180. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-05-21 11:26:37,622 [myid:] - INFO [main-SendThread(10.60.81.160:4180):ClientCnxn$SendThread@878] - Socket connection established to 10.60.81.160/10.60.81.160:4180, initiating session
2018-05-21 11:26:37,653 [myid:] - INFO [main-SendThread(10.60.81.160:4180):ClientCnxn$SendThread@1302] - Session establishment complete on server 10.60.81.160/10.60.81.160:4180, sessionid = 0x3c039e8bf59a0003, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
三、搭建HA模式的canal
1. canal基本环境
操作系统： Linux version 2.6.32-431.el6.x86_64
canal server： canal.deployer-1.0.25.tar.gz
二台机器：10.60.81.159、10.60.81.158
2．Canal安装
2.1. 官网下载Canal
canal.deployer-1.0.25.tar.gz
2.2. 新建canal用户，并赋予权限
CREATE USER canal IDENTIFIED BY \'canal\';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO \'canal\'@\'%\';
FLUSH PRIVILEGES;
（在10.60.81.157机器中进入mysql创建canal用户，并赋予权限）
2.3. 上传包到两个远程节点
scp canal.deployer-1.0.25.tar.gz root@10.60.81.159:/hgx/
scp canal.deployer-1.0.25.tar.gz root@10.60.81.158:/hgx/
2.4．二个节点安装canal
tar –zxvf canal.deployer-1.0.25.tar.gz
mv canal.deployer-1.0.25 canal
3. canal配置文件
3.1. canal.properties
#################################################
######### common argument #############
#################################################
canal.id= 1
canal.ip=
canal.port= 11111
canal.zkServers=10.60.81.160:4180,10.60.81.161:4180,10.60.81.162:4180
# flush data to zk
canal.zookeeper.flush.period = 1000
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false
# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size = 1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60
# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30
# binlog filter config
#canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB
# binlog ddl isolation
canal.instance.get.ddl.isolation = false
#################################################
######### destinations #############
#################################################
canal.destinations= kafka
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5
#canal.instance.tsdb.spring.xml=classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml=classpath:spring/tsdb/mysql-tsdb.xml
canal.instance.global.mode = spring
canal.instance.global.lazy = false
#canal.instance.global.manager.address = 127.0.0.1:1099
#canal.instance.global.spring.xml = classpath:spring/local-instance.xml
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
3.2. instance.properties
#################################################
## mysql serverId
canal.instance.mysql.slaveId=1234 ##另外一台机器改成1235，保证slaveId不重复即可
# position info
canal.instance.master.address=10.60.81.157:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
canal.instance.tsdb.dbUsername=canal
canal.instance.tsdb.dbPassword=canal
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
# username/password
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.defaultDatabaseName=
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=.*\\\\..*
# table black regex
canal.instance.filter.black.regex=
#################################################
***上述两个配置文件，黄色标记部分为修改部分在两个服务器上一样***
4. 启动canal服务
两台机器分别执行：canal/bin/zkServer.sh
5. 查看是否启动成功
5.1. 在10.60.81.159查看日志
5.2. 在10.60.81.158查看日志
***上述两服务器日志文件(canal.log)出现如图所示，说明canal的HA搭建成功
以上是关于Mysql + canal + zookeeper环境搭建的主要内容，如果未能解决你的问题，请参考以下文章
canal挂载在zookeeper，java客户端client，连接报超时
Canal：部署CanalCanal Admin & 使用ZooKeeper进行集群管理
Flink 最佳实践之使用 Canal 同步 MySQL 数据至 TiDB