Redis集群间数据迁移与对比

Posted 2023-04-26

参考技术A https://github.com/alibaba/RedisFullCheck

redis-full-check 通过全量对比源端和目的端的redis中的数据的方式来进行数据校验，其比较方式通过多轮次比较：每次都会抓取源和目的端的数据进行差异化比较，记录不一致的数据进入下轮对比（记录在sqlite3 db中）。然后通过多伦比较不断收敛，减少因数据增量同步导致的源库和目的库的数据不一致。最后sqlite中存在的数据就是最终的差异结果。

结果会保存在sqlite3 db file中，不指定的话，就是当前目录的 result.db 文件：比如有3轮比较，那么会有 result.db.1 ， result.db.2 ， result.db.3 3个文件

Redis单实例数据迁移到集群

环境说明

单机redis

192.168.41.101:6379

redis集群

192.168.41.101:7000 master
192.168.41.101:7001 master
192.168.41.101:7002

192.168.41.102:7000 master
192.168.41.102:7001
192.168.41.102:7002

迁移步骤

查看集群状态及节点槽分布

[[email protected] bin]# ./redis-cli -c -p 7000
127.0.0.1:7000> cluster nodes
ab6d9f956de325cb4cf001abc31df365a5db5234 192.168.41.102:7001 slave 923c9ea13ec8fc19bed309dfbfad094320e1ca41 0 1560304528263 7 connected
48629733acbb8a580a39403dfac92845d63c97b3 192.168.41.101:7001 master - 0 1560304529768 2 connected 5461-10921
2cec8ac00f760c45d86f7903cefad85ec36704e7 192.168.41.102:7002 slave 923c9ea13ec8fc19bed309dfbfad094320e1ca41 0 1560304531271 7 connected
923c9ea13ec8fc19bed309dfbfad094320e1ca41 192.168.41.101:7000 myself,master - 0 0 7 connected 0-5460
e65b319e83997ed6f5323a26aaccba3f35522cbd 192.168.41.101:7002 slave 923c9ea13ec8fc19bed309dfbfad094320e1ca41 0 1560304532274 7 connected
5c0888d5bcceda2904311cbd5405596217c48105 192.168.41.102:7000 master - 0 1560304530268 4 connected 10922-16383

[[email protected] src]# ./redis-trib.rb check 192.168.41.101:7000
>>> Performing Cluster Check (using node 192.168.41.101:7000)
M: 923c9ea13ec8fc19bed309dfbfad094320e1ca41 192.168.41.101:7000
   slots:0-5460 (5461 slots) master
   3 additional replica(s)
S: ab6d9f956de325cb4cf001abc31df365a5db5234 192.168.41.102:7001
   slots: (0 slots) slave
   replicates 923c9ea13ec8fc19bed309dfbfad094320e1ca41
M: 48629733acbb8a580a39403dfac92845d63c97b3 192.168.41.101:7001
   slots:5461-10921 (5461 slots) master
   0 additional replica(s)
S: 2cec8ac00f760c45d86f7903cefad85ec36704e7 192.168.41.102:7002
   slots: (0 slots) slave
   replicates 923c9ea13ec8fc19bed309dfbfad094320e1ca41
S: e65b319e83997ed6f5323a26aaccba3f35522cbd 192.168.41.101:7002
   slots: (0 slots) slave
   replicates 923c9ea13ec8fc19bed309dfbfad094320e1ca41
M: 5c0888d5bcceda2904311cbd5405596217c48105 192.168.41.102:7000
   slots:10922-16383 (5462 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

 

把101:7001和102:7000两个master的slots转移到101:7000上

./redis-trib.rb reshard --from 5c0888d5bcceda2904311cbd5405596217c48105 --to 923c9ea13ec8fc19bed309dfbfad094320e1ca41  --slots 5462 --yes 192.168.41.101:7000

./redis-trib.rb reshard --from 48629733acbb8a580a39403dfac92845d63c97b3 --to 923c9ea13ec8fc19bed309dfbfad094320e1ca41  --slots 5461 --yes 192.168.41.101:7000

 

查看转以后槽点分配

>>> Performing Cluster Check (using node 192.168.41.101:7000)
M: 923c9ea13ec8fc19bed309dfbfad094320e1ca41 192.168.41.101:7000
   slots:0-16383 (16384 slots) master
   3 additional replica(s)
S: ab6d9f956de325cb4cf001abc31df365a5db5234 192.168.41.102:7001
   slots: (0 slots) slave
   replicates 923c9ea13ec8fc19bed309dfbfad094320e1ca41
M: 48629733acbb8a580a39403dfac92845d63c97b3 192.168.41.101:7001
   slots: (0 slots) master
   0 additional replica(s)
S: 2cec8ac00f760c45d86f7903cefad85ec36704e7 192.168.41.102:7002
   slots: (0 slots) slave
   replicates 923c9ea13ec8fc19bed309dfbfad094320e1ca41
S: e65b319e83997ed6f5323a26aaccba3f35522cbd 192.168.41.101:7002
   slots: (0 slots) slave
   replicates 923c9ea13ec8fc19bed309dfbfad094320e1ca41
M: 5c0888d5bcceda2904311cbd5405596217c48105 192.168.41.102:7000
   slots: (0 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

把192.168.41.101:6379的appendonly.aof 文件复制到101:7000的aof文件目录下

cp 6379目录/appendonly.aof   7000目录

重启101:7000节点加载aof文件

[[email protected] bin]# ./redis-cli -c -p 7000 shutdown

[[email protected] bin]# ./redis-server ../redis.conf 

查看数据是否正确

dbsize

 

将101:7000的槽 均匀分配给其他两个master节点

./redis-trib.rb reshard --from 923c9ea13ec8fc19bed309dfbfad094320e1ca41  --to 5c0888d5bcceda2904311cbd5405596217c48105  --slots 5462 --yes 192.168.41.101:7000

./redis-trib.rb reshard --from 923c9ea13ec8fc19bed309dfbfad094320e1ca41  --to 48629733acbb8a580a39403dfac92845d63c97b3  --slots 5461 --yes 192.168.41.101:7000

再次查看集群状态和槽分配。