Redis Cluster两slave节点处理
Posted 疯狂的骆驼
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Redis Cluster两slave节点处理相关的知识，希望对你有一定的参考价值。
Redis Cluster两slave节点重建处理
背景
系统：CentOS 7.6
Redis Cluster
| IP | 端口 | 角色 |
| 192.168.100.11 | 6380 | Master |
| 192.168.100.11 | 6381 | slave |
| 192.168.100.12 | 6380 | Master |
| 192.168.100.12 | 6381 | Slave |
| 192.168.100.13 | 6380 | Master |
| 192.168.100.13 | 6381 | Slave |
因为某种原因导致（192.168.100.12）操作系统宕机后无法修复。
查看集群节点信息，可以看到redis cluster的3个master节点均正常
[root]# redis-cli -p 6380 -a redis密码 -c cluster nodes
4f61cdeb684fec56a82cc5f3cc72f0def5df05c3 :0@0 slave,fail,noaddr 297a72d200be658a37658bd0ed9bda01d96af43f 1567608093527 1567608091611 9 disconnected
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128078000 11 connected 10923-16383
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128077000 10 connected 0-5460
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 myself,master - 0 1684128076000 9 connected 5461-10922
3c2fdc0e8e7295b14aff8c6909b35c38f4939907 :0@0 slave,fail,noaddr 7d221e66c3bece129bbe013687feaea093cc3714 1567608092521 1567608091611 11 disconnected
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128079071 10 connected处理过程
从集群中剔除ID为：“4f61cdeb684fec56a82cc5f3cc72f0def5df05c3”、“3c2fdc0e8e7295b14aff8c6909b35c38f4939907”的节点
[root]# redis-cli -p 6380 -a redis密码 -c cluster forget 4f61cdeb684fec56a82cc5f3cc72f0def5df05c3
[root]# redis-cli -p 6380 -a redis密码 -c cluster forget 3c2fdc0e8e7295b14aff8c6909b35c38f4939907
再次查看集群节点信息，可以看到异常的两个节点从redis cluster中剔除
[root]# redis-cli -p 6380 -a redis密码 -c cluster nodes
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128261000 11 connected 10923-16383
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128261513 10 connected 0-5460
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 myself,master - 0 1684128260000 9 connected 5461-10922
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128259508 10 connected
待192.168.100.12上的redis服务重建后，重新加入节点
[root]# redis-cli -p 6380 -a redis密码 -c
127.0.0.1:6380> cluster meet 192.168.100.12 6380
OK
127.0.0.1:6380> cluster meet 192.168.100.12 6381
OK
查看redis cluster节点信息，可以看到新增的两个节点是以master的角色加入到集群中的
10.12.31.79:6380> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 master - 0 1684128613906 0 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128614907 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128615909 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128615000 11 connected 10923-16383
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128612502 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 myself,master - 0 1684128611000 12 connected
连接10.12.31.79:6380，配置为某个master的slave
10.12.31.79:6380> cluster replicate 297a72d200be658a37658bd0ed9bda01d96af43f
OK
10.12.31.79:6380> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 master - 0 1684128712119 0 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128711118 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128713121 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128711000 11 connected 10923-16383
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128711000 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 myself,slave 297a72d200be658a37658bd0ed9bda01d96af43f 0 1684128709000 9 connected
分别连接10.12.31.79:6381，配置为某个master的slave
10.12.31.79:6381> cluster replicate 7d221e66c3bece129bbe013687feaea093cc3714
OK
10.12.31.79:6381> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 myself,slave 7d221e66c3bece129bbe013687feaea093cc3714 0 1684128760000 11 connected
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128762000 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 slave 297a72d200be658a37658bd0ed9bda01d96af43f 0 1684128760000 9 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128763000 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128763957 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128763000 11 connected 10923-16383
后续根据节点分布可以重启master节点进行调整。
Redis cluster 简单搭建
环境
两台虚拟机虚拟6个节点，每台3个节点，创建出3 master，3 slave 环境
| 系统 | IP | Redis节点 |
|---|---|---|
| CentOS 6 | 192.168.153.12 | 7000、7001、7002 |
| CentOS 7 | 192.168.153.11 | 7003、7004、7005 |
Redis版本：
redis-3.0.6.tar.gz安装 Redis
Redis官方由于xx原因无法访问，Redis的包需要自行下载。
tar zxvf redis-3.0.6.tar.gz && cd redis-3.0.6 && make && make install创建 Redis 节点
在两台机器上分别创建一个 redis_cluster 文件夹，这里放置在 /usr/local 目录下。（路径和文件夹名都是可以随便定义的，你可以在 /tmp/ 下创建一个 xx 目录，做为集群目录也可以。）
mkdir -p /usr/local/redis_cluster/然后在 192.168.153.12 下创建 7000、7001、7002 3个目录。在 192.168.153.11 下创建7003、7004、7005 3个目录。
在192.168.153.12执行
mkdir -p /usr/local/redis_cluster/700{0..2}在192.168.153.11执行
mkdir -p /usr/local/redis_cluster/700{3..5}并将 redis.conf 分别拷贝到700* 目录里面。redis.conf在前面解压的源码包里面
cd redis-3.0.6 && cp redis.conf /usr/local/redis_cluster/7000
cp redis.conf /usr/local/redis_cluster/7001
... 以此类推分别修改每个700*目录下的 redis.conf
daemonize yes #做为守护进程运行
port 7000 #这里是7000目录下，所以改为7000，如果是7001目录下就修改为7001
bind 192.168.153.12 #本机ip
appendonly yes
cluster-enabled yes #开启集群
cluster-config-file nodes-7000.conf #集群配置文件，同 port 选项，在7001目录下则修改为7001
cluster-node-timeout 5000启动节点
执行命令的时候不会有回显。如果有回显，说明启动有问题。
192.168.153.12
redis-server /usr/local/redis_cluster/7000/redis.conf
redis-server /usr/local/redis_cluster/7001/redis.conf
redis-server /usr/local/redis_cluster/7002/redis.conf192.168.153.11
redis-server /usr/local/redis_cluster/7003/redis.conf
redis-server /usr/local/redis_cluster/7004/redis.conf
redis-server /usr/local/redis_cluster/7005/redis.conf检查启动情况
192.168.153.12
ps -ef|grep redis
root 4215 1 0 19:27 ? 00:00:13 redis-server 192.168.153.12:7000 [cluster]
root 4219 1 0 19:27 ? 00:00:13 redis-server 192.168.153.12:7001 [cluster]
root 4223 1 0 19:27 ? 00:00:13 redis-server 192.168.153.12:7002 [cluster]
root 5192 5133 0 21:32 pts/2 00:00:00 grep redis
192.168.153.11
ps -ef|grep redis
root 1138 1 0 14:19 ? 00:00:00 redis-server 192.168.153.11:7003 [cluster]
root 1142 1 0 14:19 ? 00:00:00 redis-server 192.168.153.11:7004 [cluster]
root 1146 1 0 14:19 ? 00:00:00 redis-server 192.168.153.11:7005 [cluster]创建集群
官方提供一个创建集群的工具redis-trib，它位于 Redis 源码的 src 文件夹中， 它是一个 Ruby 程序， 这个程序通过向实例发送特殊命令来完成创建新集群， 检查集群， 或者对集群进行重新分片（reshared）等工作。
所以要安装 Ruby 及 RubyGems。
yum install ruby rubygems -y由于众所周知的原因，Gems的源也是没法用的，我们需要将它替换为国内源。
gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/
gem sources -l
https://gems.ruby-china.org
# 确保只有 gems.ruby-china.org用 Gems 安装 Redis
gem install redis上面这个命令会提示你
redis requires Ruby version >= 2.2.2，让你升级Ruby
因为默认安装的是最新版 Redis 4.x 版本。而我本次安装的是 Redis 3.0.6 的，gem安装的也要配套,所以要用下面这个命令
gem install redis -v 3.0.6redis-trib.rb 这个工具在源码包下的 src文件夹，现在将它复制到 /usr/local/bin/ 目录，这样就可以直接在命令行下使用该工具了。
cp /tmp/redis-3.0.6/src/redis-trib.rb /usr/local/bin/ 运行 redis-trib.rb 工具创建集群
# redis-trib.rb create --replicas 1 192.168.153.12:7000 192.168.153.12:7001 192.168.153.12:7002 192.168.153.11:7003 192.168.153.11:7004 192.168.153.11:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.153.11:7003
192.168.153.12:7000
192.168.153.11:7004
Adding replica 192.168.153.12:7001 to 192.168.153.11:7003
Adding replica 192.168.153.11:7005 to 192.168.153.12:7000
Adding replica 192.168.153.12:7002 to 192.168.153.11:7004
M: 60aa133262cd817fa729cc2c1103a25e3a9b2311 192.168.153.12:7000
slots:5461-10922 (5462 slots) master
S: 30570cd99b32ffbc5f39ef7df58059a53d51ea50 192.168.153.12:7001
replicates b6454920091c772bb82f166683fd899fddffe35d
S: 560764d15cad022d0d83286d6a83f6aa880cd0e9 192.168.153.12:7002
replicates fa0fcc6e09c5dca5824f23f0b012b1eb3efe247b
M: b6454920091c772bb82f166683fd899fddffe35d 192.168.153.11:7003
slots:0-5460 (5461 slots) master
M: fa0fcc6e09c5dca5824f23f0b012b1eb3efe247b 192.168.153.11:7004
slots:10923-16383 (5461 slots) master
S: 2dc77357e6fb97f15a0fa03ab43e796ed8fb2e2a 192.168.153.11:7005
replicates 60aa133262cd817fa729cc2c1103a25e3a9b2311
Can I set the above configuration? (type ‘yes‘ to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 192.168.153.12:7000)
M: 60aa133262cd817fa729cc2c1103a25e3a9b2311 192.168.153.12:7000
slots:5461-10922 (5462 slots) master
M: 30570cd99b32ffbc5f39ef7df58059a53d51ea50 192.168.153.12:7001
slots: (0 slots) master
replicates b6454920091c772bb82f166683fd899fddffe35d
M: 560764d15cad022d0d83286d6a83f6aa880cd0e9 192.168.153.12:7002
slots: (0 slots) master
replicates fa0fcc6e09c5dca5824f23f0b012b1eb3efe247b
M: b6454920091c772bb82f166683fd899fddffe35d 192.168.153.11:7003
slots:0-5460 (5461 slots) master
M: fa0fcc6e09c5dca5824f23f0b012b1eb3efe247b 192.168.153.11:7004
slots:10923-16383 (5461 slots) master
M: 2dc77357e6fb97f15a0fa03ab43e796ed8fb2e2a 192.168.153.11:7005
slots: (0 slots) master
replicates 60aa133262cd817fa729cc2c1103a25e3a9b2311
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.集群认证
测试 Redis 集群比较简单的办法就是使用 redis-rb-cluster 或者 redis-cli 。注意端口变化
[[email protected] ~]# redis-cli -h 192.168.153.12 -c -p 7000
192.168.153.12:7000> set foo bar
-> Redirected to slot [12182] located at 192.168.153.11:7004
OK
192.168.153.11:7004> set hello world
-> Redirected to slot [866] located at 192.168.153.11:7003
OK
192.168.153.11:7003> get foo
-> Redirected to slot [12182] located at 192.168.153.11:7004
"bar"
192.168.153.11:7004> get hello
-> Redirected to slot [866] located at 192.168.153.11:7003
"world"以上是关于Redis Cluster两slave节点处理的主要内容，如果未能解决你的问题，请参考以下文章
Redis master/slave,sentinel,Cluster简单总结