Redis Cluster部署手册
Posted 一泽涟漪
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Redis Cluster部署手册相关的知识，希望对你有一定的参考价值。
Redis Cluster：Redis Cluster提供了一种真正的实时的高可用解决方案，在Redis Cluster集群中，任意一个节点的宕机均不会对服务产生影响，slave角色的节点可以无缝接替宕机的master节点对外继续提供服务。
例如，在一个三主三从的Redis Cluster中，Redis Cluster供定义了16384(0~16383)个slots（槽），并将这些slots均匀分布Redis Cluster的三个主节点中，如0~5000个slots属于主节点A，5001~10000个slots属于主节点B，10001~16383个slot属于主节点C，Redis中任意一个key经过CRC16校验后的校验值始终在0~16383范围中，Redis Cluster根据该key的校验值将该Key存入和该slot值一致的节点上，A1、B1、C1分别为A、B、C的从节点，自动从对应的主节点同步slots信息和slots中的key数据，当主节点宕机后，从节点立刻执行故障转义变成主节点继续提供服务。
一个三主三从的Redis Cluster如下所示
我们以三主三从的架构进行Redis Cluster环境部署
| 主机名 | IP地址 | 角色 | 安装目录 | 版本 |
| redis-master-01 | 172.16.101.54 | Master | /usr/local/redis | Version 3.12 |
| redis-master-02 | 172.16.101.55 | Master | ||
| redis-master-03 | 172.16.101.56 | Master | ||
| redis-slave-01 | 172.16.101.58 | Slave | ||
| redis-slave-02 | 172.16.101.59 | Slave | ||
| redis-slave-03 | 172.16.101.60 | Slave |
一.各节点系统环境配置
1.新建运行redis的用户和组
# groupadd -r dba # useradd -r -g dba -G root redis # cat /home/redis/.bash_profile ................ PATH=/usr/local/redis/src:$PATH:$HOME/.local/bin:$HOME/bin ................
2.系统内核参数优化
# cat /etc/sysctl.conf ....................... vm.overcommit_memory=1 vm.swappiness=1 fs.aio-max-nr=1048576 fs.file-max= 7672460 net.ipv4.ip_local_port_range=9000 65500 net.core.rmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_default=262144 net.core.wmem_max=1048586 net.core.somaxconn = 2048 kernel.sem= 50100 64128000 50100 1280 kernel.shmall=5242880 kernel.shmmax=12884901888 .......................
3. redis运行资源优化
# cat /etc/security/limits.conf .................................. redis soft nproc 65536 redis hard nproc 65536 redis soft nofile 65536 redis hard nofile 65536 ..................................
4. 大页面文件优化
# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
二.各节点redis安装
# mkdir /usr/local/redis # chown -R redis.dba /usr/local/redis $ cd /usr/local/redis $ tar -zxf redis-3.2.12.tar.gz $ mv redis-3.2.12/* . $ rm -rf redis-3.2.12* $ make $ mkdir {data,log,conf} $ mv redis.conf conf
三. 配置各节点redis配置文件，如下所示，这里仅列出主要参数
bind 0.0.0.0 protected-mode no tcp-backlog 2048 timeout 86400 tcp-keepalive 600 daemonize yes supervised systemd pidfile "/var/run/redis_6379.pid" loglevel notice logfile "/usr/local/redis/log/redis.log" databases 64 save 14400 1 save 7200 10 save 3600 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename "dump.rdb" dir "/usr/local/redis/data" replica-serve-stale-data yes replica-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-ping-replica-period 2 repl-timeout 10 repl-disable-tcp-nodelay no repl-backlog-size 10mb repl-backlog-ttl 7200 replica-priority 100 maxclients 65000 maxmemory 12gb maxmemory-policy allkeys-lru maxmemory-samples 5 replica-ignore-maxmemory yes lazyfree-lazy-eviction no lazyfree-lazy-expire yes lazyfree-lazy-server-del yes replica-lazy-flush no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes slowlog-log-slower-than 10000 slowlog-max-len 256 aof-use-rdb-preamble yes cluster-enabled yes cluster-config-file "/usr/local/redis/nodes-6379.conf" cluster-node-timeout 10000 cluster-replica-validity-factor 0 cluster-migration-barrier 1 cluster-require-full-coverage no cluster-replica-no-failover no
四.将redis配置为系统服务
# cat /usr/lib/systemd/system/redis.service [Unit] Description=Redis In-Memory Data Store Documentation=https://redis.io/documentation After=syslog.target After=network.target [Service] Type=notify User=redis Group=dba Restart=no # Disable OOM kill on the Redis OOMScoreAdjust=-1000 Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj Environment=PG_OOM_ADJUST_VALUE=0 ExecStart=/usr/local/redis/src/redis-server /usr/local/redis/conf/redis.conf ExecStop=/bin/kill -15 $MAINPID LimitNOFILE=65536 [Install] WantedBy=multi-user.target
# systemctl daemon-reload
# systemctl enable redis
# systemctl start redis
# systemctl status redis
各节点启动成功后，会自动生成一个长度为40字节的唯一的node-id,且在默认开启集群功能后，各节点均认为自己是主节点，从log中我们可以看出
172.16.101.54
25630:M 12 Apr 00:59:59.477 * No cluster configuration found, I\'m 89a83af689b194b18c7f5c0dae105c329a6a831f
172.16.101.55
6232:M 12 Apr 00:59:59.450 * No cluster configuration found, I\'m 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112
172.16.101.56
28099:M 12 Apr 00:59:59.478 * No cluster configuration found, I\'m 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04
172.16.101.58
25490:M 12 Apr 01:01:07.832 * No cluster configuration found, I\'m 310edd63d50d3d1412e6413b9c4d2ff31e2fae5f
172.16.101.59
27836:M 12 Apr 01:01:07.835 * No cluster configuration found, I\'m 63eef92da475574c62485543ac57898d13618520
172.16.101.60
15314:M 12 Apr 01:01:07.839 * No cluster configuration found, I\'m c7095b2f9adf4adada4fb0afd953ab7439ccc058
查看当前集群节点信息
redis-master-01:6379> info cluster # Cluster cluster_enabled:1 redis-master-01:6379> cluster info cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:1 cluster_size:0 cluster_current_epoch:0 cluster_my_epoch:0 cluster_stats_messages_sent:0 cluster_stats_messages_received:0 redis-master-01:6379> cluster nodes 89a83af689b194b18c7f5c0dae105c329a6a831f :6379 myself,master - 0 0 0 connected
通过以上可以看出
- 当前集群功能已经开启，cluster_enabled:1
- 集群功能fail状态，cluster_state:fail
- 集群未分配任何slots，cluster_slots_assigned:0
- 集群中只能查看自身节点的简单信息，cluster_known_nodes:1、 89a83af689b194b18c7f5c0dae105c329a6a831f :6379 myself,master - 0 0 0 connected
五.集群部署
1. 将各个节点加入集群
可在任意一个主节点操作，如在redis-master-01节点上，将其他所有节点添加到集群中
redis-master-01:6379> cluster meet 172.16.101.55 6379 redis-master-01:6379> cluster meet 172.16.101.56 6379 redis-master-01:6379> cluster meet 172.16.101.58 6379 redis-master-01:6379> cluster meet 172.16.101.59 6379 redis-master-01:6379> cluster meet 172.16.101.60 6379
添加完之后再次查看集群节点信息
redis-master-01:6379> cluster info ............................. cluster_known_nodes:6 .............................
redis-master-01:6379> cluster nodes 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 172.16.101.55:6379 master - 0 1586625723577 1 connected c7095b2f9adf4adada4fb0afd953ab7439ccc058 172.16.101.60:6379 master - 0 1586625721574 0 connected 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 172.16.101.56:6379 master - 0 1586625723076 2 connected 63eef92da475574c62485543ac57898d13618520 172.16.101.59:6379 master - 0 1586625722075 4 connected 89a83af689b194b18c7f5c0dae105c329a6a831f 172.16.101.54:6379 myself,master - 0 0 5 connected 310edd63d50d3d1412e6413b9c4d2ff31e2fae5f 172.16.101.58:6379 master - 0 1586625723577 3 connected
2. 为各个主节点分配slots
[redis@redis-master-01 ~]$ redis-cli -h redis-master-01 cluster addslots {0..5000} [redis@redis-master-01 ~]$ redis-cli -h redis-master-02 cluster addslots {5001..10000} [redis@redis-master-01 ~]$ redis-cli -h redis-master-03 cluster addslots {10001..16383}
这里需要注意的点：
- 如果一个slot已经分配给一个主节点后，无法再分配给另外一个节点的，否则会报错“(error) ERR Slot * is already busy”
- slot的最大数量为16384（0~16383），如果分配的数量大于16384，会报错“(error) ERR Invalid or out of range slot”
再次查看当前集群信息
redis-master-01:6379> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:5 cluster_my_epoch:5 cluster_stats_messages_sent:2573 cluster_stats_messages_received:2573 redis-master-01:6379> cluster nodes 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 172.16.101.55:6379 master - 0 1586626222109 1 connected 5001-10000 c7095b2f9adf4adada4fb0afd953ab7439ccc058 172.16.101.60:6379 master - 0 1586626221105 0 connected 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 172.16.101.56:6379 master - 0 1586626222612 2 connected 10001-16383 63eef92da475574c62485543ac57898d13618520 172.16.101.59:6379 master - 0 1586626220600 4 connected 89a83af689b194b18c7f5c0dae105c329a6a831f 172.16.101.54:6379 myself,master - 0 0 5 connected 0-5000 310edd63d50d3d1412e6413b9c4d2ff31e2fae5f 172.16.101.58:6379 master - 0 1586626221607 3 connected
通过以上可以看出，当前集群共分配了16384个slots，且16484个slots均在线，每个主节点分配的slots范围如“cluster nodes”返回值中所示。我们并未给172.16.101.58/59/60节点分配slots，所以上图我们看到该三个节点依然没有任何slots产生。
此时，一个三主节点的集群已经可以正常提供服务，但是任何一个主节点宕机之后都会导致集群失效。
3. 为每个主节点添加对应的从节点
添加从节点需要在每个从节点上操作
将redis-slave-01添加为主节点redis-master-01的从节点，在redis-slave-01上操作
redis-slave-01:6379> cluster replicate 89a83af689b194b18c7f5c0dae105c329a6a831f
其中89a83af689b194b18c7f5c0dae105c329a6a831f为主节点redis-master-01的node-id
同理将redis-slave-02/03添加为redis-master-02/03的从节点
redis-slave-02:6379> cluster replicate 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 redis-slave-03:6379> cluster replicate 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04
添加完成之后，我们再次查看集群信息
[redis@redis-master-01 ~]$ redis-cli -h redis-master-01 cluster nodes | sort -k 2 89a83af689b194b18c7f5c0dae105c329a6a831f 172.16.101.54:6379 myself,master - 0 0 5 connected 0-5000 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 172.16.101.55:6379 master - 0 1586627102928 1 connected 5001-10000 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 172.16.101.56:6379 master - 0 1586627103932 2 connected 10001-16383 310edd63d50d3d1412e6413b9c4d2ff31e2fae5f 172.16.101.58:6379 slave 89a83af689b194b18c7f5c0dae105c329a6a831f 0 1586627103932 5 connected 63eef92da475574c62485543ac57898d13618520 172.16.101.59:6379 slave 6c9558d05f3d5a1b2dac4ebe31d581b1581f6112 0 1586627102426 4 connected c7095b2f9adf4adada4fb0afd953ab7439ccc058 172.16.101.60:6379 slave 684908fde2ccc85fa2caf2fbbbfaa0e9f470cf04 0 1586627102928 2 connected
此时，一个三主三从的Redis Cluster已经部署完成。
以上是关于Redis Cluster部署手册的主要内容，如果未能解决你的问题，请参考以下文章