记一次redis集群异常.(error) CLUSTERDOWN The cluster is down

Posted 2023-04-19

参考技术A 之前使用docker-compose在测试服务器上搭建了一个redis测试集群.运行了很久时间都没有异常.
无奈机房有次事故,服务器被无故重启了.然后重启redis集群也没有任何异常,但是get,set等方法就出
现标题中的错误.
下面是错误信息:

看到大概就猜到什么问题了,因为集群就是要满足所有的16364个槽点全部分配才会成功。

通过上面的错误信息就知道,是缺少槽点.由于数据比较多,人工去审查少了那些槽点
挺费劲的,干脆直接用脚本跑(不管你有没有,我一律加上完事)

执行脚本后.重启redis

cluster_state:fail => cluster_state:ok

从上数据得出,redis已经正常了.(说明,这里搭建的redis是三主三从,主6378,6377,6376,从6375,6374,6373)

至此,redis集群问题已经解决.

我脚本里写的只是往主节点6378 addslots 槽点的
其他主节点也有可能会继续报错
所以 把其它主节点也 addslots 槽点(0-168363)

记一次RabbitMQ服务器异常断电之后，服务重启异常的处理过程

问题描述：
机房突然停电，rabbitmq的主机异常断电，集群服务全部需要重启。但是在执行service rabbitmq-server start 启动主节点服务的时候，没有反应，服务没有启动，命令也执行卡住了。必须Ctrl+C结束进程：

[[email protected] rabbitmq]# service rabbitmq-server start
Starting rabbitmq-server (via systemctl):  ^C
[[email protected] rabbitmq]#

查看/var/log/rabbitmq/startup_log 发现有如下报错信息：

[[email protected] rabbitmq]# tail -1000 startup_log 
BOOT FAILED
===========

Timeout contacting cluster nodes: [‘[email protected]‘,‘[email protected]‘].

BACKGROUND
==========

This cluster node was shut down while other nodes were still running.
To avoid losing data, you should start the other nodes first, then
start this one. To force this node to start, first invoke
"rabbitmqctl force_boot". If you do so, any changes made on other
cluster nodes after this one was shut down may be lost.

DIAGNOSTICS
===========

attempted to contact: [‘[email protected]‘,‘[email protected]‘]

[email protected]:
  * connected to epmd (port 4369) on s1-1
  * epmd reports: node ‘rabbit‘ not running at all
                  no other nodes on s1-1
  * suggestion: start the node
[email protected]:
  * unable to connect to epmd (port 4369) on slave-2: address (cannot connect to host/port)

current node details:
- node name: ‘[email protected]‘
- home dir: /var/lib/rabbitmq
- cookie hash: oqRyxdQQXO31mzM8U0ysNA==

{"init terminating in do_boot",timeout_waiting_for_tables}

解决：
根据/var/log/rabbitmq/startup_log日志最后的报错信息{"init terminating in do_boot",timeout_waiting_for_tables}，在网上查询到原因，和linux下rabbitmq大致有关系的，主要有这三种说法：
1、5672端口被占用了，导致服务起不来
2、/var/log/rabbitmq目录的权限不对，需要重新赋权限
3、/var/lib/rabbitmq/mnesia这个数据目录异常，删除原来的数据目录，重新启动服务

方法一：检查端口，发现并没有5672的这个端口：

[[email protected] rabbitmq]# netstat -anp|grep 5672
tcp        0      0 193.168.0.90:3306       131.10.10.120:56727     ESTABLISHED 3666/mysqld         
tcp6       0      0 193.168.0.90:56727      193.168.0.93:9092       ESTABLISHED 4891/java           
[[email protected] rabbitmq]# netstat -ano|grep 5672
tcp        0      0 193.168.0.90:3306       131.10.10.120:56727     ESTABLISHED keepalive (54.12/0/0)
tcp6       0      0 193.168.0.90:56727      193.168.0.93:9092       ESTABLISHED keepalive (50.53/0/0)

方法二：修改/var/log/rabbitmq权限，进去/var/log/rabbitmq/目录，发现该目录下面的文件确实存在权限不统一的问题，于是修改权限重新启动服务，还是失败：

[[email protected] rabbitmq]# cd /var/log/rabbitmq/
[[email protected] rabbitmq]# ll
total 11740
-rw-r--r--  1 rabbitmq rabbitmq   29075 May 14 11:14 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq  159053 Apr 29 03:19 [email protected]g-20180429.gz
-rw-r--r--  1 rabbitmq rabbitmq 1756006 May  7 03:11 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq 9881632 May 13 03:17 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq    3108 May 14 11:14 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq     950 Apr 28 14:22 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq    1677 May  4 15:25 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq  159530 May 11 10:11 [email protected]
-rw-r--r--  1 root     root           0 May  7 15:14 shutdown_err
-rw-r--r--  1 root     root          44 May  7 15:14 shutdown_log
-rw-r--r--. 1 root     root         103 May 14 11:15 startup_err
-rw-r--r--. 1 root     root        1323 May 14 11:15 startup_log
[[email protected] rabbitmq]# chown -R rabbitmq:rabbitmq /var/log/rabbitmq/
[[email protected] rabbitmq]# ll
total 11740
-rw-r--r--  1 rabbitmq rabbitmq   29075 May 14 11:14 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq  159053 Apr 29 03:19 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq 1756006 May  7 03:11 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq 9881632 May 13 03:17 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq    3108 May 14 11:14 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq     950 Apr 28 14:22 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq    1677 May  4 15:25 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq  159530 May 11 10:11 [email protected]
-rw-r--r--  1 rabbitmq rabbitmq       0 May  7 15:14 shutdown_err
-rw-r--r--  1 rabbitmq rabbitmq      44 May  7 15:14 shutdown_log
-rw-r--r--. 1 rabbitmq rabbitmq     103 May 14 11:15 startup_err
-rw-r--r--. 1 rabbitmq rabbitmq    1323 May 14 11:15 startup_log

但是修改了权限之后，服务还是起不来：

[[email protected] rabbitmq]# service rabbitmq-server start    
Starting rabbitmq-server (via systemctl):  ^C
[[email protected] rabbitmq]# 

方法三：删除原有的数据目录，然后重新启动服务

[[email protected] rabbitmq]# cd /var/lib/rabbitmq/
[[email protected] rabbitmq]# ll
total 4020
-rw-r----- 1 rabbitmq rabbitmq 4114398 May 14 11:15 erl_crash.dump
drwxr-x--- 4 rabbitmq rabbitmq      94 May 14 11:38 mnesia
[[email protected] rabbitmq]# mv mnesia mnesia.bak
[[email protected] rabbitmq]# ll
total 4020
-rw-r----- 1 rabbitmq rabbitmq 4114398 May 14 11:15 erl_crash.dump
drwxr-x--- 4 rabbitmq rabbitmq      94 May 14 11:38 mnesia.bak

然后重新启动服务成功：

[[email protected] rabbitmq]# service rabbitmq-server start    
Starting rabbitmq-server (via systemctl):                  [  OK  ]
[[email protected] rabbitmq]# ps -ef|grep rabbitmq
rabbitmq  3131     1  0 May13 ?        00:00:00 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
root     19908     1  0 11:41 ?        00:00:00 /bin/sh /etc/rc.d/init.d/rabbitmq-server start
root     19910 19908  0 11:41 ?        00:00:00 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/rabbitmq-server
root     19914 19910  0 11:41 ?        00:00:00 /bin/sh /usr/sbin/rabbitmq-server
root     19932 19914  0 11:41 ?        00:00:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmq-server 
rabbitmq 19935 19932  0 11:41 ?        00:00:00 /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 20158 19935 17 11:41 ?        00:00:04 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -B i -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/ebin -noshell -noinput -s rabbit boot -sname [email protected] -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/[email protected]"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/[email protected]"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/[email protected]" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/[email protected]" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 20316 20158  0 11:41 ?        00:00:00 inet_gethost 4
rabbitmq 20317 20316  0 11:41 ?        00:00:00 inet_gethost 4
root     20406 16497  0 11:42 pts/5    00:00:00 grep --color=auto rabbitmq
[[email protected] rabbitmq]# 

注意，这只是主节点的处理方法，在两台从节点，需要做如下操作：
1、检查两台从节点的/var/lib/rabbitmq/.erlang.cookie文件内容是否和主节点是保持一致的
2、删除原有的数据/var/lib/rabbitmq/mnesia目录，执行rabbitmq-server -detached重新启动服务
3、在两台从节点上执行下面的命令，重新加入集群：

#rabbitmqctl stop_app 
#rabbitmqctl reset 
#rabbitmqctl join_cluster [email protected]   # [email protected]里面的master-2是主节点的主机名，注意修改
#rabbitmqctl start_app

备节点执行完毕上面的步骤之后，需要在主节点验证集群的正确性：

[[email protected] rabbitmq]# rabbitmqctl cluster_status
Cluster status of node ‘[email protected]‘
[{nodes,[{disc,[‘[email protected]‘,‘[email protected]‘,‘[email protected]‘]}]},
 {running_nodes,[‘[email protected]‘,‘[email protected]‘,‘[email protected]‘]},
 {cluster_name,<<"[email protected]">>},
 {partitions,[]},
 {alarms,[{‘[email protected]‘,[]},
          {‘[email protected]‘,[nodedown]},
          {‘[email protected]‘,[]}]}]

集群验证成功之后，使用主节点IP+端口登录，界面发现输入之前的用户名和密码，已经登录不进去了，需要在主节点重新创建管理用户并且赋予密码和访问权限

[[email protected] rabbitmq]# rabbitmqctl add_user admin password123   #创建用户和密码
Creating user "admin"
[[email protected] rabbitmq]# rabbitmqctl set_user_tags admin administrator   #给用户赋予管理员权限  
Setting tags for user "admin" to [administrator]
[[email protected] rabbitmq]#  rabbitmqctl  set_permissions -p "/" admin ".*" ".*" ".*"  #给管理员赋予访问权限  
Setting permissions for user "admin" in vhost "/"

然后再使用主节点的IP+端口重新登录，输入用户名和密码，服务恢复。

后记：问题解决之后，发现还有一种说法：
Are you running in a clustered configuration? If so, rabbit might be?waiting for the other nodes to come up.?