纳尼？？Rabbitmq居然被一个逗号给坑了？？

Posted 2020-11-23

前言

这个问题发生在部署一套新的环境。搭建一个单节点的Rabbitmq，搭建步骤并没有按照我在单机版Rabbitmq部署这篇文章的步骤，而是按照同事写的部署文档搭建的。其中有一些细微的差别，比如修改配置文件。

部署环境：centos7

问题描述

步骤如下：
1、先安装erlang；
2、安装Rabbitmq的rpm包
3、修改配置文件
4、启动服务

但是在启动的时候，失败了：

[root@vm193-168-1-66 ~]# systemctl start rabbitmq-server
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

根据提示查看日志systemctl status rabbitmq-server.service：

[root@vm193-168-1-66 ~]# systemctl status rabbitmq-server.service
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2019-12-17 09:12:50 CST; 8s ago
  Process: 5378 ExecStop=/usr/sbin/rabbitmqctl shutdown (code=exited, status=69)
  Process: 5143 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=64)
 Main PID: 5143 (code=exited, status=64)
Dec 17 09:12:50 vm193-168-1-66 systemd[1]: Failed to start RabbitMQ broker.
Dec 17 09:12:50 vm193-168-1-66 systemd[1]: Unit rabbitmq-server.service entered failed state.
Dec 17 09:12:50 vm193-168-1-66 systemd[1]: rabbitmq-server.service failed.

并没有找到非常明显的原因

再查看系统日志，同样的结果：

[root@vm193-168-1-66 ~]# journalctl -xe
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * epmd reports: node ‘rabbit‘ not running at all
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: no other nodes on vm193-168-1-66
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * suggestion: start the node
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: Current node details:
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * node name: ‘rabbitmqcli-13769-rabbit@vm193-168-1-66‘
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * effective user‘s home directory: /var/lib/rabbitmq
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * Erlang cookie hash: WQikY88cKw2MYZAMQRGnMA==
Dec 17 09:15:42 vm193-168-1-66 systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rabbitmq-server.service has failed.
-- 
-- The result is failed.
Dec 17 09:15:42 vm193-168-1-66 systemd[1]: Unit rabbitmq-server.service entered failed state.
Dec 17 09:15:42 vm193-168-1-66 systemd[1]: rabbitmq-server.service failed.
Dec 17 09:15:43 vm193-168-1-66 sshd[13480]: Received disconnect from 125.133.100.146: 11: Bye Bye [preauth]
Dec 17 09:15:48 vm193-168-1-66 sshd[13949]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=2
Dec 17 09:15:48 vm193-168-1-66 sshd[13949]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Dec 17 09:15:50 vm193-168-1-66 sshd[13949]: Failed password for root from 210.212.237.67 port 35528 ssh2
Dec 17 09:15:50 vm193-168-1-66 sshd[13949]: Received disconnect from 210.212.237.67: 11: Bye Bye [preauth]
lines 1317-1338/1338 (END)

原因分析

由于上边的两段日志都没有找到明显的原因，于是尝试再次启动rabbitmq，同时tail -f开启messages日志：tail -f /var/log/messages，发现有如下日志内容（比较长，可略过这段日志直接看后边）：

Dec 17 09:26:48 vm193-168-1-66 systemd: rabbitmq-server.service holdoff time over, scheduling restart.
Dec 17 09:26:48 vm193-168-1-66 systemd: Cannot add dependency job for unit proc-sys-fs-binfmt_misc.automount, ignoring: Unit is masked.
Dec 17 09:26:48 vm193-168-1-66 systemd: Starting RabbitMQ broker...
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: 2019-12-17 09:26:48.706 [error] Unable to parse erlang terms from  RABBITMQ_CONFIG_FILE file: /etc/rabbitmq/rabbitmq.config
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: Error: {512,erl_parse,
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: ["syntax error before: ","‘]‘"]}
Dec 17 09:26:48 vm193-168-1-66 systemd: rabbitmq-server.service: main process exited, code=exited, status=64/n/a
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: Error: unable to perform an operation on node ‘rabbit@vm193-168-1-66‘. Please see diagnostics information and suggestions below.
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: Most common reasons for this are:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * CLI tool fails to authenticate with the server (e.g. due to CLI tool‘s Erlang cookie not matching that of the server)
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Target node is not running
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: In addition to the diagnostics info below:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Consult server logs on node rabbit@vm193-168-1-66
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * If target node is configured to use long node names, don‘t forget to use --longnames with CLI tools
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: DIAGNOSTICS
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: ========
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: attempted to contact: [‘rabbit@vm193-168-1-66‘]
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: rabbit@vm193-168-1-66:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * connected to epmd (port 4369) on vm193-168-1-66
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * epmd reports: node ‘rabbit‘ not running at all
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: no other nodes on vm193-168-1-66
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * suggestion: start the node
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: Current node details:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * node name: ‘rabbitmqcli-14476-rabbit@vm193-168-1-66‘
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * effective user‘s home directory: /var/lib/rabbitmq
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Erlang cookie hash: WQikY88cKw2MYZAMQRGnMA==
Dec 17 09:26:49 vm193-168-1-66 systemd: Failed to start RabbitMQ broker.
Dec 17 09:26:49 vm193-168-1-66 systemd: Unit rabbitmq-server.service entered failed state.
Dec 17 09:26:49 vm193-168-1-66 systemd: rabbitmq-server.service failed.

从上面的/var/log/massages日志找到了日志里下面三行重点提示：

    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: 2019-12-17 09:26:48.706 [error] Unable to parse erlang terms from  RABBITMQ_CONFIG_FILE file: /etc/rabbitmq/rabbitmq.config
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: Error: {512,erl_parse,
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: ["syntax error before: ","‘]‘"]}

可以看出，在512行前面存在语法问题。

另外可以用下面的命令检测配置文件/etc/rabbitmq/rabbitmq.config是否正确

[root@vm193-168-1-66 lib]# erl -noshell -eval ‘io:format("~p~n", [file:consult("/etc/rabbitmq/rabbitmq.config")]).‘ -eval ‘init:stop().‘
{error,{512,erl_parse,["syntax error before: ","‘]‘"]}}     #结果也是提示在512行之前有语法错误
[root@vm193-168-1-66 lib]# 

因为整个部署过程中只有/etc/rabbitmq/rabbitmq.config这个配置文件改动了唯一的一行内容：第66行，去掉了{loopback_users, []},这一行前面的配置。

仔细检查文件，发现{loopback_users, []},这一行后边的内容全部注释掉了的，因此其实应该是要去掉{loopback_users, []},末尾的逗号的：

解决办法

尝试着去掉了行尾的逗号，再次检测配置文件，发现结果是OK了：

[root@vm193-168-1-66 rabbitmq]# erl -noshell -eval ‘io:format("~p~n", [file:consult("/etc/rabbitmq/rabbitmq.config")]).‘ -eval ‘init:stop().‘
{ok,[[{rabbit,[{loopback_users,[]}]},
      {kernel,[]},
      {rabbitmq_management,[]},
      {rabbitmq_shovel,[{shovels,[]}]},
      {rabbitmq_stomp,[]},
      {rabbitmq_mqtt,[]},
      {rabbitmq_amqp1_0,[]},
      {rabbitmq_auth_backend_ldap,[]},
      {lager,[]}]]}
[root@vm193-168-1-66 rabbitmq]# 

再次启动rabbitmq，就成功了：

后记：

这是一个教训：对于这种注释掉很多内容的配置文件，修改完了之后，最好过滤掉注释了的内容，查看一下剩下的行有没有什么格式、符号的问题！否则，就只有靠爬坑减肥了......