Corosync+Pacemaker+DRBD+NFS高可用实例配置
Posted koumm
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Corosync+Pacemaker+DRBD+NFS高可用实例配置相关的知识，希望对你有一定的参考价值。
环境说明:
操作系统: CentOS 6.6 x64,本文采用rpm方式安装corosync+pacemaker+drbd+nfs。
本文与上文配置进行了一个对比，实现相同的功能，具体哪个好，还是根据需求以及对哪个方案理解比较透，Heartbeat+DRBD+NFS高可用实例配置http://koumm.blog.51cto.com/703525/1737702
一、双机Heartbeat配置
1. app1,app2配置hosts文件，以及主机名。
[root@app1 soft]# vi /etc/hosts
127.0.0.1 localhost
localhost.localdomain localhost4 localhost4.localdomain4
::1
localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.24 app1
192.168.0.25 app2
10.10.10.24 app1-priv
10.10.10.25 app2-priv
说明：10段是心跳IP, 192.168段是业务IP, 采用VIP地址是192.168.0.26。
2. 关闭selinux与防火墙
sed -i \'/SELINUX/s/enforcing/disabled/\' /etc/selinux/config
setenforce 0
chkconfig iptables off
service iptables stop
3. 配置各节点ssh互信，好像可配\\可不配,方便管理。
app1:
[root@app1 ~]# ssh-keygen -t rsa -f ~/.ssh/id_rsa -P \'\'
[root@app1 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@app2
app2:
[root@app2 ~]# ssh-keygen -t rsa -f ~/.ssh/id_rsa -P \'\'
[root@app2 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@app1
二、DRDB安装配置
1. app1,app2配置hosts文件以及准备磁盘分区
app1: /dev/sdb1 —> app2: /dev/sdb1
2. app1,app2安装drbd并安装
(1) 下载drbd安装包, CentOS6.6采用kmod-drbd84-8.4.5-504.1安装包才可用。
http://rpm.pbone.net/
drbd84-utils-8.9.1-1.el6.elrepo.x86_64.rpm
kmod-drbd84-8.4.5-504.1.el6.x86_64.rpm
# rpm -ivh drbd84-utils-8.9.5-1.el6.elrepo.x86_64.rpm
kmod-drbd84-8.4.5-504.1.el6.x86_64.rpm
Preparing...
########################################### [100%]
1:drbd84-utils ########################################### [ 50%]
2:kmod-drbd84 ###########################################
[100%]
Working. This may take some time ...
Done.
#
(2) 加载DRBD到内核模块
app1,app2分别操作，并加入到/etc/rc.local文件中。
modprobe drbd
lsmode |grep
drbd
3. 创建修改配置文件。节点1，节点2一样配置。
[root@app1 ~]# vi /etc/drbd.d/global_common.conf
global {
usage-count no;
}
common {
protocol C;
disk {
on-io-error detach;
no-disk-flushes;
no-md-flushes;
}
net {
sndbuf-size 512k;
max-buffers 8000;
unplug-watermark 1024;
max-epoch-size
8000;
cram-hmac-alg "sha1";
shared-secret "hdhwXes23sYEhart8t";
after-sb-0pri
disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
syncer {
rate 300M;
al-extents 517;
}
}
resource data {
on app1 {
device /dev/drbd0;
disk /dev/sdb1;
address
10.10.10.24:7788;
meta-disk internal;
}
on app2 {
device /dev/drbd0;
disk /dev/sdb1;
address 10.10.10.25:7788;
meta-disk internal;
}
}
4. 初始化资源
在app1和app2上分别执行：
# drbdadm create-md data
initializing activity log
NOT initializing bitmap
Writing meta
data...
New drbd meta data block successfully created.
5. 启动服务
在app1和app2上分别执行：或采用 drbdadm up data
# service drbd start
Starting DRBD resources: [
create res: data
prepare disk:
data
adjust disk: data
adjust net: data
]
..........
#
6. 查看启动状态, 两节点应均处于Secondary状态。
cat /proc/drbd #或者直接使用命令drbd-overview
节点1:
[root@app1 drbd.d]# cat /proc/drbd
version: 8.4.5
(api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96
build by root@node1.magedu.com, 2015-01-02 12:06:20
0: cs:Connected
ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0
dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:20964116
节点2:
[root@app2 drbd.d]# cat /proc/drbd
version: 8.4.5
(api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96
build by root@node1.magedu.com, 2015-01-02 12:06:20
0: cs:Connected
ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0
dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:20964116
7. 将其中一个节点配置为主节点
我们需要将其中一个节点设置为Primary，在要设置为Primary的节点上执行如下两条命令均可：
drbdadm --
--overwrite-data-of-peer primary data
主节点查看同步状态：
[root@app1
drbd.d]# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
GIT-hash:
1d360bde0e095d495786eaeb2a1ac76888e4db96 build by root@node1.magedu.com,
2015-01-02 12:06:20
0: cs:SyncSource ro:Primary/Secondary
ds:UpToDate/Inconsistent C r-----
ns:1229428 nr:0 dw:0 dr:1230100 al:0
bm:0 lo:0 pe:2 ua:0 ap:0 ep:1 wo:d oos:19735828
[>...................] sync\'ed: 5.9% (19272/20472)M
finish:
0:27:58 speed: 11,744 (11,808) K/sec
[root@app1 drbd.d]#
8. 创建文件系统
文件系统的挂载只能在Primary节点进行，只有在设置了主节点后才能对drbd设备进行格式化, 格式化与手动挂载测试。
[root@app1 ~]# mkfs.ext4 /dev/drbd0
[root@app1 ~]# mount /dev/drbd0
/data
三、安装配置NFS
1. app1,app2节点配置nfs
# vi /etc/exports
/data 192.168.0.0/24(rw,no_root_squash)
2. app1,app2节点配置nfs
# service rpcbind start
# service nfs start
# chkconfig rpcbind on
# chkconfig nfs on
四、corosync+pacemaker
1. app1,app2配置安装corosync pacemaker
# yum install corosync pacemaker -y
2. app1,app2安装crmsh
RHEL自6.4起不再提供集群的命令行配置工具crmsh，要实现对集群资源管理，还需要独立安装crmsh。
crmsh的rpm安装可从如下地址下载：http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/
[root@app1 crm]# yum install python-dateutil -y
说明：python-pssh、pssh依懒于python-dateutil包
[root@app1 crm]# rpm -ivh pssh-2.3.1-4.2.x86_64.rpm
python-pssh-2.3.1-4.2.x86_64.rpm crmsh-2.1-1.6.x86_64.rpm
warning:
pssh-2.3.1-4.2.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 17280ddf: NOKEY
Preparing... ###########################################
[100%]
1:python-pssh
########################################### [ 33%]
2:pssh ########################################### [ 67%]
3:crmsh ###########################################
[100%]
[root@app1 crm]#
[root@app1 crm]#
3. 创建corosync配置文件，app1,app2一样。
cd /etc/corosync/
cp corosync.conf.example corosync.conf
vi /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual
page
compatibility: whitetank
totem {
version: 2
secauth: on
threads: 0
interface {
ringnumber: 0
bindnetaddr: 10.10.10.0
mcastaddr: 226.94.8.8
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: no
logfile:
/var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
service {
ver: 1
name:
pacemaker
}
aisexec {
user: root
group:
root
}
4. 创建认证文件，app1,app2一样
各节点之间通信需要安全认证，需要安全密钥，生成后会自动保存至当前目录下，命名为authkey,权限为400。
[root@app1 corosync]# corosync-keygen
Corosync Cluster Engine
Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Press keys on your
keyboard to generate entropy (bits = 128).
Press keys on your keyboard to
generate entropy (bits = 192).
Press keys on your keyboard to generate
entropy (bits = 256).
Press keys on your keyboard to generate entropy (bits
= 320).
Press keys on your keyboard to generate entropy (bits = 384).
Press keys on your keyboard to generate entropy (bits = 448).
Press keys
on your keyboard to generate entropy (bits = 512).
Press keys on your
keyboard to generate entropy (bits = 576).
Press keys on your keyboard to
generate entropy (bits = 640).
Press keys on your keyboard to generate
entropy (bits = 704).
Press keys on your keyboard to generate entropy (bits
= 768).
Press keys on your keyboard to generate entropy (bits = 832).
Press keys on your keyboard to generate entropy (bits = 896).
Press keys
on your keyboard to generate entropy (bits = 960).
Writing corosync key to
/etc/corosync/authkey.
[root@app1 corosync]#
5. 将刚才配置的两个文件同步至app2
# scp authkeys corosync.conf root@app2:/etc/corosync/
6. 启动corosync\\pacemaker服务，测试能否正常提供服务
节点1:
[root@app1 ~]# service corosync start
Starting Corosync
Cluster Engine (corosync): [OK]
[root@app1 ~]# service pacemaker start
Starting Pacemaker Cluster
Manager [OK]
配置服务开机自启动：
chkconfig corosync on
chkconfig pacemaker on
节点2:
[root@app2 ~]# service corosync start
Starting Corosync
Cluster Engine (corosync): [OK]
[root@app1 ~]# service pacemaker start
Starting Pacemaker Cluster
Manager [OK]
配置服务开机自启动：
chkconfig corosync on
chkconfig pacemaker on
7. 测试corosync,pacemaker,crmsh安装情况
(1) 查看节点情况
[root@app1 ~]# crm status
Last updated: Tue Jan 26 13:13:19 2016
Last
change: Mon Jan 25 17:46:04 2016 via cibadmin on app1
Stack: classic openais
(with plugin)
Current DC: app1 - partition with quorum
Version:
1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
0 Resources
configured
Online: [ app1 app2 ]
(2) 查看端口启动情况
# netstat -tunlp
Active Internet connections (only servers)
Proto
Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
udp 0 0
10.10.10.25:5404 0.0.0.0:*
2828/corosync
udp 0 0 10.10.10.25:5405
0.0.0.0:* 2828/corosync
udp
0 0 226.94.8.8:5405 0.0.0.0:*
2828/corosync
(3) 查看日志
[root@app1 corosync]# tail -f /var/log/cluster/corosync.log
可以查看日志中关键信息：
Jan 23 16:09:30 corosync [MAIN ] Corosync Cluster Engine
(\'1.4.7\'): started and ready to provide service.
Jan 23 16:09:30 corosync
[MAIN ] Successfully read main configuration file
\'/etc/corosync/corosync.conf\'.
....
Jan 23 16:09:30 corosync [TOTEM ]
Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jan 23 16:09:31 corosync [TOTEM ] The network interface [10.10.10.24] is now
up.
Jan 23 16:09:31 corosync [TOTEM ] A processor joined or left the
membership and a new membership was formed.
Jan 23 16:09:48 corosync [TOTEM
] A processor joined or left the membership and a new membership was formed.
[root@app1 corosync]#
五、配置pacemaker
1. 基本配置
corosync默认启用了stonith功能，而我们要配置的集群并没有stonith设备，因此在配置集群的全局属性时要对其禁用。
# crm
crm(live)# configure ##进入配置模式
crm(live)configure# property stonith-enabled=false ##禁用stonith设备
crm(live)configure# property no-quorum-policy=ignore ##不具备法定票数时采取的动作
crm(live)configure# rsc_defaults resource-stickiness=100
##设置默认的资源黏性，只对当前节点有效。
crm(live)configure#
verify ##校验
crm(live)configure#
commit ##校验没有错误再提交
crm(live)configure#
show ##查看当前配置
node app1
node app2
property cib-bootstrap-options: \\
dc-version=1.1.11-97629de \\
cluster-infrastructure="classic openais (with plugin)" \\
expected-quorum-votes=2 \\
stonith-enabled=false \\
default-resource-stickiness=100 \\
no-quorum-policy=ignore
或:
# crm configure property stonith-enabled=false
# crm configure property
no-quorum-policy=ignore
# crm configure property
default-resource-stickiness=100
2. 资源配置
#命令使用经验说明：verify报错的，可以直接退出，也可以采用edit编辑，修改正确为止。
# crm configure edit
可以直接编辑配置文件
(1) 添加VIP
不要单个资源提交，等所有资源及约束一起建立之后提交。
crm(live)configure# primitive vip
ocf:heartbeat:IPaddr params ip=192.168.0.26 cidr_netmask=24 nic=eth0:1 op
monitor interval=30s timeout=20s on-fail=restart
crm(live)configure#
verify #验证一下参数是否正确
说明:
primitive :定义资源命令
myip
:资源ID名，可自行定义
ocf:heartbeat:IPaddr :资源代理(RA)
params
ip=192.168.0.26 :定义VIP
op monitor :监控该资源
interval=30s :间隔时间
timeout=20s :超时时间
on-fail=restart :如服务非正常关闭，让其重启，如重启不了，再转移至其他节点
(2) 添加drdb服务
crm(live)configure# primitive mydrbd ocf:linbit:drbd params
drbd_resource=data op monitor role=Master interval=20 timeout=30 op monitor
role=Slave interval=30 timeout=30 op start timeout=240 op stop timeout=100
crm(live)configure# verify
把drbd设为主从资源：
crm(live)configure# ms ms_mydrbd mydrbd meta master-max=1 master-node-max=1
clone-max=2 clone-node-max=1 notify=true
crm(live)configure# verify
(3) 文件系统挂载服务：
crm(live)configure# primitive mystore ocf:heartbeat:Filesystem params
device=/dev/drbd0 directory=/data fstype=ext4 op start timeout=60s op stop
timeout=60s op monitor interval=30s timeout=40s on-fail=restart
crm(live)configure# verify
(3) 创建约束,很关键，VIP，DRBD, 目录挂载均在一台节点上，而且VIP，目录挂载均依懒于主DRBD.
创建组资源，vip与mystore一起。
crm(live)configure# group
g_service vip mystore
crm(live)configure# verify
创建位置约束，组资源的启动依懒于drbd主节点
crm(live)configure# colocation c_g_service inf:
g_service ms_mydrbd:Master
创建位置约整，mystore存储挂载依赖于drbd主节点
crm(live)configure# colocation mystore_with_drbd_master inf: mystore
ms_mydrbd:Master
启动顺序依懒，drbd启动后，创建g_service组资源
crm(live)configure# order
o_g_service inf: ms_mydrbd:promote g_service:start
crm(live)configure#
verify
crm(live)configure# commit
3. 配置完成后，查看状态
[root@app1 ~]# crm status
Last updated: Mon Jan 25 22:24:55 2016
Last
change: Mon Jan 25 22:24:46 2016
Stack: classic openais (with plugin)
Current DC: app2 - partition with quorum
Version: 1.1.11-97629de
2
Nodes configured, 2 expected votes
4 Resources configured
Online: [ app1 app2 ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ app1 ]
Slaves: [ app2 ]
Resource Group: g_service
vip
(ocf::heartbeat:IPaddr): Started app1
mystore
(ocf::heartbeat:Filesystem): Started app1
[root@app1 ~]#
#说明：切换测试时有时会出现警告提示，影响真实状态查看，可以采用如下方式清除，提示哪个资源报警就清哪个，清理后，再次crm status查看状态显示正常。
Failed actions:
mystore_stop_0 on app1 \'unknown error\'
(1): call=97, status=complete, last-rc-change=\'Tue Jan 26 14:39:21 2016\',
queued=6390ms, exec=0ms
[root@app1 ~]# crm resource cleanup mystore
Cleaning up mystore on app1
Cleaning up mystore on app2
Waiting for 2 replies from the CRMd.. OK
[root@app1 ~]#
(1) 查看DRBD挂载目录
[root@app2 ~]# df -h
Filesystem Size Used Avail Use% Mounted
on
/dev/mapper/vg_app2-lv_root
35G 3.7G 30G
11% /
tmpfs 497M 45M 452M 10% /dev/shm
/dev/sda1 477M 34M 418M 8% /boot
192.168.1.26:/data
20G 44M 19G 1% /mnt
/dev/drbd0 20G 44M 19G 1% /data
[root@app2 ~]#
(2) 查看DRBD主备情况
[root@app2 ~]# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by
root@node1.magedu.com, 2015-01-02 12:06:20
0: cs:Connected
ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:20484 nr:336
dw:468 dr:21757 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
[root@app1 ~]# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by
root@node1.magedu.com, 2015-01-02 12:06:20
0: cs:Connected
ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:20484
dw:20484 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
(3) NFS客户端挂载读写正常
[root@vm15 ~]# df -h
Filesystem Size Used Avail Use% Mounted
on
/dev/sda3 21G 4.6G 15G 24% /
/dev/sda1
99M 23M 72M 25% /boot
tmpfs 7.4G 0 7.4G 0%
/dev/shm
/dev/mapper/vg-data 79G 71G 4.2G 95% /data
192.168.0.26:/data/ 5.0G 138M 4.6G 3% /mnt
[root@vm15 ~]#
[root@vm15 ~]#
[root@vm15 ~]# cd /mnt
[root@vm15 mnt]# ls
abc.txt lost+found
[root@vm15 mnt]# cp abc.txt a.txt
[root@vm15
mnt]#
[root@vm15 mnt]#
[root@vm15 mnt]# ls
a.txt abc.txt
lost+found
[root@vm15 mnt]#
[root@vm15 mnt]#
[root@vm15 mnt]#
4. 关机节点1测试
(1) 关闭app1节点，资源全都在节点2启动
[root@app2 ~]# crm status
Last updated: Tue Jan 26 13:31:54 2016
Last
change: Tue Jan 26 13:30:21 2016 via cibadmin on app1
Stack: classic openais
(with plugin)
Current DC: app2 - partition with quorum
Version:
1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
4 Resources
configured
Online: [ app2 ]
OFFLINE: [ app1 ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ app2 ]
Stopped: [ app1 ]
Resource Group: g_service
vip
(ocf::heartbeat:IPaddr): Started app2
mystore
(ocf::heartbeat:Filesystem): Started app2
[root@app2 ~]#
(2) 磁盘目录挂载成功
[root@app2 ~]# df -h
Filesystem Size
Used Avail Use% Mounted on
/dev/mapper/vg_app2-lv_root 36G 3.7G 30G
11% /
tmpfs 1004M 44M 960M 5% /dev/shm
/dev/sda1 485M 39M 421M 9% /boot
/dev/drbd0 5.0G 138M 4.6G 3% /data
[root@app2 ~]#
(3) DRBD也切换成了主节点：
[root@app2 ~]# cat /proc/drbd
version: 8.4.3
(api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515
build by gardner@, 2013-11-29 12:28:00
0: cs:WFConnection ro:Primary/Unknown
ds:UpToDate/DUnknown C r-----
ns:0 nr:144 dw:148 dr:689 al:1 bm:0 lo:0
pe:0 ua:0 ap:0 ep:1 wo:d oos:4
[root@app2 ~]#
节点1启动后，可以直接加入，资源也无需要再次切换。
5. 节点切换测试
# crm node standby app2 #app2离线
查看资源，节点资源直接切换到app1上面，还是重启效果好。
[root@app1 ~]# crm status
Last updated:
Tue Jan 26 14:30:05 2016
Last change: Tue Jan 26 14:29:59 2016 via
crm_attribute on app2
Stack: classic openais (with plugin)
Current DC:
app2 - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes
configured, 2 expected votes
4 Resources configured
Node app2: standby
Online: [ app1 ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ app1 ]
Stopped: [ app2 ]
Resource Group: g_service
vip
(ocf::heartbeat:IPaddr): Started app1
mystore
(ocf::heartbeat:Filesystem): Started app1
[root@app1 ~]#
6. 配置stonith,之前配置是关闭的，这里补充一下主要测试功能，实现环境中例如IBM x系列服务器可以采用ipmi等stonith设备配置。
本文采用VMware ESXi5.1虚拟机，stonith也是采用VMware
ESXi的fence设备fence_vmware_soap
注：在测试corosync+pacemaker过程中出现无法快速reboot/shutdown.stonith对一些服务器无法重启时配置该操作很有用。
需要在app1,app2安装fence-agents安装包。
# yum install fence-agents
安装之后位置以及stonith测试功能
[root@app1 ~]# /usr/sbin/fence_vmware_soap -a
192.168.0.61 -z -l root -p 876543 -o list
...
...
DRBD_HEARTBEAT_APP1,564d09c3-e8ee-9a01-e5f4-f1b11f03c810
DRBD_HEARTBEAT_APP2,564dddb8-f4bf-40e6-dbad-9b97b97d3d25
...
...
例如：重启虚拟机：
[root@app1 ~]# /usr/sbin/fence_vmware_soap -a 192.168.0.61 -z -l
root -p 876543 -n DRBD_HEARTBEAT_APP2 -o reboot
[root@app1 ~]# crm
crm(live)# configure
crm(live)configure#
primitive vm-fence-app1 stonith:fence_vmware_soap params ipaddr=192.168.0.61
login=root passwd=876543 port=app1 ssl="1" pcmk_host_list="DRBD_HEARTBEAT_APP1"
retry_on="10" shell_timeout="120" login_timeout="120" action="reboot" op start
interval="0" timeout="120"
crm(live)configure# primitive vm-fence-app2
stonith:fence_vmware_soap params ipaddr=192.168.0.61 login=root passwd=876543
port=app2 ssl="1" pcmk_host_list="DRBD_HEARTBEAT_APP2" retry_on="10"
shell_timeout="120" login_timeout="120" action="reboot" op start interval="0"
timeout="120"
crm(live)configure# location l-vm-fence-app1 vm-fence-app1
-inf: app1
crm(live)configure# location l-vm-fence-app2 vm-fence-app2 -inf:
app2
crm(live)configure# property stonith-enabled=true
crm(live)configure#
verify
crm(live)configure# commit
[root@app1 ~]# crm status
Last updated: Tue Jan 26 16:50:53 2016
Last
change: Tue Jan 26 16:50:27 2016 via crmd on app2
Stack: classic openais
(with plugin)
Current DC: app2 - partition with quorum
Version:
1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
6 Resources
configured
Online: [ app1 app2 ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ app2 ]
Slaves: [ app1 ]
Resource Group: g_service
vip
(ocf::heartbeat:IPaddr): Started app2
mystore
(ocf::heartbeat:Filesystem): Started app2
vm-fence-app1
(stonith:fence_vmware_soap): Started app2
vm-fence-app2
(stonith:fence_vmware_soap): Started app1
查看整个配置文件：
[root@app1 ~]# crm
crm(live)# configure
crm(live)configure# show
xml
<?xml version="1.0" ?>
<cib num_updates="4" dc-uuid="app2"
update-origin="app2" crm_feature_set="3.0.7" validate-with="pacemaker-1.2"
update-client="crmd" epoch="91" admin_epoch="0" cib-last-written="Tue Jan 26
16:50:27 2016" have-quorum="1">
<configuration>
<crm_config>
<cluster_property_set
id="cib-bootstrap-options">
<nvpair
id="cib-bootstrap-options-dc-version" name="dc-version"
value="1.1.10-14.el6-368c726"/>
<nvpair
id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure"
value="classic openais (with plugin)"/>
<nvpair
id="cib-bootstrap-options-expected-quorum-votes" name="expected-quorum-votes"
value="2"/>
<nvpair name="stonith-enabled" value="false"
id="cib-bootstrap-options-stonith-enabled"/>
<nvpair
name="no-quorum-policy" value="ignore"
id="cib-bootstrap-options-no-quorum-policy"/>
<nvpair
name="default-resource-stickiness" value="100"
id="cib-bootstrap-options-default-resource-stickiness"/>
<nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh"
value="1453798227"/>
</cluster_property_set>
</crm_config>
<nodes>
<node id="app2"
uname="app2">
<instance_attributes
id="nodes-app2">
<nvpair id="nodes-app2-standby"
name="standby" value="off"/>
</instance_attributes>
</node>
<node id="app1" uname="app1">
<instance_attributes id="nodes-app1">
<nvpair
id="nodes-app1-standby" name="standby" value="off"/>
</instance_attributes>
</node>
</nodes>
<resources>
<primitive id="vm-fence-app1" class="stonith"
type="fence_vmware_soap">
<instance_attributes
id="vm-fence-app1-instance_attributes">
<nvpair name="ipaddr"
value="192.168.0.61"
id="vm-fence-app1-instance_attributes-ipaddr"/>
<nvpair
name="login" value="root"
id="vm-fence-app1-instance_attributes-login"/>
<nvpair
name="passwd" value="xjj876543"
id="vm-fence-app1-instance_attributes-passwd"/>
<nvpair
name="port" value="app1"
id="vm-fence-app1-instance_attributes-port"/>
<nvpair
name="ssl" value="1"
id="vm-fence-app1-instance_attributes-ssl"/>
<nvpair
name="pcmk_host_list" value="DRBD_HEARTBEAT_APP1"
id="vm-fence-app1-instance_attributes-pcmk_host_list"/>
<nvpair name="retry_on" value="10"
id="vm-fence-app1-instance_attributes-retry_on"/>
<nvpair
name="shell_timeout" value="120"
id="vm-fence-app1-instance_attributes-shell_timeout"/>
<nvpair name="login_timeout" value="120"
id="vm-fence-app1-instance_attributes-login_timeout"/>
<nvpair name="action" value="reboot"
id="vm-fence-app1-instance_attributes-action"/>
</instance_attributes>
<operations>
<op
name="start" interval="0" timeout="120"
id="vm-fence-app1-start-0"/>
</operations>
</primitive>
<primitive id="vm-fence-app2" class="stonith"
type="fence_vmware_soap">
<instance_attributes
id="vm-fence-app2-instance_attributes">
<nvpair name="ipaddr"
value="192.168.0.61"
id="vm-fence-app2-instance_attributes-ipaddr"/>
<nvpair
name="login" value="root"
id="vm-fence-app2-instance_attributes-login"/>
<nvpair
name="passwd" value="xjj876543"
id="vm-fence-app2-instance_attributes-passwd"/>
<nvpair
name="port" value="app2"
id="vm-fence-app2-instance_attributes-port"/>
<nvpair
name="ssl" value="1"
id="vm-fence-app2-instance_attributes-ssl"/>
<nvpair
name="pcmk_host_list" value="DRBD_HEARTBEAT_APP2"
id="vm-fence-app2-instance_attributes-pcmk_host_list"/>
<nvpair name="retry_on" value="10"
id="vm-fence-app2-instance_attributes-retry_on"/>
<nvpair
name="shell_timeout" value="120"
id="vm-fence-app2-instance_attributes-shell_timeout"/>
<nvpair name="login_timeout" value="120"
id="vm-fence-app2-instance_attributes-login_timeout"/>
<nvpair name="action" value="reboot"
id="vm-fence-app2-instance_attributes-action"/>
</instance_attributes>
<operations>
<op
name="start" interval="0" timeout="120"
id="vm-fence-app2-start-0"/>
</operations>
</primitive>
<group id="g_service">
<primitive id="vip" class="ocf" provider="heartbeat"
type="IPaddr">
<instance_attributes
id="vip-instance_attributes">
<nvpair name="ip"
value="192.168.0.26" id="vip-instance_attributes-ip"/>
<nvpair name="cidr_netmask" value="24"
id="vip-instance_attributes-cidr_netmask"/>
<nvpair
name="nic" value="eth0:1" id="vip-instance_attributes-nic"/>
</instance_attributes>
<operations>
<op name="monitor" interval="30s" timeout="20s" on-fail="restart"
id="vip-monitor-30s"/>
</operations>
</primitive>
<primitive id="mystore" class="ocf"
provider="heartbeat" type="Filesystem">
<instance_attributes
id="mystore-instance_attributes">
<nvpair name="device"
value="/dev/drbd0" id="mystore-instance_attributes-device"/>
<nvpair name="directory" value="/data"
id="mystore-instance_attributes-directory"/>
<nvpair
name="fstype" value="ext4"
id="mystore-instance_attributes-fstype"/>
</instance_attributes>
<operations>
<op name="start" timeout="60s" interval="0"
id="mystore-start-0"/>
<op name="stop" timeout="60s"
interval="0" id="mystore-stop-0"/>
<op name="monitor"
interval="30s" timeout="40s" on-fail="restart"
id="mystore-monitor-30s"/>
</operations>
</primitive>
</group>
<master
id="ms_mydrbd">
<meta_attributes
id="ms_mydrbd-meta_attributes">
<nvpair name="master-max"
value="1" id="ms_mydrbd-meta_attributes-master-max"/>
<nvpair
name="master-node-max" value="1"
id="ms_mydrbd-meta_attributes-master-node-max"/>
<nvpair
name="clone-max" value="2"
id="ms_mydrbd-meta_attributes-clone-max"/>
<nvpair
name="clone-node-max" value="1"
id="ms_mydrbd-meta_attributes-clone-node-max"/>
<nvpair
name="notify" value="true" id="ms_mydrbd-meta_attributes-notify"/>
</meta_attributes>
<primitive id="mydrbd" class="ocf"
provider="linbit" type="drbd">
<instance_attributes
id="mydrbd-instance_attributes">
<nvpair
name="drbd_resource" value="data"
id="mydrbd-instance_attributes-drbd_resource"/>
</instance_attributes>
<operations>
<op name="monitor" role="Master" interval="20" timeout="30"
id="mydrbd-monitor-20"/>
<op name="monitor" role="Slave"
interval="30" timeout="30" id="mydrbd-monitor-30"/>
<op
name="start" timeout="240" interval="0" id="mydrbd-start-0"/>
<op name="stop" timeout="100" interval="0"
id="mydrbd-stop-0"/>
</operations>
</primitive>
</master>
</resources>
<constraints>
<rsc_colocation id="c_g_service"
score="INFINITY" rsc="g_service" with-rsc="ms_mydrbd"
with-rsc-role="Master"/>
<rsc_colocation
id="mystore_with_drbd_master" score="INFINITY" rsc="mystore"
with-rsc="ms_mydrbd" with-rsc-role="Master"/>
<rsc_order
id="o_g_service" score="INFINITY" first="ms_mydrbd" first-action="promote"
then="g_service" then-action="start"/>
<rsc_location
id="l-vm-fence-app1" rsc="vm-fence-app1" score="-INFINITY"
node="app1"/>
<rsc_location id="l-vm-fence-app2"
rsc="vm-fence-app2" score="-INFINITY" node="app2"/>
</constraints>
</configuration>
</cib>
清空资源，重新配置操作方法：
[root@app2 ~]# crm status
Last updated: Wed Jan 27 10:39:24 2016
Last change: Tue Jan 26 16:50:27 2016 via crmd on app2
Stack: classic openais (with plugin)
Current DC: app2 - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
6 Resources configured
Online: [ app1 app2 ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ app2 ]
Slaves: [ app1 ]
Resource Group: g_service
vip (ocf::heartbeat:IPaddr): Started app2
mystore (ocf::heartbeat:Filesystem): Started app2
vm-fence-app1 (stonith:fence_vmware_soap): Started app2
vm-fence-app2 (stonith:fence_vmware_soap): Started app1
[root@app2 ~]#
先依次关闭资源 ：
[root@app2 ~]#
[root@app2 ~]# crm resource stop vm-fence-app2
[root@app2 ~]# crm resource stop vm-fence-app1
[root@app2 ~]# crm resource stop mystore
[root@app2 ~]# crm resource stop vip
[root@app2 ~]# crm resource stop ms_mydrbd
[root@app2 ~]# crm status
Last updated: Wed Jan 27 10:40:28 2016
Last change: Wed Jan 27 10:40:23 2016 via cibadmin on app2
Stack: classic openais (with plugin)
Current DC: app2 - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
6 Resources configured
Online: [ app1 app2 ]
[root@app2 ~]#
再清空配置：
[root@app2 ~]# crm configure erase
INFO: resource references in colocation:c_g_service updated
INFO: resource references in colocation:mystore_with_drbd_master updated
INFO: resource references in order:o_g_service updated
[root@app2 ~]#
[root@app2 ~]#
[root@app2 ~]# crm status
Last updated: Wed Jan 27 10:40:58 2016
Last change: Wed Jan 27 10:40:52 2016 via crmd on app2
Stack: classic openais (with plugin)
Current DC: app2 - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
0 Resources configured
Online: [ app1 app2 ]
[root@app2 ~]#
[root@app2 ~]#
就可以再次重新配置了。
8. 配置小结：
之前多次未成功配置的成功主要在于资源的排列与定位启动上面，造成切换，启动均不成功，这个也是corosync+pacemaker的配置要理解的重点, DRBD+可以实现很多种组合，本文仅提供技术实现参考。
以上是关于Corosync+Pacemaker+DRBD+NFS高可用实例配置的主要内容，如果未能解决你的问题，请参考以下文章
Corosync+Pacemaker+DRBD+NFS高可用实例配置
drbd+corosync+pacemaker构建高可用MySQL集群
基于corosync和pacemaker+drbd实现mfs高可用
Corosync+pacemaker+DRBD+mysql（mariadb）实现高可用（ha）的mysql集群（centos7）