CentOS 6.5上部署drbd
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了CentOS 6.5上部署drbd相关的知识，希望对你有一定的参考价值。
下面开始在CentOS 6.5上安装并配置drbd，环境的话继续使用之前安装完heartbeat的两台主机，同时也是为后面实现heartbeat存储和数据库高可用做准备，所以如果需要单独操作，请按照之前安装heartbeat的准备工作进行配置，然后根据下面的步骤进行也可以。
1、环境准备
这里我以其中一台为例，介绍准备工作包括哪些点：
#检查防火墙是否关闭（或者开启7788端口）
[[email protected] ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
#检查SELinux是否禁用
[[email protected] ~]# getenforce
Disabled
#检查是否已经添加了时间同步定时任务
[[email protected] ~]# crontab -l
0 * * * * /usr/sbin/ntpdate 210.72.145.44 64.147.116.229 time.nist.gov
#检查hosts文件中是否有两个节点的记录
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.49.133 heartbeat01.contoso.com heartbeat01
192.168.49.134 heartbeat02.contoso.com heartbeat02
#检查是否更换国内yum源
[[email protected] ~]# ll /etc/yum.repos.d/
total 32
-rw-r--r-- 1 root root 2006 Sep 18 2014 CentOS-Base.repo
-rw-r--r--. 1 root root 1926 Nov 27 2013 CentOS-Base.repo.bak_2016-07-28
-rw-r--r--. 1 root root 638 Nov 27 2013 CentOS-Debuginfo.repo.bak_2016-07-28
-rw-r--r--. 1 root root 630 Nov 27 2013 CentOS-Media.repo.bak_2016-07-28
-rw-r--r--. 1 root root 3664 Nov 27 2013 CentOS-Vault.repo.bak_2016-07-28
-rw-r--r--. 1 root root 120 Jul 25 18:24 cobbler-config.repo.bak_2016-07-28
另外，跟heartbeat不同的是，drbd需要两块硬盘，所以这里我们还需要添加一块硬盘，我是在虚拟机操作的，直接添加一块2G的硬盘。
[[email protected] ~]# fdisk -l
Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000693e3
Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 2611 20458496 8e Linux LVM
Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
2、安装drbd
drbd的安装有多种方式，可以源码编译安装，也可以使用rpm包进行安装，当然在centos上也可以使用yum安装，这里我采用yum安装的方式。
rpm -Uvhhttp://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm yum update -y yum -y install kernel* yum -y install drbd83-utils kmod-drbd83 modprobe drbd
因为更新了内核，所以需要重启才能加载drbd模块，所以使用yum安装完kernel之后需要重启两台服务器，如果不重启则会出现以下问题：
[[email protected] ~]# uname -r
2.6.32-431.el6.x86_64
[[email protected] ~]# lsmod|grep drbd
[[email protected] ~]# modprobe drbd
FATAL: Module drbd not found.
[[email protected] ~]# echo $?
1
重启之后重试：
[[email protected] ~]# uname -r
2.6.32-642.4.2.el6.x86_64
[[email protected] ~]# lsmod |grep drbd
drbd 332493 0
[[email protected] ~]# /sbin/modprobe drbd
[[email protected] ~]# echo $?
0
3、准备drbd设备
先对新添加的磁盘进行分区操作，这里仍然以其中一台为例。
[[email protected] ~]# fdisk -cu /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x198bc436.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won‘t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x198bc436
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-4194303, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4194303, default 4194303):
Using default value 4194303
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
4、编辑drbd配置文件
根据drbd的官方文档，drbd的配置文件是/etc/drbd.conf，但是这个文件中只有简单的2行内容，它引用了/etc/drbd.d/下的global_common.conf和*.res文件，这点和nginx、apache的配置文件类似，我们可以把多个resource相同的定义放到global中去，然后添加不同的res文件，从而更有效和方便的管理drbd的配置文件。
下面是/etc/drbd.conf的默认配置：
[[email protected] ~]# egrep -v "#|^$" /etc/drbd.conf
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
下面是/etc/drbd.d/global_common.conf的默认配置：
[[email protected] ~]# egrep -v "#|^$" /etc/drbd.d/global_common.conf
global {
usage-count yes;
}
common {
protocol C;
handlers {
}
startup {
}
disk {
}
net {
}
syncer {
}
}
下面是我新建的一个配置文件test.res的内容：
resource test
{
startup {
wfc-timeout 30;
outdated-wfc-timeout 20;
degr-wfc-timeout 30;
}
net {
cram-hmac-alg sha1;
shared-secret sync_disk;
}
syncer {
rate 10M;
al-extents 257;
on-no-data-accessible io-error;
}
on heartbeat01.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.49.133:7788;
flexible-meta-disk internal;
}
on heartbeat02.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.49.134:7788;
meta-disk internal;
}
}
然后可以将此配置文件使用scp拷贝到节点2上去，两边的配置文件需要保持一致。
[[email protected] drbd.d]# scp test.res heartbeat02:/etc/drbd.d/
The authenticity of host ‘heartbeat02 (192.168.49.134)‘ can‘t be established.
RSA key fingerprint is f9:ce:14:5d:cd:bb:3c:b4:0d:0b:fc:21:3a:92:43:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘heartbeat02,192.168.49.134‘ (RSA) to the list of known hosts.
[email protected]‘s password:
test.res 100% 444 0.4KB/s 00:00
5、启动drbd
drbdadm create-md clusterdb #初始化metadata数据存储 service drbd start #启动drbd服务
上面的操作需要到两台server上运行，这里以其中一台为例演示结果：
[[email protected] drbd.d]# drbdadm create-md test
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
[[email protected] drbd.d]# service drbd start
Starting DRBD resources: [ d(test) s(test) n(test) ]..........
***************************************************************
DRBD‘s startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
reboot the timeout is 30 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
expire after 30 seconds. [wfc-timeout]
(These values are for resource ‘test‘; 0 sec -> wait forever)
To abort waiting enter ‘yes‘ [ 11]:
.
这里给出很多信息，大多是因为对端节点并没有启动，所以配置文件中的等待时间生效，但是在节点1启动后，我们再启动节点2时的信息就没有这么多了，因为节点1已经启动，所以无须等待。
[[email protected] drbd.d]# service drbd start
Starting DRBD resources: [ d(test) s(test) n(test) ].
此时，两个节点上都是Secondary的状态，所以我们需要手动的指定哪个节点为primary。
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2096028
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2096028
在要作为主节点的server上执行如下命令：
drbdadm -- --overwrite-data-of-peer primary all（是两个--，不是打错）
这里我在节点1上执行该命令。
[[email protected] drbd.d]# drbdadm -- --overwrite-data-of-peer primary all
再次看一下两个节点的状态：
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:217088 nr:0 dw:0 dr:217752 al:0 bm:13 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1878940
[=>..................] sync‘ed: 10.6% (1878940/2096028)K
finish: 0:03:01 speed: 10,344 (10,336) K/sec
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
ns:0 nr:485248 dw:485248 dr:0 al:0 bm:29 lo:1 pe:1 ua:0 ap:0 ep:1 wo:f oos:1610780
[===>................] sync‘ed: 23.3% (1610780/2096028)K
finish: 0:02:34 speed: 10,424 (10,324) want: 10,240 K/sec
可以看到现在heartbeat01是Primary的状态，而heartbeat02是Secondary的状态，而且已经开始从主上面同步数据到从节点上了。
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:2096028 nr:0 dw:0 dr:2096692 al:0 bm:128 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
同步完成，两个节点都是UpToDate的状态。
6、创建文件系统并挂载
下面在主节点上为drbd设置创建文件系统
[[email protected] drbd.d]# mkfs.ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131072 inodes, 524007 blocks
26200 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
然后创建挂载点，将drbd设备挂载到挂载点从而进行访问。
注意：从节点不需要为drbd创建文件系统，也无须挂载，在主节点挂掉之后，可以直接转换角色，然后将文件系统挂载到挂载点进行使用。
[[email protected] drbd.d]# mkdir /data
[[email protected] drbd.d]# mount /dev/drbd0 /data
[[email protected] drbd.d]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
19G 1.6G 16G 10% /
tmpfs 238M 0 238M 0% /dev/shm
/dev/sda1 477M 83M 369M 19% /boot
/dev/drbd0 2.0G 3.0M 1.9G 1% /data
7、切换Primary和Secondary节点测试
这里，我创建了99个txt文件在/data目录下，等下看切换了主从节点之后，能否在从节点上看到这些文件。
下面在主节点（heartbeat01）上操作：
首先要取消挂载：
[[email protected] drbd.d]# umount /data
[[email protected] drbd.d]# mount -n
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
然后执行下面的命令切换角色：
[[email protected] drbd.d]# drbdadm secondary test
此时看到主节点已经是Secondary的角色了，而从节点暂时没有处理，也还是Secondary的角色。
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:2162432 nr:0 dw:66404 dr:2097405 al:23 bm:128 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
下面到从节点（heartbeat02）上操作：
[[email protected] drbd.d]# drbdadm primary test
或者
[[email protected] drbd.d]# drbdadm -- --overwrite-data-of-peer primary all
[[email protected] drbd.d]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:2162432 dw:2162432 dr:664 al:0 bm:128 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
此时，从节点（heartbeat02）已经是primary状态了。
[[email protected] drbd.d]# mkdir /data
[[email protected] drbd.d]# mount /dev/drbd0 /data
创建目录并挂载，已经在heartbeat02的/data目录看到之前创建的99个txt文件了，说明drbd已经将数据同步成功了。
本文出自 “IT小二郎” 博客，请务必保留此出处http://jerry12356.blog.51cto.com/4308715/1856250
以上是关于CentOS 6.5上部署drbd的主要内容，如果未能解决你的问题，请参考以下文章