Linux集群存储配置

Posted 2020-08-09

RH436_EX集群

介绍Linux环境下集群架构，Linux开源集群软件的安装及配置使用，软件与软件之间的组合。实现高可用集群，负载均衡集群;负载均衡、高可用集群与存储集群间的多集群混合架构使用。

本次试验我使用了多种安装包：pcs为集群的基本架构，fence为警报装置，targetcli和iscsi实现远程磁盘共享，dlm和lvm2.cluster为集群中解决底层共享的逻辑卷，gfs-utiils中gfs2与clvm可扩展性集群共享存储。
实验环境中多台PC，真实机 ：foundation0；低层存储：noded；三个节点：nodea、nodeb、nodec。

首先我们讲所用PC端的防火墙关掉
在/etc/selinux/config或/etc/sysconfig/selinux 配置文件内第7行的 内容改成SELINUX=diabled
[[email protected] ~]# setenforce 0
[[email protected] ~]# getenforce
Permissive
并用命令修改当前的状态

安装集群架构包pcs，并将用户hacluster加入其中及修改了
hacluster的密码为redhat
[[email protected]、[email protected]、[email protected]]：
yum -y install pcs ;systemctl restart pcsd;systemctl enable pcsd;systemctl stop firewalld.service ;systemctl disable firewalld.service ;echo redhat |passwd --stdin hacluster
[[email protected]]：
[[email protected] ~]# pcs cluster auth nodea.cluster0.example.com nodeb.cluster0.example.com nodec.cluster0.example.com
Username: hacluster
Password:
nodea.cluster0.example.com: Authorized
nodeb.cluster0.example.com: Authorized
nodec.cluster0.example.com: Authorized
[[email protected]、[email protected]、[email protected]]：
systemctl restart pcsd
真机打开浏览器浏览集群架构
[[email protected] ]：
firefox https://nodea.cluster0.example.com:2224

 

1.png

 

1、用户名：hacluster
2、密码：redhat

 

2.png

 

1、集群名：cluster-1
2-4、所加入的节点：nodea.cluster0.example.com，nodeb.cluster0.example.com，nodec.cluster0.example.com

[[email protected] ]：
[[email protected] ~]# mkdir /virtio
[[email protected] ~]# cd /virtio/
[[email protected] virtio]# ls
[[email protected] virtio]# dd if=/dev/zero of=/virtio/vdisk1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 14.5206 s, 73.9 MB/s
[[email protected] virtio]# cd
[[email protected] ~]# yum -y install targetcli 安装targetcli包
[[email protected] virtio]# targetcli 进入磁盘编辑

 

3.png

 

1、在/backstores/fileio下create iscsi_store /virtio/vdisk1 创建iscsi_store名接vdisk1目录
2、在/iscsi下create iqn.2016-09.com.example.storage 创建一个广播名（用来发现）
3、在/iscsi/iqn.2016-09.com.example.storage/tpg1/acls下 create iqn.2016-09.com.example:node 创建一个访问控制列表（用来允许客户端连接的密钥 node为域）
4、在/iscsi/iqn.2016-09.com.example.storage/tpg1/luns下create /backstores/fileio/iscsi_store 标识每一个设备的ID（默认从0开始）
5、会自动生成一个IP 地址
[[email protected] virtio]# systemctl restart target ；systmctl enable target

[[email protected] 、[email protected]、[email protected]]：
yum -y install iscsi*
[[email protected] ~]# vim /etc/iscsi/initiatorname.iscsi 修改配置文件
[[email protected] ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2016-09.com.example:node
[[email protected] ~]# scp /etc/iscsi/initiatorname.iscsi 将修改好的配置文件cp到nodeb、nodec上 [email protected]:/etc/iscsi/initiatorname.iscsi
[[email protected] ~]# scp /etc/iscsi/initiatorname.iscsi [email protected]:/etc/iscsi/initiatorname.iscsi
[[email protected] ~]# systemctl restart iscsid;systemctl enable iscsid.service
ln -s ‘/usr/lib/systemd/system/iscsid.service‘ ‘/etc/systemd/system/multi-user.target.wants/iscsid.service‘

[[email protected] ~]# iscsiadm -m discovery -t sendtargets -p 192.168.1.13
192.168.1.13:3260,1 iqn.2016-09.com.example.storage
[[email protected] ~]# iscsiadm -m discovery -t sendtargets -p 192.168.2.13
192.168.2.13:3260,1 iqn.2016-09.com.example.storage
如果上面的命令不能连接noded上的磁盘，那就最先做targetcli和iscsi后做pcs。
[[email protected] ~]# iscsiadm -m node -T iqn.2016-09.com.example.storage -p 192.168.1.13 -l
Logging in to [iface: default, target: iqn.2016-09.com.example.storage, portal: 192.168.1.13,3260] (multiple)
Login to [iface: default, target: iqn.2016-09.com.example.storage, portal: 192.168.1.13,3260] successful.
[[email protected] ~]#
[[email protected] ~]# iscsiadm -m node -T iqn.2016-09.com.example.storage -p 192.168.2.13 -l
Logging in to [iface: default, target: iqn.2016-09.com.example.storage, portal: 192.168.2.13,3260] (multiple)
Login to [iface: default, target: iqn.2016-09.com.example.storage, portal: 192.168.2.13,3260] successful.
[[email protected] ~]# fdisk -l

Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000272f4

Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux

Disk /dev/sda: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes

Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
/dev/sda和/dev/sdb为远程noded上的同一个虚拟磁盘，sda、sdb是2个接口做双网卡的备份。

[[email protected] ~]# yum -y install device-mapper-multipath 做多路径及线路预警
[[email protected] ~]# modprobe md_multipath 添加multipath模块
[[email protected] ~]# multipath -l
Sep 21 21:31:19 | /etc/multipath.conf does not exist, blacklisting all devices.
Sep 21 21:31:19 | A default multipath.conf file is located at
Sep 21 21:31:19 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Sep 21 21:31:19 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
提示报错,缺少文件
[[email protected] ~]# systemctl restart multipathd.service ;systemctl enable multipathd
[[email protected] ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
[[email protected] ~]# multipath -l
Sep 21 21:35:25 | vda: No fc_host device for ‘host-1‘
Sep 21 21:35:25 | vda: No fc_host device for ‘host-1‘
Sep 21 21:35:25 | vda: No fc_remote_port device for ‘rport--1:-1-0‘
mpatha (3600140597279744f247488683521b4c2) dm-0 LIO-ORG ,iscsi-test
size=1.0G features=‘0‘ hwhandler=‘0‘ wp=rw
|-+- policy=‘service-time 0‘ prio=0 status=active
| - 2:0:0:0 sda 8:0 active undef running-+- policy=‘service-time 0‘ prio=0 status=enabled
`- 3:0:0:0 sdb 8:16 active undef running
提示上面信息就对了，原先的sda/sdb现在统称为mpatha名，不过mpatha名也可以自定义。
下面将mpatha定义为mymu
[[email protected] ~]# vim .bash_profile
[[email protected] ~]# cat .bashp
cat: .bashp: No such file or directory
[[email protected] ~]# cat .bash_profile
.bash_profile

#Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/lib/udev

export PATH
[[email protected] ~]# source .bash_profile
[[email protected] ~]# scp .bash_profile [email protected]:/root/.bash_profile
[email protected]‘s password:
.bash_profile 100% 190 0.2KB/s 00:00
[[email protected] ~]# scp .bash_profile [email protected]:/root/.bash_profile
[email protected]‘s password:
.bash_profile 100% 190 0.2KB/s 00:00
[[email protected] ~]# scsi_id -u -g /dev/sda
3600140597279744f247488683521b4c2
[[email protected] ~]# scsi_id -u -g /dev/sdb
3600140597279744f247488683521b4c2
上面的修改配置文件是便于使用scsi_id命令查看UUID的信息

[[email protected] ~]# vim /etc/multipath.conf
user_friendly_name no 开启默认命名 23行

1、wwid 3600140597279744f247488683521b4c2 填写UUID 73行
2、 alias mymu 匿名 74行
[[email protected] ~]# systemctl restart multipathd.service
[[email protected] ~]# multipath -l
Sep 21 21:57:01 | vda: No fc_host device for ‘host-1‘
Sep 21 21:57:01 | vda: No fc_host device for ‘host-1‘
Sep 21 21:57:01 | vda: No fc_remote_port device for ‘rport--1:-1-0‘
mymu (3600140597279744f247488683521b4c2) dm-0 LIO-ORG ,iscsi-test
size=1.0G features=‘0‘ hwhandler=‘0‘ wp=rw
|-+- policy=‘service-time 0‘ prio=0 status=active
| - 2:0:0:0 sda 8:0 active undef running-+- policy=‘service-time 0‘ prio=0 status=enabled
`- 3:0:0:0 sdb 8:16 active undef running

[[email protected] ~]# more /proc/partitions 查看磁盘简略信息
major minor #blocks name

253 0 10485760 vda
253 1 10484142 vda1
8 0 1048576 sda
8 16 1048576 sdb
252 0 1048576 dm-0

[[email protected] ~]# yum -y install dlm lvm2-cluster.x86_64 将本地的lvm升级成lvm2-cluster 解决集群中的逻辑卷
[[email protected] ~]# vim /etc/lvm/lvm.conf 在配置文件内567行改成locking_type = 3 ， 1为单机版，3为克隆模式 。或用命令lvmconf --enable-cluster自动改为3模式

[[email protected]]：
[[email protected] ~]# mkdir /etc/cluster
[[email protected] ~]# cd /etc/cluster/
[[email protected] cluster]# ls
[[email protected] cluster]# dd if=/dev/zero of=/etc/cluster/fence_xvm.key bs=4k count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000129868 s, 31.5 MB/s
[[email protected] cluster]# yum -y install fence-virt* 安装fence是对资源内clvm和dlm做对应
[[email protected] cluster]# fence_virtd -c
以上都回车
Interface [virbr0]: br0
回车
Replace /etc/fence_virt.conf with the above [y/N]? y
[[email protected] cluster]# ls
fence_xvm.key
[[email protected] cluster]# scp fence_xvm.key [email protected]:/etc/cluster
Warning: Permanently added ‘nodea.cluster0.example.com,172.25.0.10‘ (ECDSA) to the list of known hosts.
fence_xvm.key 100% 4096 4.0KB/s 00:00
[[email protected] cluster]# scp fence_xvm.key [email protected]:/etc/cluster
Warning: Permanently added ‘nodeb.cluster0.example.com,172.25.0.11‘ (ECDSA) to the list of known hosts.
fence_xvm.key 100% 4096 4.0KB/s 00:00
[[email protected] cluster]# scp fence_xvm.key [email protected].cluster0.example.com:/etc/cluster
Warning: Permanently added ‘nodec.cluster0.example.com,172.25.0.12‘ (ECDSA) to the list of known hosts.
fence_xvm.key 100% 4096 4.0KB/s 00:00

firefox https://nodea.cluster0.example.com:2224
add添加一个fence

 

5.png

 

1、选择fence_xvm模式
2、名字：fence-1
3、指定对应的PC：nodea,noodeb,nodec

 

6.png

 

add添加一个

 

7.png

 

1、ocf:heartbeat
2、IPaddr2模式
3、名字：vip
4、IP地址：172.25.0.110

add添加一个

 

8.png

1、ocf:pacemaker
2、controld模式
3、Clone√ 克隆模式
4、名字：cluster-dlm

 

add添加一个

 

9.png

 

1、ocf:heartbeat
2、clvm模式
3、Clone√ 克隆模式
4、名字：cluster-clvm

 

10.png

 

[[email protected]】：
[[email protected] cluster]# fdisk -l
Disk /dev/mapper/mymu: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
[[email protected] cluster]# cd /dev/mapper/
[[email protected] mapper]# ls
total 0
crw-------. 1 root root 10, 236 Sep 21 20:49 control
lrwxrwxrwx. 1 root root 7 Sep 21 21:56 mymu -> ../dm-0
[[email protected] mapper]# fdisk /dev/mapper/mymu
Device Boot Start End Blocks Id System
/dev/mapper/mymu1 8192 1851391 921600 83 Linux
[[email protected] mapper]# partprobe
[[email protected] mapper]# fdisk -l
Disk /dev/mapper/mymu: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
Disk label type: dos
Disk identifier: 0xd8583874

       Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mymu1 8192 1851391 921600 83 Linux

Disk /dev/mapper/mymu1: 943 MB, 943718400 bytes, 1843200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes

[[email protected] mapper]# pvcreate /dev/mapper/mymu1
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
Physical volume "/dev/mapper/mymu1" successfully created
[[email protected] mapper]# vgcreate vg1 /dev/mapper/mymu1
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
Clustered volume group "vg1" successfully created
[[email protected] mapper]# lvcreate -L 800M -n lv1 vg1
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
Logical volume "lv1" created.

[[email protected] ~]# lvs
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv1 vg1 -wi-a----- 800.00m

[[email protected] ~]# lvs
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv1 vg1 -wi-a----- 800.00m

[[email protected] mapper]# yum -y install gfs2-utils.x86_64
[[email protected] mapper]# mkfs.gfs2 -p lock_dlm -t cluster-1:mygfs2 -j 4 /dev/vg1/lv1
-p：类型
-t：集群名：自定义名
-j：n+1 n为节点个数，1为保留
/dev/vg1/lv1 is a symbolic link to /dev/dm-2
This will destroy any data on /dev/dm-2
Are you sure you want to proceed? [y/n]y
Device: /dev/vg1/lv1
Block size: 4096
Device size: 0.78 GB (204800 blocks)
Filesystem size: 0.78 GB (204797 blocks)
Journals: 4
Resource groups: 6
Locking protocol: "lock_dlm"
Lock table: "cluster-1:mygfs2"
UUID: e89181f7-1251-8c4b-496e-305aeff9b780

[[email protected] ~]# mkdir /zzz
[[email protected] ~]# mount /dev/vg1/lv1 /zzz/
[[email protected] ~]# cd /zzz/
[[email protected] zzz]# ls
[[email protected] zzz]# vim zlm
[[email protected] zzz]# cat zlm
haha.................

[[email protected] ~]# mkdir /zzz
[[email protected] ~]# mount /dev/vg1/lv1 /zzz/
[[email protected] zzz]# cat /zzz/zlm
haha.................

[[email protected] ~]# mkdir /zzz
[[email protected] ~]# mount /dev/vg1/lv1 /zzz/
[email protected] zzz]# cat /zzz/zlm
haha.................
[[email protected] zzz]# vim zlm
[[email protected] zzz]# cat zlm
haha

[[email protected] zzz]# cat zlm
haha

上面文件查看说明底层文件是可以同步了，如果要对文件做对应的权限控制可以天acl或磁盘配额
acl的添加权限为mount -o remount,acl /zzz
磁盘配额的添加权限为mount -o remount,quota=on /zzz

[[email protected] ~]# umount /zzz/
[[email protected] ~]# umount /zzz/
[[email protected] ~]# umount /zzz/

[[email protected] ~]# yum -y install httpd

 

11.png

1、为nodea、nodeb、nodec添加了优先级
2、当前ip落在nodea上

 

 

12.png

 

1、添加一个资源名WEB先绑定vip

 

13.png

 

1、ocf:heartbeat
2、filesystem模式
3、绑定WEB上
4、资源名：web-file
5、磁盘：/dev/vg1/lv1
6、挂在地方：/var/www/html
7、磁盘格式：gfs2
8、权限：acl

 

14.png

 

1、systemd
2、httpd模式
3、绑定WEB
4、资源名：web-httpd

 

15.png

 

[[email protected] html]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 10G 1.2G 8.9G 12% /
devtmpfs 904M 0 904M 0% /dev
tmpfs 921M 61M 861M 7% /dev/shm
tmpfs 921M 28M 893M 4% /run
tmpfs 921M 0 921M 0% /sys/fs/cgroup
/dev/mapper/vg1-lv1 800M 518M 283M 65% /var/www/html
说明/dev/mapper/vg1-lv1自动挂在再 /var/www/html上了

[[email protected] ~]# cd /var/www/html/
[[email protected] html]# ls
zlm
[[email protected] html]# cat zlm
haha
刚才测试的文件还在，说明文件已经写到noded下的底层磁盘
[[email protected] html]# vim index.html
[[email protected] html]# cat index.html
nodea

[[email protected] html]# pcs cluster standby
*****当关掉nodea后，http的服务就红了，底层的磁盘不能跑到nodeb上。随后cluster-clvm也红 了，现在还不知道什么原因，看到此篇文章知道最后失败的原因可以留言讨论。

 

17.png

 

 

16.png