Glusterfs映射成block设备

Posted 2020-11-13 missyou-shiyh

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Glusterfs映射成block设备相关的知识，希望对你有一定的参考价值。

Glusterfs分布式存储软件可以创建集群文件设备，但有时我们需要将文件设备转换为block设备。如下示例为Glusterfs + tcmu，将文件设备转换为block设备。

环境说明：

?	主机名	IP	设备
? ? ? ? 存储节点	oel7db01	192.168.56.201	在gluster上创建一个3副本的volume,然后在其中的一个存储节点上挂载这个volume，然后在这个gluster文件系统上创建5个文件，这5个文件后期用来模拟成block设备
	oel7db02	192.168.56.202
	oel7db03	192.168.56.203
计算节点	oel7db04	192.168.56.204	每个计算节点上最终会看到5个block设备，每个block设备来自于3个链路（因为有3个存储节点）
计算节点	oel7db05	192.168.56.205	每个计算节点上最终会看到5个block设备，每个block设备来自于3个链路（因为有3个存储节点）

存储节点安装软件：

yum install -y glusterfs-server

yum install -y tcmu-runner

yum install -y tcmu-runner-handler-glfs

安装完成后，存储节点可以看到如下的初始状态：

[root@oel7db01 network-scripts]# targetcli

targetcli shell version 2.1.fb46

For help on commands, type ‘help‘.

/> ls

o- / ......................................................................................................................... [...]

o- backstores .............................................................................................................. [...]

| o- block .................................................................................................. [Storage Objects: 0]

| o- fileio ................................................................................................. [Storage Objects: 0]

| o- pscsi .................................................................................................. [Storage Objects: 0]

| o- ramdisk ................................................................................................ [Storage Objects: 0]

| o- user:glfs .............................................................................................. [Storage Objects: 0]

| o- user:qcow .............................................................................................. [Storage Objects: 0]

o- iscsi ............................................................................................................ [Targets: 0]

o- loopback ......................................................................................................... [Targets: 0]

存储节点配置

fallocate -l 2G file1.img

fallocate -l 2G file2.img

fallocate -l 2G file3.img

fallocate -l 2G file4.img

fallocate -l 2G file5.img

运行targetcli命令进入targetcli中，继续运行如下命令：

/backstores/user:glfs create LUN1 2G gv1@oel7db02/file1.img

/backstores/user:glfs create LUN2 2G gv1@oel7db02/file2.img

/backstores/user:glfs create LUN3 2G gv1@oel7db02/file3.img

/backstores/user:glfs create LUN4 2G gv1@oel7db02/file4.img

/backstores/user:glfs create LUN5 2G gv1@oel7db02/file5.img

/iscsi/ create iqn.2016-04.org.gluster:oel7db02

从每个客户端获取iscsi的iqn:

more /etc/iscsi/initiatorname.iscsi

创建本存储节点到该客户端的iqn连接

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/acls/ create iqn.1988-12.com.oracle:ea8fb3b19c204

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/acls/ create iqn.1988-12.com.oracle:ea8fb3b19c205

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN1

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN2

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN3

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN4

/iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN5

saveconfig

需要在所有的存储节点上运行以上步骤。

但在/backstores/user:glfs下创建LUN时，指定为当前节点的volume,例如：当前节点为oel7db02,则使用的命令如下（同时需要从第1个存储节点上获取刚刚生成的WWN号）：

/backstores/user:glfs create LUN1 2G gv1@oel7db02/file1.img 965b4d0e-fedc-4729-9748-625a397c5f00

/backstores/user:glfs create LUN2 2G gv1@oel7db02/file2.img 69317210-2ea1-4c8b-87b3-98e52b35a788

/backstores/user:glfs create LUN3 2G gv1@oel7db02/file3.img 0f7da034-945b-4da6-9fa3-1c54ae6a8d66

/backstores/user:glfs create LUN4 2G gv1@oel7db02/file4.img d8304b6f-a77e-49cc-8a4e-fa95a34cbd93

/backstores/user:glfs create LUN5 2G gv1@oel7db02/file5.img f0594c4c-8e43-4a30-abf1-e0beb3129e3c

最终存储节点配置如下所示：

技术图片

4、计算节点配置

iscsiadm -m discovery -t sendtargets -p 192.168.56.201

iscsiadm -m discovery -t sendtargets -p 192.168.56.202

iscsiadm -m discovery -t sendtargets -p 192.168.56.203

iscsiadm -m node -l

最终，计算节点能看到所有的盘：

[root@oel7db04 ~]# fdisk -l |grep dev

Disk /dev/sda: 42.9 GB, 42949672960 bytes, 83886080 sectors

/dev/sda1 * 2048 2099199 1048576 83 Linux

/dev/sda2 2099200 83886079 40893440 8e Linux LVM

Disk /dev/mapper/ol_oel7db-root: 37.7 GB, 37706792960 bytes, 73646080 sectors

Disk /dev/mapper/ol_oel7db-swap: 4160 MB, 4160749568 bytes, 8126464 sectors

Disk /dev/sdb: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdc: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdd: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sde: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdf: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdg: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdh: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdi: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdj: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdk: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdl: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdm: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdn: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdo: 2147 MB, 2147483648 bytes, 4194304 sectors

Disk /dev/sdp: 2147 MB, 2147483648 bytes, 4194304 sectors

[root@oel7db04 ~]#

安装multipath:

# yum install -y device-mapper-multipath

cat > /etc/multipath.conf <<-EOF

# LIO iSCSI

devices {

device {

vendor "LIO-ORG"

path_grouping_policy "multibus"

path_selector "queue-length 0"

path_checker "directio"

prio "const"

rr_weight "uniform"

}

EOF

查看链路：

[root@oel7db04 ~]# multipath -ll

36001405f0594c4c8e434a30abf1e0beb dm-6 LIO-ORG ,TCMU device