云物理机ironic对接ceph云盘ceph-iscsi-gateway
Posted 抛物线.
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了云物理机ironic对接ceph云盘ceph-iscsi-gateway相关的知识，希望对你有一定的参考价值。
云物理机对接ceph云盘
1300042631@qq.com
2022/04/29
文章目录
- 云物理机对接ceph云盘
背景说明
根据社区反馈当前OpenStack已经超大规模部署，生产环境也使用逐渐上线了云物理机功能，并对ironic组件及其ironic-inspect周边服务进行了二次开发了，但是云物理机本身默认使用的是本地盘，即直插物理磁盘，无论是系统盘，还是数据盘，均是。但是这样虽然性能稳定，但是也带来了一定的困扰，因为云物理机并不想kvm云主机那样支持快速弹性伸缩，比如cpu和内存规格变更，云主机可以做到实时变更，但是云物理机目前无法做到。
- 比如某个业务线一年前申请的云物理机，突然发现数据盘不足，想要改配，这就比较麻烦，因为牵涉到订单和价格计费的问题，不通的数据盘大小，收费和raid模式不同。比如原来是4Tx2的数据盘，因为业务需求，改成8Tx2，这就需要投入人力改订单改套餐flavor，改ironic的数据库nodes表中的properties字段等相关人工介入的操作。比较浪费人力，且存在风险。因为ironic基本是通过ipmitool来管控的，稍不注意动了某个字段，就有可能对云物理机本身造成影响。
- 还有一些场景，比如云物理机上架后，装机时，频繁失败，或者boot failed，排查一圈发现时系统盘未做raid导致。其实，熟悉云物理机裸金属的小伙伴都知道，看似高大上的云物理机自动装机，无非还是通过pxe\\dhcp\\tftp\\get镜像\\cloud_init装机。因为启动盘未做raid导致装机失败，生产环境，云物理机开机关机大概需要20来分钟，所以排查起来也非常耗时。
- 所以针对两个比较重要的场景，需要对云物理机和也支持弹性伸缩，当然这里仅指数据盘和系统盘，不支持内存和cpu核数。
- 根据当前云主机底层使用的块存储ceph场景，计划采用ceph-iscsi-gw来定制ironic对接云盘问题，具体详情及配置方案如下，云盘性能方面还是可观的。
一、基础环境信息
1. 版本依赖
云物理机对接ceph云盘
1300042631@qq.com
2022/04/29
一、基础环境信息
1. 版本依赖
| 应用 | 使用版本 | 版本要求 |
|---|---|---|
| OS | CentOS Linux release 7.6.1810 (Core) | Red Hat Enterprise Linux/CentOS 7.5 (or newer);Linux kernel v4.16 (or newer) |
| Ceph | Ceph Nautilus Stable (12.2.x) | Ceph Luminous Stable |
| Ceph-Iscsi-Gateway | ceph-iscsi-3.2 | ceph-iscsi-3.2 or newer package |
| Targetcli | targetcli-2.1.fb47 | targetcli-2.1.fb47 or newer package |
| Python-rtslib | python-rtslib-2.1.fb68 | python-rtslib-2.1.fb68 or newer package |
| Tcmu-runner | tcmu-runner-1.5.2 | tcmu-runner-1.4.0 or newer package |
2. 集群信息：
| IP | 用途 | OS |
|---|---|---|
| 192.168.9.101 | ceph-node1 | CentOS Linux release 7.6.1810 (Core) |
| 192.168.10.135 | ceph-node2 | CentOS Linux release 7.6.1810 (Core) |
| 192.168.9.190 | ceph-node3 | CentOS Linux release 7.6.1810 (Core) |
| 192.31.162.123 | rg2-test-control001.ostack.hfb3.iflytek.net | CentOS Linux release 7.6.1810 (Core) |
| 192.31.162.124 | rg2-test-control002.ostack.hfb3.iflytek.net | CentOS Linux release 7.6.1810 (Core) |
| 192.31.162.125 | rg2-test-control003.ostack.hfb3.iflytek.net | CentOS Linux release 7.6.1810 (Core) |
二、 环境部署
iSCSI 网关提供一个高可用性 （HA） iSCSI 目标，用于将 RADOS 块储存设备 （RBD） 映像导出为 SCSI 磁盘。
iSCSI 协议允许客户端(发起方)通过TCP/IP网络向存储设备（目标）发送SCSI命令,
使没有本机Ceph客户端支持的客户端能够访问 Ceph 块存储。
其中包括Microsoft Windows甚至Bios。
每个iSCSI网关都利用 Linux IO 目标内核子系统 （LIO） 来提供 iSCSI 协议支持。
LIO 利用用户空间直通 （TCMU） 与 Ceph 的 librbd 库进行交互，
并向 iSCSI 客户端公开 RBD 映像。
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-iQEgQ6Dg-1665744655793)(ceph.png)]
1. 部署一套ceph环境，并确认集群状态为ok
# 当前测试集群的状态
[root@ceph-node1 ~]# ceph -s
cluster:
id: c0df9eb6-5d23-4f14-8136-e2351fa215f7
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 28h)
mgr: ceph-node1(active, since 2w)
mds: cephfs-demo:1 0=ceph-node2=up:active 2 up:standby
osd: 15 osds: 15 up (since 2w), 15 in (since 2w)
rgw: 1 daemon active (ceph-node1)
task status:
data:
pools: 13 pools, 544 pgs
objects: 32.73k objects, 126 GiB
usage: 395 GiB used, 1.1 TiB / 1.4 TiB avail
pgs: 544 active+clean
io:
client: 1.5 KiB/s rd, 1 op/s rd, 0 op/s wr
[root@ceph-node1 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.44571 root default
-3 0.46884 host ceph-node1
0 hdd 0.09769 osd.0 up 0.81003 1.00000
3 hdd 0.09769 osd.3 up 0.81003 1.00000
6 hdd 0.09769 osd.6 up 1.00000 1.00000
9 hdd 0.09769 osd.9 up 1.00000 1.00000
10 hdd 0.07809 osd.10 up 1.00000 1.00000
-5 0.48843 host ceph-node2
1 hdd 0.09769 osd.1 up 0.95001 1.00000
4 hdd 0.09769 osd.4 up 1.00000 1.00000
7 hdd 0.09769 osd.7 up 1.00000 1.00000
11 hdd 0.09769 osd.11 up 1.00000 1.00000
13 hdd 0.09769 osd.13 up 0.90001 1.00000
-7 0.48843 host ceph-node3
2 hdd 0.09769 osd.2 up 1.00000 1.00000
5 hdd 0.09769 osd.5 up 0.81003 1.00000
8 hdd 0.09769 osd.8 up 1.00000 1.00000
12 hdd 0.09769 osd.12 up 0.81003 1.00000
14 hdd 0.09769 osd.14 up 1.00000 1.00000
[root@ceph-node1 ~]#
2. 定制配置 /etc/ceph/ceph.conf
1). 降低检测关闭 OSD 的默认检测信号间隔,来降低启动超时
[osd]
osd heartbeat grace = 20
osd heartbeat interval = 5
2). 更新下 monitor 运行状态
ceph tell <daemon_type>.<id> config set <parameter_name> <new_value>
ceph tell osd.* config set osd_heartbeat_grace 20
ceph tell osd.* config set osd_heartbeat_interval 5
3). 更新下 OSD 的运行状态
ceph daemon <daemon_type>.<id> config set osd_client_watch_timeout 15
ceph daemon osd.0 config set osd_heartbeat_grace 20
ceph daemon osd.0 config set osd_heartbeat_interval 5
3. 部署ceph-iscsi
# 可以使用ceph-ansible 也可以使用命令行分步的形式,具体参考官方网站
# https://docs.ceph.com/en/quincy/rbd/iscsi-target-cli-manual-install/
1. 在所有的iscsi gateway 节点， 安装Ceph-iscsi:
[root@ceph-node1 ~]# yum install ceph-iscsi
2. 在所有的iscsi gateway 节点， 安装tcmu-runner:
[root@ceph-node1 ~]# yum install tcmu-runner
# 事实上 安装方式可以选择源码编译或者rpm包,这个包不是很方便找:
# https://1.chacra.ceph.com/r/tcmu-runner/master/
#9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/x86_64/tcmu-runner-1.5.2-1.el7.x86_64.rpm
# https://1.chacra.ceph.com/r/tcmu-runner/master/
#9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/x86_64/libtcmu-devel-1.5.2-1.el7.x86_64.rpm
#https://1.chacra.ceph.com/r/tcmu-runner/master/
#9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/x86_64/libtcmu-1.5.2-1.el7.x86_64.rpm
3. 开始配置iscsi gateway:
1). ceph-iscsi默认使用pool名称为rbd的存储池，所以需要新建一个,之后查询下是否新建成功：
[root@ceph-node1 ~]# ceph osd lspools
2). 新建一个iscsi-gateway.cfg配置文件
[root@ceph-node1 ~]# touch /etc/ceph/iscsi-gateway.cfg
3). 编辑/etc/ceph/iscsi-gateway.cfg文件
[root@ceph-node1 ~]# cat /etc/ceph/iscsi-gateway.cfg
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph
# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring
# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.
# To support the API, the bear minimum settings are:
api_secure = false
# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
trusted_ip_list = 192.168.9.101,192.168.10.135,192.168.33.146,192.31.162.123
4). 复制 iscsi-gateway.cfg file to all iSCSI gateway nodes.
4. on all iSCSI gateway nodes, enable and start the API service:
[root@ceph-node1 ~]# systemctl daemon-reload
[root@ceph-node1 ~]# systemctl enable rbd-target-gw
[root@ceph-node1 ~]# systemctl start rbd-target-gw
[root@ceph-node1 ~]# systemctl enable rbd-target-api
[root@ceph-node1 ~]# systemctl start rbd-target-api
4. 调试ceph-iscsi
# gwcli 将创建和配置 iSCSI 目标和 RBD 映像: 配置是相对简单的，这里参考官方的案例
1. As root, on a iSCSI gateway node, start the iSCSI gateway command-line interface:
[root@ceph-node1 ~]# gwcli
2. Go to iscsi-targets and create a target with the name iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw:
> /> cd /iscsi-targets
> /iscsi-targets> create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
3. Create the iSCSI gateways.
The IPs used below are the ones that will be used for iSCSI data like READ and WRITE commands.
They can be the same IPs used for management operations listed in trusted_ip_list,
but it is recommended that different IPs are used.
> /iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways
> /iscsi-target...-igw/gateways> create ceph-gw-1 192.168.9.101
> /iscsi-target...-igw/gateways> create ceph-gw-2 192.168.10.135
If not using RHEL/CentOS or using an upstream or ceph-iscsi-test kernel,
the skipchecks=true argument must be used.
This will avoid the Red Hat kernel and rpm checks:
> /iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways
> /iscsi-target...-igw/gateways> create ceph-gw-1 192.168.9.101 skipchecks=true
> /iscsi-target...-igw/gateways> create ceph-gw-2 192.168.10.135 skipchecks=true
4. Add a RBD image with the name disk_1 in the pool rbd:
> /iscsi-target...-igw/gateways> cd /disks
> /disks> create pool=rbd image=disk_1 size=90G
5. Create a client with the initiator name iqn.1994-05.com.redhat:rh7-client:
> /disks> cd /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts
> /iscsi-target...eph-igw/hosts> create iqn.1994-05.com.redhat:rh7-client
6. Set the client’s CHAP username to myiscsiusername and password to myiscsipassword:
> /iscsi-target...at:rh7-client> auth username=myiscsiusername password=myiscsipassword
# CHAP must always be configured. Without CHAP, the target will reject any login requests.
7. Add the disk to the client:
> /iscsi-target...at:rh7-client> disk add rbd/disk_1
8. example:
[root@ceph-node1 ~]# gwcli
/> ls
o- / ......................................................................................................................... [...]
o- cluster ......................................................................................................... [Clusters: 1]
| o- ceph ............................................................................................................ [HEALTH_OK]
| o- pools ......................................................................................................... [Pools: 13]
| | o- .rgw.root ............................................................. [(x3), Commit: 0.00Y/252112656K (0%), Used: 768K]
| | o- backups ............................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 192K]
| | o- cephfs_data .......................................................... [(x3), Commit: 0.00Y/252112656K (0%), Used: 0.00Y]
| | o- cephfs_metadata .........................................云物理机ironic对接ceph云盘ceph-iscsi-gateway