OpenStack实践系列⑨云硬盘服务Cinder

Posted 2020-09-21 reblue520

OpenStack实践系列⑨云硬盘服务Cinder
八、cinder
8.1存储的三大分类

块存储：硬盘，磁盘阵列DAS，SAN存储
文件存储：nfs，GluserFS，Ceph（PB级分布式文件系统），MooserFS（缺点Metadata数据丢失，虚拟机就毁了）

11.2网络类型选择

对象存储：swift，S3

8.2 cinder控制节点的部署

安装cinder

[root@node1 ~]# yum install openstack-cinder python-cinderclient -y

 

修改cinder配置文件

[default]
rpc_backend = rabbit
glance_host = 192.168.3.199
auth_strategy = keystone

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

[oslo_messaging_rabbit]
rabbit_host = 192.168.3.199 # rabbitmq的主机
rabbit_port = 5672 # rabbitmq的端口
rabbit_userid = openstack # rabbitmq的用户
rabbit_password = openstack # rabbitmq的密码

[database]
connection = mysql://cinder:cinder@192.168.3.199/cinder # 配置mysql地址

[keystone_authtoken]
auth_uri = http://192.168.3.199:5000
auth_url = http://192.168.3.199:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder

修改后结果如下

[root@node1 cinder]# grep -n \'^[a-Z]\' /etc/cinder/cinder.conf 
421:glance_host = 192.168.3.199
536:auth_strategy = keystone
2294:rpc_backend = rabbit
2516:connection = mysql://cinder:cinder@192.168.3.199/cinder
2641:auth_uri = http://192.168.3.199:5000 
2642:auth_url = http://192.168.3.199:35357 
2643:auth_plugin = password 
2644:project_domain_id = default
2645:user_domain_id = default
2646:project_name = service
2647:username = cinder
2648:password = cinder
2874:lock_path = /var/lib/cinder/tmp
3173:rabbit_host = 192.168.3.199
3177:rabbit_port = 5672
3189:rabbit_userid = openstack
3193:rabbit_password = openstack

 

修改nova的配置文件

[root@node1 ~]# vim /etc/nova/nova.conf 
os_region_name = RegionOne # 通知nova使用cinder [cinder]部分

 

执行同步数据库操作

[root@node1 ~]# su -s /bin/sh -c "cinder-manage db sync" cinder

检查导入数据库结果
MariaDB [(none)]> use cinder
Database changed
MariaDB [cinder]> show tables;
+----------------------------+
| Tables_in_cinder |
+----------------------------+
| backups |
| cgsnapshots |
| consistencygroups |
| driver_initiator_data |
| encryption |
| image_volume_cache_entries |
| iscsi_targets |
| migrate_version |
| quality_of_service_specs |
| quota_classes |
| quota_usages |
| quotas |
| reservations |
| services |
| snapshot_metadata |
| snapshots |
| transfers |
| volume_admin_metadata |
| volume_attachment |
| volume_glance_metadata |
| volume_metadata |
| volume_type_extra_specs |
| volume_type_projects |
| volume_types |
| volumes |
+----------------------------+
25 rows in set (0.00 sec)

 

创建一个cinder用户,加入service项目，给予admin角色

[root@node1 ~]# source admin-openrc.sh
[root@node1 ~]# openstack user create --domain default --password-prompt cinder
User Password:
Repeat User Password: # （密码需要配置成cinder就是/etc/cinder/cinder.conf配置文件中配置的2648行）
+-----------+----------------------------------+
| Field | Value |
+-----------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 420d7573e9fc43b3b263f31bb6dd76e2 |
| name | cinder |
+-----------+----------------------------------+
[root@node1 ~]# openstack role add --project service --user cinder admin

 

重启nova-api服务和启动cinder服务

[root@node1 ~]# systemctl restart openstack-nova-api.service
[root@node1 ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service

[root@node1 ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service

创建服务（包含V1和V2）

[root@node1 ~]# openstack service create --name cinder --description "OpenStack Block Storage" volume
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 6e3b2c3940d14300ab28aed272ade1d3 |
| name | cinder |
| type | volume |
+-------------+----------------------------------+

[root@node1 ~]# openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| name | cinderv2 |
| type | volumev2 |
+-------------+----------------------------------+

 

分别对V1和V2创建三个环境（admin，internal，public）的endpoint

[root@node1 ~]# openstack endpoint create --region RegionOne volume public http://192.168.3.199:8776/v1/%\\(tenant_id\\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 007497468db7456d81157962f8740540 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+

[root@node1 ~]# openstack endpoint create --region RegionOne volume internal http://192.168.3.199:8776/v1/%\\(tenant_id\\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | e7543b96b69342bcabead7ad8a583860 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+

[root@node1 ~]# openstack endpoint create --region RegionOne volume admin http://192.168.3.199:8776/v1/%\\(tenant_id\\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 12e4bea586384d43b16e4de5a00afb1b |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 6e3b2c3940d14300ab28aed272ade1d3 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.3.199:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+

[root@node1 ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.3.199:8776/v2/%\\(tenant_id\\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 07c56b0033454fbda201f3cc58ce0a1b |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+

[root@node1 ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.3.199:8776/v2/%\\(tenant_id\\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 66b34a18d4de456ab32ebba24831b959 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+

[root@node1 ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.3.199:8776/v2/%\\(tenant_id\\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | fc0bed271a7048a5aff7e63aebd0199a |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.3.199:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+

 

8.3 cinder存储节点的部署（此处使用nova的计算节点即node2.chinasoft.com）

　　本文中cinder后端存储使用ISCSI（类似于nova-computer使用的kvm），ISCSI使用LVM，在定义好的VG中，每创建一个云硬盘，就会增加一个LV，使用ISCSI发布。
在存储节点上加一个硬盘

 

查看磁盘添加情况

[root@node2 ~]# fdisk -l

Disk /dev/sdb: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0004c2a9

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 616447 307200 83 Linux
/dev/sda2 616448 155811839 77597696 8e Linux LVM

Disk /dev/mapper/centos-root: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-data: 42.9 GB, 42945478656 bytes, 83877888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

创建一个pv和vg（名为cinder-volumes）

[root@node2 ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created
[root@node2 ~]# vgcreate cinder-volumes /dev/sdb
Volume group "cinder-volumes" successfully created

修改lvm的配置文件中添加filter，只有instance可以访问

[root@node2 ~]# vim /etc/lvm/lvm.conf
131 filter = [ "a/sdb/", "r/.*/"]

 

存储节点安装

[root@node2 ~]# yum install openstack-cinder targetcli python-oslo-policy -y

修改存储节点的配置文件，在这里直接拷贝控制节点的文件
[root@node1 ~]# scp /etc/cinder/cinder.conf 192.168.3.200:/etc/cinder/

修改存储(即计算节点node2.chinasoft.com)上的/etc/cinder/cinder.conf文件

添加如下配置：
默认没有[lvm]需要自己创建
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm后端存储
volume_group = cinder-volumes # vg的名称：刚才创建的
iscsi_protocol = iscsi # 使用iscsi协议
iscsi_helper = lioadm

[root@node2 cinder]# grep -n \'^[a-Z]\' /etc/cinder/cinder.conf
421:glance_host = 192.168.3.199
536:auth_strategy = keystone
540:enabled_backends = lvm # lvm 使用的后端是lvm，要对应添加的[lvm]，自定义也可以
2294:rpc_backend = rabbit
2516:connection = mysql://cinder:cinder@192.168.3.199/cinder
2641:auth_uri = http://192.168.3.199:5000
2642:auth_url = http://192.168.3.199:35357
2643:auth_plugin = password
2644:project_domain_id = default
2645:user_domain_id = default
2646:project_name = service
2647:username = cinder
2648:password = cinder
2874:lock_path = /var/lib/cinder/tmp
3173:rabbit_host = 192.168.3.199
3177:rabbit_port = 5672
3189:rabbit_userid = openstack
3193:rabbit_password = openstack
[lvm] # 此行不是grep过滤出来的,因为是在配置文件最后添加上的，其对应的是540行的lvm
3416:volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm后端存储
3417:volume_group = cinder-volumes # vg的名称：刚才创建的
3418:iscsi_protocol = iscsi # 使用iscsi协议
3419:iscsi_helper = lioadm

启动存储节点的cinder(这里是node2)

[root@node2 cinder]# systemctl enable openstack-cinder-volume.service target.service
Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack-cinder-volume.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
[root@node2 cinder]# systemctl start openstack-cinder-volume.service target.service

 

查看云硬盘服务状态（如果是虚拟机作为宿主机，时间不同步，会产生问题）

[root@node1 ~]# source admin-openrc.sh 
[root@node1 ~]# cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | node1.chinasoft.com | nova | enabled | up | 2017-04-28T09:59:54.000000 | - |
| cinder-volume | node2.chinasoft.com@lvm | nova | enabled | up | 2017-04-28T09:59:58.000000 | - |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+

 

创建一个云硬盘,将云硬盘挂载到虚拟机上,启动在虚拟机实例详情可以查看到，云硬盘挂载的情况
具体步骤：
1.创建云硬盘

2.点击 动作选项下的 管理已连接云硬盘，选择需要的虚拟机 将云硬盘挂载到指定的虚拟机上

在虚拟机中对挂载的硬盘进行分区格式化,如果有时不想挂载这个云硬盘了，一定不要删掉，生产环境一定要注意，否则虚拟机会出现error，应该使用umont确定卸载了，再使用dashboard进行删除云硬盘

# ssh cirros@192.168.3.103

$ sudo fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/vda1 * 16065 2088449 1036192+ 83 Linux

Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn\'t contain a valid partition table

# 分区
$ sudo fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x3fecc8a5.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won\'t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2097151, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): 
Using default value 2097151

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

查看分区，已经生成了新的分区/dev/vdb
$ sudo fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/vda1 * 16065 2088449 1036192+ 83 Linux

Disk /dev/vdb: 1073 MB, 1073741824 bytes
9 heads, 8 sectors/track, 29127 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xfaacdc93

Device Boot Start End Blocks Id System
/dev/vdb1 2048 2097151 1047552 83 Linux


# 格式化
$ sudo mkfs.ext4 /dev/vdb1
mke2fs 1.42.2 (27-Mar-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 261888 blocks
13094 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376

Allocating group tables: done 
Writing inode tables: done 
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

# 挂载
$ sudo mkdir /data
$ sudo mount /dev/vdb1 /data

$ df -h
Filesystem Size Used Available Use% Mounted on
/dev 242.3M 0 242.3M 0% /dev
/dev/vda1 23.2M 18.0M 4.0M 82% /
tmpfs 245.8M 0 245.8M 0% /dev/shm
tmpfs 200.0K 72.0K 128.0K 36% /run
/dev/vdb1 1006.9M 17.3M 938.5M 2% /data

 

方法二：从云硬盘启动虚拟机

①先创建一个demo2的云硬盘

②动作-作为实例启动

九、虚拟机创建流程：

第一阶段：用户操作
1)用户使用Dashboard或者CLI连接keystone，发送用户名和密码，待keystone验证通过，keystone会返回给dashboard一个authtoken
2）Dashboard会带着上述的authtoken访问nova-api进行创建虚拟机请求
3）nova-api会通过keytoken确认dashboard的authtoken认证消息。

第二阶段：nova内组件交互阶段
4）nova-api把用户要创建的虚拟机的信息记录到数据库中.
5）nova-api使用rpc-call的方式发送请求给消息队列
6）nova-scheduler获取消息队列中的消息
7）nova-scheduler和查看数据库中要创建的虚拟机信息和计算节点的信息，进行调度
8)nova-scheduler把调度后的信息发送给消息队列
9）nova-computer获取nova-schedur发送给queue的消息
10）nova-computer通过消息队列发送消息给nova-conudctor，想要获取数据库中的要创建虚拟机信息
11）nova-conductor获取消息队列的消息
12）nova-conductor读取数据库中要创建虚拟机的信息
13）nova-conductor把从数据库获取的消息返回给消息队列
14）nova-computer获取nova-conducter返回给消息队列的信息

第三阶段：nova和其他组件进行交互
15）nova-computer通过authtoken和数据库返回的镜像id请求glance服务
16）glance会通过keystone进行认证
17）glance验证通过后把镜像返回给nova-computer
18）nova-computer通过authtoken和数据库返回的网络id请求neutron服务
19）neutron会通过keystone进行认证
20）neutron验证通过后把网络分配情况返回给nova-computer
21）nova-computer通过authtoken和数据库返回的云硬盘请求cinder服务
22）cinder会通过keystone进行认证
23）cinder验证通过后把云硬盘分配情况返回给nova-computer

第四阶段：nova创建虚拟机
24)nova-compute通过libvirt调用kvm根据已有的信息创建虚拟机，动态生成xml
25)nova-api会不断的在数据库中查询信息并在dashboard显示虚拟机的状态

生产场景注意事项：
1、新加的一个计算节点，创建虚拟机时间会很长，因为第一次使用计算节点，没有镜像，计算节点要把glance的镜像放在后端文件（/var/lib/nova/instance/_base）下，
镜像如果很大，自然会需要很长时间，然后才会在后端文件的基础上创建虚拟机（写时复制copy on write）。
2、创建虚拟机失败的原因之一：创建网桥失败。要保证eth0网卡配置文件的BOOTPROTE是static而不是dhcp状态。