使用kolla-ansible部署多节点OpenStack(T版)及对接Ceph

Posted 2023-01-13 抛物线.

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了使用kolla-ansible部署多节点OpenStack(T版)及对接Ceph相关的知识，希望对你有一定的参考价值。

推荐感兴趣的小伙伴先阅读官方文档
近期使用kolla-ansible部署一套OpenStack多节点测试环境，用于测试计算节点宕机撤离的生产场景。
虽然官方文档写的非常详细，但是整个部署过程也多多少少遇到些问题

文章目录

现在将整个基于Kolla-ansible 快速部署的OpenStack多节点的详细操作过程，记录下来，方便小伙伴们快速搭建自己的环境。

① 这个是kolla-ansible的官方网站

https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html

② 这个是ansible的官方网站。

http://docs.ansible.com/

③ 这个是ceph的官方网站。

https://docs.openstack.org/kolla-ansible/queens/reference/ceph-guide.html

④ 这个是swift的官方网站。

https://docs.openstack.org/kolla-ansible/pike/reference/swift-guide.html

⑥ 我之前也做了docker的基础实验：

https://blog.csdn.net/qq_28513801/category_8592442.html

一、环境准备

基础环境是OpenStack T 版本.
对应的Ceph存储镜像Tag
Kolla-ansible版本为10.0.0

1、基础环境资源

主要组件	版本
Keystone*	T
Nova*	T
Glance*	T
Neutron*	T
Cinder*	T
Ironic *	T
Ceph*	T
Swift*	T
Haproxy	\|
Keepalived	\|

2、服务器资源架构

每台控制节点上各有两张网卡eth0 、eth1 (无地址)
每台存储节点上各有4块数据盘，其中 vdd为Ceph Cache加速盘

|用途|主机名|eth0|CPU|内存|vda |vdb | vdc| vdd| vde|
|–|–|–|–|–|–|–|–|–|–|–|
| 部署节点 | deploy | 172.31.234.212 | 8C |16G| 50G|\\ |\\ | \\ | \\ |
| 控制节点 | control01 | 172.31.234.212 | 8C |16G| 50G|\\ |\\ | \\ | \\ |
| 控制节点 | control02 | 172.31.234.52| 8C|16G| 50G|\\ |\\ | \\ | \\ |
| 控制节点| control03 | 172.31.234.142 | 8C |16G| 50G|\\ |\\ | \\ | \\ |
| 计算节点| compute01 | 172.31.234.246| 16C |32G| 50G|\\ |\\ | \\ | \\ |
| 计算节点 | compute02 | 172.31.234.226 | 16C |32G| 50G|\\ |\\ | \\ | \\ |
| 存储节点| ceph-01 | 172.31.234.27 | 4C |8G| 50G| 100G |100G | 80G | 100G |
| 存储节点| ceph-02 | 172.31.234.214 | 4C |8G| 50G| 100G |100G | 80G | 100G |
| 存储节点| ceph-03 | 172.31.234.218 | 4C |8G| 50G| 100G |100G | 80G | 100G |
| 网络节点| network01 | 172.31.234.179 | 8C |16G| 50G|\\ |\\ | \\ | \\ |

二、软件安装

2.1 关于kolla-ansible及本环境

关于新版本部署：
opentstack 版本和 Kolla-ansible 版本的对应关系如下：
Train 9.x.x
Stein 8.x.x
Rocky 7.x.x
queens 6.x.x
Pike 5.x.x
Ocata 4.x.x
pip3 install kolla-ansible==9.0.0.0rc1

为了方便自动补全，这里安装个插件

[root@control01 ~]# yum install -y bash-completion
[root@control01 ~]# source /usr/share/bash-completion/bash_completion

[root@control01 ~]#

本次安装环境信息：
[root@control01 ~]# cat /etc/hosts
127.0.0.1 localhost
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.234.212 control01
172.31.234.52 control02
172.31.234.142 control03
172.31.234.179 network01
172.31.234.246 compute01
172.31.234.226 compute02
172.31.234.27 ceph-01
172.31.234.214 ceph-02
172.31.234.218 ceph-03
172.31.241.232 registry

# BEGIN ANSIBLE GENERATED HOSTS
172.31.234.212 control01
172.31.234.52 control02
172.31.234.142 control03
172.31.234.179 network01
172.31.234.246 compute01
172.31.234.226 compute02
172.31.234.27 ceph-01
172.31.234.214 ceph-02
172.31.234.218 ceph-03
# END ANSIBLE GENERATED HOSTS

2.2 相关软件安装

[root@control01 ~]#   yum install -y yum-utils device-mapper-persistent-data lvm2

[root@control01 ~]# yum install docker-ce -y

[root@control01 ~]# systemctl daemon-reload
[root@control01 ~]# systemctl enable docker
[root@control01 ~]# systemctl start docker

[root@control01 ~]# yum install python-pip ansible –y 
[root@control01 ~]# pip install -U pip 

# 配置镜像加速

[root@control01 ~]# mkdir -p /etc/docker
[root@control01 ~]# tee /etc/docker/daemon.json <<-'EOF'

  "registry-mirrors": ["https://registry.docker-cn.com"]

EOF
[root@control01 ~]# systemctl daemon-reload
[root@control01 ~]# systemctl restart docker

2.3 节点初始化操作

所有节点均执行该初始化操作，避免安装途中报错

#!/bin/sh
sed -i 's/SELINUX=.*/SELINUX=Disabled/g' /etc/selinux/config
echo '' > /etc/resolv.conf
echo nameserver 114.114.114.114 >> /etc/resolv.conf
echo search novalocal >> /etc/resolv.conf
echo " net.ipv4.ip_forward = 1 ">> /etc/sysctl.conf&&sysctl -p
yum install vim wget -y
systemctl stop firewalld
systemctl disable firewalld

2.4 安装 kolla-ansible(version==9.0.0)

pip install kolla-ansible==9.0.0 #这里版本严格使用9.0.0，否则会报异常错误

可以适当添加源 
https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/

例如：
pip install kolla-ansible==9.0.0  -i  https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/


如果安装过程出现报错，可以先忽略该模块，如果需要就在安装。

例如：

Cannot uninstall 'PyYAML'. It is a distutils installed project 
and thus we cannot accurately determine which files
 belong to it which would lead to only a partial uninstall


则可以先忽略：
pip install kolla-ansible --ignore-installed PyYAML  -i  https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/

需要就在安装：
 
 pip  install XXXX==9.0.0

2.5 copy文件并配置下ansible的并行数


mkdir -p /etc/kolla
chown $USER:$USER /etc/kolla
cp -r /usr/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/　　　　##Copy globals.yml and passwords.yml
cp /usr/share/kolla-ansible/ansible/inventory/* .　　　　##Copy all-in-one and multinode inventory files

##### 1. Configure Ansible /etc/ansible/ansible.cfg

[defaults]
host_key_checking=False
pipelining=True
forks=100

2.6 配置主机清单文件，根据部署场景，这里选择 multinode 文件(下面贴出了全部的配置)


[root@control01 ~]# cat multinode
# These initial groups are the only groups required to be modified. The
# additional groups are for more control of the environment.
[control]
# These hostname must be resolvable from your deployment host
control01
control02
control03

# The above can also be specified as follows:
#control[01:03]     ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
network01

[compute]
compute01
compute02

[monitoring]
network01

# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]
ceph-01
ceph-02
ceph-03

[deployment]
localhost       ansible_connection=local

[baremetal:children]
control
network
compute
storage
monitoring

# You can explicitly specify which hosts run each project by updating the
# groups in the sections below. Common services are grouped together.
[chrony-server:children]
haproxy

[chrony:children]
control
network
compute
storage
monitoring

[collectd:children]
compute

[grafana:children]
monitoring

[etcd:children]
control

[influxdb:children]
monitoring

[prometheus:children]
monitoring

[kafka:children]
control

[karbor:children]
control

[kibana:children]
control

[telegraf:children]
compute
control
monitoring
network
storage

[elasticsearch:children]
control

[haproxy:children]
#network
control

[hyperv]
#hyperv_host

[hyperv:vars]
#ansible_user=user
#ansible_password=password
#ansible_port=5986
#ansible_connection=winrm
#ansible_winrm_server_cert_validation=ignore

[mariadb:children]
control

[rabbitmq:children]
control

[outward-rabbitmq:children]
control

[qdrouterd:children]
control

[monasca-agent:children]
compute
control
monitoring
network
storage

[monasca:children]
monitoring

[storm:children]
monitoring

[mongodb:children]
control

[keystone:children]
control

[glance:children]
control

[nova:children]
control

[neutron:children]
network

[openvswitch:children]
network
compute
manila-share

[opendaylight:children]
network

[cinder:children]
control

[cloudkitty:children]
control

[freezer:children]
control

[memcached:children]
control

[horizon:children]
control

[swift:children]
control

[barbican:children]
control

[heat:children]
control

[murano:children]
control

[solum:children]
control

[ironic:children]
control

[ceph:children]
control

[magnum:children]
control

[qinling:children]
control

[sahara:children]
control

[mistral:children]
control

[manila:children]
control

[ceilometer:children]
control

[aodh:children]
control

[cyborg:children]
control
compute

[congress:children]
control

[panko:children]
control

[gnocchi:children]
control

[tacker:children]
control

[trove:children]
control

# Tempest
[tempest:children]
control

[senlin:children]
control

[vmtp:children]
control

[vitrage:children]
control

[watcher:children]
control

[rally:children]
control

[searchlight:children]
control

[octavia:children]
control

[designate:children]
control

[placement:children]
control

[bifrost:children]
deployment

[zookeeper:children]
control

[zun:children]
control

[skydive:children]
monitoring

[redis:children]
control

[blazar:children]
control

# Additional control implemented here. These groups allow you to control which
# services run on which hosts at a per-service level.
#
# Word of caution: Some services are required to run on the same host to
# function appropriately. For example, neutron-metadata-agent must run on the
# same host as the l3-agent and (depending on configuration) the dhcp-agent.

# Glance
[glance-api:children]
glance

# Nova
[nova-api:children]
nova

[nova-conductor:children]
nova

[nova-super-conductor:children]
nova

[nova-novncproxy:children]
nova

[nova-scheduler:children]
nova

[nova-spicehtml5proxy:children]
nova

[nova-compute-ironic:children]
nova

[nova-serialproxy:children]
nova

# Neutron
[neutron-server:children]
control

[neutron-dhcp-agent:children]
neutron

[neutron-l3-agent:children]
neutron

[neutron-metadata-agent:children]
neutron

[neutron-bgp-dragent:children]
neutron

[neutron-infoblox-ipam-agent:children]
neutron

[neutron-metering-agent:children]
neutron

[ironic-neutron-agent:children]
neutron

# Ceph
[ceph-mds:children]
ceph

[ceph-mgr:children]
ceph

[ceph-nfs:children]
ceph

[ceph-mon:children]
ceph

[ceph-rgw:children]
ceph

[ceph-osd:children]
storage

# Cinder
[cinder-api:children]
cinder

[cinder-backup:children]
storage

[cinder-scheduler:children]
cinder

[cinder-volume:children]
storage

# Cloudkitty
[cloudkitty-api:children]
cloudkitty

[cloudkitty-processor:children]
cloudkitty

# Freezer
[freezer-api:children]
freezer

[freezer-scheduler:children]
freezer

# iSCSI
[iscsid:children]
compute
storage
ironic

[tgtd:children]
storage

# Karbor
[karbor-api:children]
karbor

[karbor-protection:children]
karbor

[karbor-operationengine:children]
karbor

# Manila
[manila-api:children]
manila

[manila-scheduler:children]
manila

[manila-share:children]
network

[manila-data:children]
manila

# Swift
[swift-proxy-server:children]
swift

[swift-account-server:children]
storage

[swift-container-server:children]
storage

[swift-object-server:children]
storage

# Barbican
[barbican-api:children]
barbican

[barbican-keystone-listener:children]
barbican

[barbican-worker:children]
barbican

# Heat
[heat-api:children]
heat

[heat-api-cfn:children]
heat

[heat-engine:children]
heat

# Murano
[murano-api:children]
murano

[murano-engine:children]
murano

# Monasca
[monasca-agent-collector:children]
monasca-agent

[monasca-agent-forwarder:children]
monasca-agent

[monasca-agent-statsd:children]
monasca-agent

[monasca-api:children]
monasca

[monasca-grafana:children]
monasca

[monasca-log-api:children]
monasca

[monasca-log-transformer:children]
monasca

[monasca-log-persister:children]
monasca

[monasca-log-metrics:children]
monasca

[monasca-thresh:children]
monasca

[monasca-notification:children]
monasca

[monasca-persister:children]
monasca

# Storm
[storm-worker:children]
storm

[storm-nimbus:children]
storm

# Ironic
[ironic-api:children]
ironic

[ironic-conductor:children]
ironic

[ironic-inspector:children]
ironic

[ironic-pxe:children]
ironic

[ironic-ipxe:children]
ironic

# Magnum
[magnum-api:children]
magnum

[magnum-conductor:children]
magnum

# Qinling
[qinling-api:children]
qinling

[qinling-engine:children]
qinling

# Sahara
[sahara-api:children]
sahara

[sahara-engine:children]
sahara

# Solum
[solum-api:children]
solum

[solum-worker:children]
solum

[solum-deployer:children]
solum

[solum-conductor:children]
solum

[solum-application-deployment:children]
solum

[solum-image-builder:children]
solum

# Mistral
[mistral-api:children]
mistral

[mistral-executor:children]
mistral

[mistral-engine:children]
mistral

[mistral-event-engine:children]
mistral

# Ceilometer
[ceilometer-central:children]
ceilometer

[ceilometer-notification:children]
ceilometer

[ceilometer-compute:children]
compute

[ceilometer-ipmi:children]
compute

# Aodh
[aodh-api:children]
aodh

[aodh-evaluator:children]
aodh

[aodh-listener:children]
aodh

[aodh-notifier:children]
aodh

# Cyborg
[cyborg-api:children]
cyborg

[cyborg-agent:children]
compute

[cyborg-conductor:children]
cyborg

# Congress
[congress-api:children]
congress

[congress-datasource:children]
congress

[congress-policy-engine:children]
congress

# Panko
[panko-api:children]
panko

# Gnocchi
[gnocchi-api:children]
gnocchi

[gnocchi-statsd:children]
gnocchi

[gnocchi-metricd:children]
gnocchi

# Trove
[trove-api:children]
trove

[trove-conductor:children]
trove

[trove-taskmanager:children]
trove

# Multipathd
[multipathd:children]
compute
storage

# Watcher
[watcher-api:children]
watcher

[watcher-engine:children]
watcher

[watcher-applier:children]
watcher

# Senlin
[senlin-api:children]
senlin

[senlin-engine:children]
senlin

# Searchlight
[searchlight-api:children]
searchlight

[searchlight-listener:children]
searchlight

# Octavia
[octavia-api:children]
octavia

[octavia-health-manager:children]
octavia

[octavia-housekeeping:children]
octavia

[octavia-worker:children]
octavia

# Designate
[designate-api:children]
designate

[designate-central:children]
designate

[designate-producer:children]
designate

[designate-mdns:children]
network

[designate-worker:children]
designate

[designate-sink:children]
designate

[designate-backend-bind9:children]
designate

# Placement
[placement-api:children]
placement

# Zun
[zun-api:children]
zun

[zun-wsproxy:children]
zun

[zun-compute:children]
compute

# Skydive
[skydive-analyzer:children]
skydive

[skydive-agent:children]
compute
network

# Tacker
[tacker-server:children]
tacker

[tacker-conductor:children]
tacker

# Vitrage
[vitrage-api:children]
vitrage

[vitrage-notifier:children]
vitrage

[vitrage-graph:children]
vitrage

[vitrage-ml:children]
vitrage

# Blazar
[blazar-api:children]
blazar

[blazar-manager:children]
blazar

# Prometheus
[prometheus-node-exporter:children]
monitoring
control
compute
network
storage

[prometheus-mysqld-exporter:children]
mariadb

[prometheus-haproxy-exporter:children]
haproxy

[prometheus-memcached-exporter:children]
memcached

[prometheus-cadvisor:children]
monitoring
control
compute
network
storage

[prometheus-alertmanager:children]
monitoring

[prometheus-openstack-exporter:children]
monitoring

[prometheus-elasticsearch-exporter:children]
elasticsearch

[prometheus-blackbox-exporter:children]
monitoring

[masakari-api:children]
control

[masakari-engine:children]
control

[masakari-monitors:children]
compute

2.7 配置global.yml(下面贴出了全部的配置及安装的组件)

kolla-ansible多节点安装openstack
 kolla-ansible 部署多region
 kolla-ansible部署单节点OpenStack-Pike
 kolla-ansible部署openstack
 适用于OpenStack  Ussuri 版本的 Kolla-Ansible部署文档
 kolla-ansible部署openstack yoga版本