使用kolla-ansible部署多节点OpenStack(T版)及对接Ceph

Posted 抛物线.

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了使用kolla-ansible部署多节点OpenStack(T版)及对接Ceph相关的知识，希望对你有一定的参考价值。

推荐感兴趣的小伙伴先阅读官方文档
近期使用kolla-ansible部署一套OpenStack多节点测试环境，用于测试计算节点宕机撤离的生产场景。
虽然官方文档写的非常详细，但是整个部署过程也多多少少遇到些问题

现在将整个基于Kolla-ansible 快速部署的OpenStack多节点的详细操作过程，记录下来，方便小伙伴们快速搭建自己的环境。

① 这个是kolla-ansible的官方网站

https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html

② 这个是ansible的官方网站。

http://docs.ansible.com/

③ 这个是ceph的官方网站。

https://docs.openstack.org/kolla-ansible/queens/reference/ceph-guide.html

④ 这个是swift的官方网站。

https://docs.openstack.org/kolla-ansible/pike/reference/swift-guide.html

⑥ 我之前也做了docker的基础实验：

https://blog.csdn.net/qq_28513801/category_8592442.html

一、环境准备

  • 基础环境是OpenStack T 版本.
  • 对应的Ceph存储镜像Tag
  • Kolla-ansible版本为10.0.0
1、基础环境资源
主要组件版本
Keystone*T
Nova*T
Glance*T
Neutron*T
Cinder*T
Ironic *T
Ceph*T
Swift*T
Haproxy\\
Keepalived\\
2、服务器资源架构
  • 每台控制节点上各有两张网卡eth0 、eth1 (无地址)
  • 每台存储节点上各有4块数据盘，其中 vdd为Ceph Cache加速盘
用途主机名eth0CPU内存vdavdbvdcvddvde
部署节点deploy172.31.234.2128C16G50G\\\\\\\\
控制节点control01172.31.234.2128C16G50G\\\\\\\\
控制节点control02172.31.234.528C16G50G\\\\\\\\
控制节点control03172.31.234.1428C16G50G\\\\\\\\
计算节点compute01172.31.234.24616C32G50G\\\\\\\\
计算节点compute02172.31.234.22616C32G50G\\\\\\\\
存储节点ceph-01172.31.234.274C8G50G100G100G80G100G
存储节点ceph-02172.31.234.2144C8G50G100G100G80G100G
存储节点ceph-03172.31.234.2184C8G50G100G100G80G100G
网络节点network01172.31.234.1798C16G50G\\\\\\\\

二、软件安装

2.1 关于kolla-ansible及本环境
关于新版本部署：
opentstack 版本和 Kolla-ansible 版本的对应关系如下：
Train 9.x.x
Stein 8.x.x
Rocky 7.x.x
queens 6.x.x
Pike 5.x.x
Ocata 4.x.x
pip3 install kolla-ansible==9.0.0.0rc1   

为了方便自动补全，这里安装个插件

[root@control01 ~]# yum install -y bash-completion
[root@control01 ~]# source /usr/share/bash-completion/bash_completion

[root@control01 ~]#

本次安装环境信息：
[root@control01 ~]# cat /etc/hosts
127.0.0.1 localhost
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.234.212 control01
172.31.234.52 control02
172.31.234.142 control03
172.31.234.179 network01
172.31.234.246 compute01
172.31.234.226 compute02
172.31.234.27 ceph-01
172.31.234.214 ceph-02
172.31.234.218 ceph-03
172.31.241.232 registry

# BEGIN ANSIBLE GENERATED HOSTS
172.31.234.212 control01
172.31.234.52 control02
172.31.234.142 control03
172.31.234.179 network01
172.31.234.246 compute01
172.31.234.226 compute02
172.31.234.27 ceph-01
172.31.234.214 ceph-02
172.31.234.218 ceph-03
# END ANSIBLE GENERATED HOSTS
2.2 相关软件安装
[root@control01 ~]#   yum install -y yum-utils device-mapper-persistent-data lvm2

[root@control01 ~]# yum install docker-ce -y

[root@control01 ~]# systemctl daemon-reload
[root@control01 ~]# systemctl enable docker
[root@control01 ~]# systemctl start docker

[root@control01 ~]# yum install python-pip ansible –y 
[root@control01 ~]# pip install -U pip 

# 配置镜像加速

[root@control01 ~]# mkdir -p /etc/docker
[root@control01 ~]# tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://registry.docker-cn.com"]
}
EOF
[root@control01 ~]# systemctl daemon-reload
[root@control01 ~]# systemctl restart docker
2.3 节点初始化操作
  • 所有节点均执行该初始化操作，避免安装途中报错
#!/bin/sh
sed -i 's/SELINUX=.*/SELINUX=Disabled/g' /etc/selinux/config
echo '' > /etc/resolv.conf
echo nameserver 114.114.114.114 >> /etc/resolv.conf
echo search novalocal >> /etc/resolv.conf
echo " net.ipv4.ip_forward = 1 ">> /etc/sysctl.conf&&sysctl -p
yum install vim wget -y
systemctl stop firewalld
systemctl disable firewalld
2.4 安装 kolla-ansible(version==9.0.0)
pip install kolla-ansible==9.0.0 #这里版本严格使用9.0.0，否则会报异常错误

可以适当添加源 
https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/

例如：
pip install kolla-ansible==9.0.0  -i  https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/


如果安装过程出现报错，可以先忽略该模块，如果需要就在安装。

例如：

Cannot uninstall 'PyYAML'. It is a distutils installed project 
and thus we cannot accurately determine which files
 belong to it which would lead to only a partial uninstall


则可以先忽略：
pip install kolla-ansible --ignore-installed PyYAML  -i  https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/

需要就在安装：
 
 pip  install XXXX==9.0.0
2.5 copy文件并配置下ansible的并行数

mkdir -p /etc/kolla
chown $USER:$USER /etc/kolla
cp -r /usr/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/　　　　##Copy globals.yml and passwords.yml
cp /usr/share/kolla-ansible/ansible/inventory/* .　　　　##Copy all-in-one and multinode inventory files

##### 1. Configure Ansible /etc/ansible/ansible.cfg

[defaults]
host_key_checking=False
pipelining=True
forks=100
2.6 配置主机清单文件，根据部署场景，这里选择 multinode 文件(下面贴出了全部的配置)

[root@control01 ~]# cat multinode
# These initial groups are the only groups required to be modified. The
# additional groups are for more control of the environment.
[control]
# These hostname must be resolvable from your deployment host
control01
control02
control03

# The above can also be specified as follows:
#control[01:03]     ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
network01

[compute]
compute01
compute02

[monitoring]
network01

# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]
ceph-01
ceph-02
ceph-03

[deployment]
localhost       ansible_connection=local

[baremetal:children]
control
network
compute
storage
monitoring

# You can explicitly specify which hosts run each project by updating the
# groups in the sections below. Common services are grouped together.
[chrony-server:children]
haproxy

[chrony:children]
control
network
compute
storage
monitoring

[collectd:children]
compute

[grafana:children]
monitoring

[etcd:children]
control

[influxdb:children]
monitoring

[prometheus:children]
monitoring

[kafka:children]
control

[karbor:children]
control

[kibana:children]
control

[telegraf:children]
compute
control
monitoring
network
storage

[elasticsearch:children]
control

[haproxy:children]
#network
control

[hyperv]
#hyperv_host

[hyperv:vars]
#ansible_user=user
#ansible_password=password
#ansible_port=5986
#ansible_connection=winrm
#ansible_winrm_server_cert_validation=ignore

[mariadb:children]
control

[rabbitmq:children]
control

[outward-rabbitmq:children]
control

[qdrouterd:children]
control

[monasca-agent:children]
compute
control
monitoring
network
storage

[monasca:children]
monitoring

[storm:children]
monitoring

[mongodb:children]
control

[keystone:children]
control

[glance:children]
control

[nova:children]
control

[neutron:children]
network

[openvswitch:children]
network
compute
manila-share

[opendaylight:children]
network

[cinder:children]
control

[cloudkitty:children]
control

[freezer:children]
control

[memcached:children]
control

[horizon:children]
control

[swift:children]
control

[barbican:children]
control

[heat:children]
control

[murano:children]
control

[solum:children]
control

[ironic:children]
control

[ceph:children]
control

[magnum:children]
control

[qinling:children]
control

[sahara:children]
control

[mistral:children]
control

[manila:children]
control

[ceilometer:children]
control

[aodh:children]
control

[cyborg:children]
control
compute

[congress:children]
control

[panko:children]
control

[gnocchi:children]
control

[tacker:children]
control

[trove:children]
control

# Tempest
[tempest:children]
control

[senlin:children]
control

[vmtp:children]
control

[vitrage:children]
control

[watcher:children]
control

[rally:children]
control

[searchlight:children]
control

[octavia:children]
control

[designate:children]
control

[placement:children]
control

[bifrost:children]
deployment

[zookeeper:children]
control

[zun:children]
control

[skydive:children]
monitoring

[redis:children]
control

[blazar:children]
control

# Additional control implemented here. These groups allow you to control which
# services run on which hosts at a per-service level.
#
# Word of caution: Some services are required to run on the same host to
# function appropriately. For example, neutron-metadata-agent must run on the
# same host as the l3-agent and (depending on configuration) the dhcp-agent.

# Glance
[glance-api:children]
glance

# Nova
[nova-api:children]
nova

[nova-conductor:children]
nova

[nova-super-conductor:children]
nova

[nova-novncproxy:children]
nova

[nova-scheduler:children]
nova

[nova-spicehtml5proxy:children]
nova

[nova-compute-ironic:children]
nova

[nova-serialproxy:children]
nova

# Neutron
[neutron-server:children]
control

[neutron-dhcp-agent:children]
neutron

[neutron-l3-agent:children]
neutron

[neutron-metadata-agent:children]
neutron

[neutron-bgp-dragent:children]
neutron

[neutron-infoblox-ipam-agent:children]
neutron

[neutron-metering-agent:children]
neutron

[ironic-neutron-agent:children]
neutron

# Ceph
[ceph-mds:children]
ceph

[ceph-mgr:children]
ceph

[ceph-nfs:children]
ceph

[ceph-mon:children]
ceph

[ceph-rgw:children]
ceph

[ceph-osd:children]
storage

# Cinder
[cinder-api:children]
cinder

[cinder-backup:children]
storage

[cinder-scheduler:children]
cinder

[cinder-volume:children]
storage

# Cloudkitty
[cloudkitty-api:children]
cloudkitty

[cloudkitty-processor:children]
cloudkitty

# Freezer
[freezer-api:children]
freezer

[freezer-scheduler:children]
freezer

# iSCSI
[iscsid:children]
compute
storage
ironic

[tgtd:children]
storage

# Karbor
[karbor-api:children]
karbor

[karbor-protection:children]
karbor

[karbor-operationengine:children]
karbor

# Manila
[manila-api:children]
manila

[manila-scheduler:children]
manila

[manila-share:children]
network

[manila-data:children]
manila

# Swift
[swift-proxy-server:children]
swift

[swift-account-server:children]
storage

[swift-container-server:children]
storage

[swift-object-server:children]
storage

# Barbican
[barbican-api:children]
barbican

[barbican-keystone-listener:children]
barbican

[barbican-worker:children]
barbican

# Heat
[heat-api:children]
heat

[heat-api-cfn:children]
heat

[heat-engine:children]
heat

# Murano
[murano-api:children]
murano

[murano-engine:children]
murano

# Monasca
[monasca-agent-collector:children]
monasca-agent

[monasca-agent-forwarder:children]
monasca-agent

[monasca-agent-statsd:children]
monasca-agent

[monasca-api:children]
monasca

[monasca-grafana:children]
monasca

[monasca-log-api:children]
monasca

[monasca-log-transformer:children]
monasca

[monasca-log-persister:children]
monasca

[monasca-log-metrics:children]
monasca

[monasca-thresh:children]
monasca

[monasca-notification:children]
monasca

[monasca-persister:children]
monasca

# Storm
[storm-worker:children]
storm

[storm-nimbus:children]
storm

# Ironic
[ironic-api:children]
ironic

[ironic-conductor:children]
ironic

[ironic-inspector:children]
ironic

[ironic-pxe:children]
ironic

[ironic-ipxe:children]
ironic

# Magnum
[magnum-api:children]
magnum

[magnum-conductor:children]
magnum

# Qinling
[qinling-api:children]
qinling

[qinling-engine:children]
qinling

# Sahara
[sahara-api:children]
sahara

[sahara-engine:children]
sahara

# Solum
[solum-api:children]
solum

[solum-worker:children]
solum

[solum-deployer:children]
solum

[solum-conductor:children]
solum

[solum-application-deployment:children]
solum

[solum-image-builder:children]
solum

# Mistral
[mistral-api:children]
mistral

[mistral-executor:children]
mistral

[mistral-engine:children]
mistral

[mistral-event-engine:children]
mistral

# Ceilometer
[ceilometer-central:children]
ceilometer

[ceilometer-notification:children]
ceilometer

[ceilometer-compute:children]
compute

[ceilometer-ipmi:children]
compute

# Aodh
[aodh-api:children]
aodh

[aodh-evaluator:children]
aodh

[aodh-listener:children]
aodh

[aodh-notifier:children]
aodh

# Cyborg
[cyborg-api:children]
cyborg

[cyborg-agent:children]
compute

[cyborg-conductor:children]
cyborg

# Congress
[congress-api:children]
congress

[congress-datasource:children]
congress

[congress-policy-engine:children]
congress

# Panko
[panko-api:children]
panko

# Gnocchi
[gnocchi-api:children]
gnocchi

[gnocchi-statsd:children]
gnocchi

[gnocchi-metricd:children]
gnocchi

# Trove
[trove-api:children]
trove

[trove-conductor:children]
trove

[trove-taskmanager:children]
trove

# Multipathd
[multipathd:children]
compute
storage

# Watcher
[watcher-api:children]
watcher

[watcher-engine:children]
watcher

[watcher-applier:children]
watcher

# Senlin
[senlin-api:children]
senlin

[senlin-engine:children]
senlin

# Searchlight
[searchlight-api:children]
searchlight

[searchlight-listener:children]
searchlight

# Octavia
[octavia-api:children]
octavia

[octavia-health-manager:children]
octavia

[octavia-housekeeping:children]
octavia

[octavia-worker:children]
octavia

# Designate
[designate-api:children]
designate

[designate-central:children]
designate

[designate-producer:children]
designate

[designate-mdns:children]
network

[designate-worker:children]
designate

[designate-sink:children]
designate

[designate-backend-bind9:children]
designate

# Placement
[placement-api:children]
placement

# Zun
[zun-api:children]
zun

[zun-wsproxy:children]
zun

[zun-compute:children]
compute

# Skydive
[skydive-analyzer:children]
skydive

[skydive-agent:children]
compute
network

# Tacker
[tacker-server:children]
tacker

[tacker-conductor:children]
tacker

# Vitrage
[vitrage-api:children]
vitrage

[vitrage-notifier:children]
vitrage

[vitrage-graph:children]
vitrage

[vitrage-ml:children]
vitrage

# Blazar
[blazar-api:children]
blazar

[blazar-manager:children]
blazar

# Prometheus
[prometheus-node-exporter:children]
monitoring
control
compute
network
storage

[prometheus-mysqld-exporter:children]
mariadb

[prometheus-haproxy-exporter:children]
haproxy

[prometheus-memcached-exporter:children]
memcached

[prometheus-cadvisor:children]
monitoring
control
compute
network
storage

[prometheus-alertmanager:children]
monitoring

[prometheus-openstack-exporter:children]
monitoring

[prometheus-elasticsearch-exporter:children]
elasticsearch

[prometheus-blackbox-exporter:children]
monitoring

[masakari-api:children]
control

[masakari-engine:children]
control

[masakari-monitors:children]
compute
2.7 配置global.yml(下面贴出了全部的配置及安装的组件)
[root@control01 kolla]# cat globals.yml | grep -v '^#'| grep -v '^$'
---
kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "train"
node_custom_config: "/etc/kolla/config"
kolla_internal_vip_address: "172.31.234.208"
network_interface: "eth0"
kolla_external_vip_interface: "{{ network_interface }}"
api_interface: "{{ network_interface }}"
storage_interface: "{{ network_interface }}"
cluster_interface: "{{ network_interface }}"
swift_storage_interface: "{{ storage_interface }}"
swift_replication_interface: "{