kolla-ansible多节点安装openstack

Posted 2021-04-25 DevOps

基本功能部署

基础环境

角色	操作系统	硬件配置
Depoly	CentOS 7 Server	磁盘:40GB 内存:8GB 网卡:ens3(内网) ens4(外网)
Sched	CentOS 7 Server	磁盘:40GB 内存:8GB 网卡:ens3(内网) ens4(外网)
Nova	CentOS 7 Server	磁盘:40GB 内存:8GB 网卡:ens3(内网) CPU开启嵌套虚拟化

网络配置

Deploy配置

# vim ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=4.0.0.10
PREFIX=24

# vim ifcfg-ens4
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens4
DEVICE=ens4
ONBOOT=yes
IPADDR=192.168.200.10
NETMASK=255.255.255.0
GATEWAY=192.168.200.1
DNS1=114.114.114.114

Sched配置

# cat ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=4.0.0.11
PREFIX=24

# cat ifcfg-ens4
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens4
DEVICE=ens4
ONBOOT=yes

Nova配置

# cat ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=4.0.0.12
PREFIX=24

Deploy基础环境配置

安装PIP

# yum install epel-release
# yum install python-pip # pip install -U pip

安装PIP编译环境

# yum install python-devel libffi-devel gcc openssl-devel

安装ansible

# pip install -U ansible

安装docker

# curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -
# docker --version

配置docker环境

### 配置镜像加速
# mkdir -p /etc/docker# tee /etc/docker/daemon.json <<-'EOF'
{"registry-mirrors":  ["https://ar4vj99q.mirror.aliyuncs.com"] } EOF
# systemctl daemon-reload
# systemctl restart docker# systemctl enable docker

安装kolla-ansible

# git clone http://git.trystack.cn/openstack/kolla-ansible
# cd kolla-ansible
# git checkout stable/ocata# pip install .
# cp -r etc/kolla /etc
### 拷贝inventory到当前目录，也就是后面要执行kolla-ansible命令的目录
# cp ansible/inventory/* ~/

配置kolla-ansible

### 设置网卡信息
# vim /etc/kolla/globals.ymlkolla_internal_vip_address: "4.0.0.9" 
### 选择一个没被使用的内网地址keepalived_virtual_router_id: "9" 
### 与kolla_internal_vip_address末尾相同，防止同内网环境其他openstack部署环境干扰network_interface: "ens3"neutron_external_interface: "ens4"openstack_logging_debug: "True"nova_console: "spice"
# kolla-genpwd# vim /etc/kolla/passwords.ymlkeystone_admin_password: admin

关闭防火墙和SELINUX

# systemctl stop firewalld
# systemctl disable firewalld
# setenforce 0
# vim /etc/selinux/config
SELINUX=disabled

SSH免密码登入

### Deploy执行
# ssh-keygen -t rsa
### Sched和Nova执行
# scp root@4.0.0.10:~/.ssh/id_rsa.pub ./
# cat id_rsa.pub >> ~/.ssh/authorized_keys
# chmod 600 ~/.ssh/authorized_keys

获取docker镜像

在Deploy节点上设置本地镜像源

# docker run -d -v /opt/registry:/var/lib/registry -p 4000:5000 --restart=always --name registry registry:2
# wget http://tarballs.openstack.org/kolla/images/centos-source-registry-ocata.tar.gz
# tar zxf centos-source-registry-ocata.tar.gz -C /opt/registry/
# vim /etc/kolla/globals.yml
docker_registry: "4.0.0.10:4000"
docker_namespace: "lokolla"
openstack_release: "4.0.2"
kolla_install_type: "source"

搭建http代理服务器

在Deploy节点上搭建http服务器

# yum install squid -y
### 修改配置文件，删除默认自带的对IP和端口的限制，然后添加如下规则
# vim /etc/squid/squid.conf
http_access allow all
### 设置开机自启动
# systemctl restart squid
# systemctl enable squid
### 修改ansible脚本
# vim /usr/share/kolla-ansible/ansible/kolla-host.yml 
 environment:    https_proxy : http://4.0.0.10:3128/    http_proxy : http://4.0.0.10:3128/

修改Sched/Nova的环境的配置文件

### 设置yum代理
# vim /etc/yum.conf
proxy=http://4.0.0.10:3128
# yum makecache

修改部署文件

### Deploy节点上执行 # vim ~/multinode [control]
4.0.0.11

[network]
4.0.0.11

[compute]
4.0.0.12

[monitoring]
4.0.0.11

[storage]
4.0.0.12

### 配置Sched、Nova主机基础环境，安装必备软件包 # kolla-ansible -i multinode bootstrap-servers

部署openstack

# kolla-ansible prechecks -i multinode
# kolla-ansible deploy -i multinode

验证openstack安装

# kolla-ansible post-deploy -i multinode

OpenStack更新

### 修改镜像版本# vim /etc/kolla/globals.ymlopenstack_release: "4.0.3"
# kolla-ansible pull -i multinode# kolla-ansible upgrade -i multinode

环境还原

### 将删除所有容器和卷# kolla-ansible destroy -i multinode

可选功能部署

Ceph部署

修改配置文件

# vim multinode
[storage]
92.0.0.12

# vim /etc/kolla/globals.yml
enable_ceph: "yes"
enable_ceph_rgw: "yes"
enable_cinder: "yes"

### 也可以同时配置yes
glance_backend_file: "no"
glance_backend_ceph: "yes"

# mkdir -p /etc/kolla/config
# tee /etc/kolla/config/ceph.conf <<-'EOF'
[global]
osd pool default size = 1
osd pool default min size = 1
EOF

修改Nova节点

### 添加一块总线为IDE格式为qcow2的磁盘 # fdisk -l Disk /dev/sda: 10.7 GB, 10737418240 bytes, 20971520 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes # parted /dev/sda -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
# parted /dev/sda printModel: ATA QEMU HARDDISK (scsi) Disk /dev/sda: 10.GB Sector size (logical/physical): 12B/12B Partition Table: gpt Disk Flags: Number  Start   End     Size    File system  Name                      Flags 1      049kB  10.GB  10.GB               KOLLA_CEPH_OSD_BOOTSTRAP

问题处理

虚拟机创建失败

### 创建带volume的虚拟机提示"Block device mapping is invalid"
### 1.查看nova-compute日志:VolumeNotCreated: Volume f103b3f3-d0ff-4a2b-9a5e-4b7ea5a9abdc did not finish being created even after we waited 3 seconds or 2 attempts. And its status is error.
### 2.查看cinder-volume日志：Volume group "cinder-volumes" not found
### 打印vg列表发现的确没有"cinder-volumes"，这个卷是要手动建立的
# vgdisplay
# dd if=/dev/zero of=./disk.img count=4096 bs=1MB
### 查询未被占用的loop设备
# losetup -f
# losetup /dev/loop2 disk.img
# pvcreate /dev/loop2
# vgcreate cinder-volumes /dev/loop2
### 重启容器，然后查看cinder-volume日志是否成功找到"cinder-volumes"
# docker restart cinder-volumes

创建虚拟机失败

### 创建带volume的虚拟机提示"No valid host was found"
### 1.查看nova-compute日志: 'iscsiadm -m node -T iqn.2010-10.org.openstack:volume-bbbccab7-bdd7-4086-8d0e-e14898439131 -p 127.0.0.1:3260' failed
### 从日志看来是登入iscsi服务器失败，IP地址不对
# vim /etc/kolla/cinder-volume/cinder.confmy_ip = 4.0.0.12
# docker restart cinder-volumes

ovs命令无法正常执行

# ovs-appctl ofproto/trace br-tun dl_vlan=32017-06-05T08:37:32Z|00001|daemon_unix|WARN|/var/run/openvswitch/ovs-vswitchd.pid: open: No such file or directory ovs-appctl: cannot read pidfile "/var/run/openvswitch/ovs-vswitchd.pid" (No such file or directory)
### 添加启动参数# vim /etc/kolla/openvswitch-vswitchd/config.json--pidfile=/var/run/openvswitch/ovs-vswitchd.pid
# docker restart openvswitch_vswitchd

docker pull失败

# docker pull 92.0.0.10:4000/lokolla/centos-source-fluentd:4.0.2Pulling repository 92.0.0.10:4000/lokolla/centos-source-fluentd Network timed out while trying to connect to http://92.0.0.10:4000/v1/repositories/lokolla/centos-source-fluentd/images. You may want to check your internet connection or if you are behind a proxy.
### docker本地仓库所使用的docker版本太高，降低docker版本/本地docker-py版本太高/本地docker未开启--insecure-registry
# yum remove docker-engine
# yum install docker-engine-1.12.5 -y

Ceph安装报错

TASK [ceph : Fetching Ceph keyrings] ******************************************************************************************************************************************************** [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ (ceph_files_json.stdout | from_json).changed }} fatal: [92.0.0.11]: FAILED! => {"failed": true, "msg": "The conditional check '{{ (ceph_files_json.stdout | from_json).changed }}' failed. The error was: No JSON object could be decoded"}### 解决办法，先去对应节点删除ceph-mon容器，然后执行下面的命令# ansible -i multinode  -a 'docker volume rm ceph_mon_config' ceph-monTASK [ceph : include] *********************************************************************************************************************************************************************** included: /usr/share/kolla-ansible/ansible/roles/ceph/tasks/generate_cluster.yml for 92.0.0.11 TASK [ceph : Generating initial Ceph keyrings and monmap] *********************************************************************************************************************************** fatal: [92.0.0.11]: FAILED! => {"changed": true, "failed": true, "msg": "Container exited with non-zero return code"} TASK [ceph : include] *********************************************************************************************************************************************************************** included: /usr/share/kolla-ansible/ansible/roles/ceph/tasks/distribute_keyrings.yml for 92.0.0.12 TASK [ceph : Fetching Ceph keyrings] ******************************************************************************************************************************************************** fatal: [92.0.0.12]: FAILED! => {"failed": true, "msg": "'delegate_host' is undefined"}

多contoller部署创建虚拟机失败

### rabbitmq集群部署的配置文件使用的是主机名，所以各个主机的主机名不能相同

等待virtual ip超时

### 1. 确保kolla_internal_vip_address没有被占用
### 2. 修改keepalived_virtual_router_id防止其他kolla部署环境干扰
### 3. kolla_internal_vip_address或keepalived_virtual_router_id修改后需要重启keepalived容器，使配置重新生效
### 尝试一
# 修改globals.yml的keepalived_virtual_router_id
# kolla-ansible -i multinode deploy --tags="haproxy"

### 尝试二
# docker rm -f haproxy keepalived
# systemctl stop NetworkManager
# systemctl disable NetworkManager
# kolla-ansible -i multinode deploy --tags="haproxy"

baremetal安装docker-engine失败

### 在节点上手动安装，下载超时也会导致失败
# yum clean all
# yum makecache fast
# yum install docker-engine-1.12.0 -y

mariadb启动失败

[ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL) at gcomm/src/pc.cpp:PC():271
### 解决办法# rm -rf /var/lib/docker/volumes/mariadb/_data/gvwstate.dat
# docker restart mariadb

---

欢迎关注运维自研堂订阅号，运维自研堂是一个技术分享平台，主要是运维自动化开发：linux、python、django、saltstack、tornado、bootstrap、redis、golang、docker、etcd等经验分享。

开源    创新     共享

投稿&商务合作

Mail：[e-mail]       QQ:785249378

牛人并不可怕，可怕的是牛人比我们还努力！