swarm 部署高可用harbor
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了swarm 部署高可用harbor相关的知识，希望对你有一定的参考价值。
harbor官方下载地址：
https://github.com/vmware/harbor/releases
配置参考：
https://github.com/vmware/harbor/blob/master/docs/installation_guide.md
此处使用离线下载安装包：
wget https://storage.googleapis.com/harbor-releases/release-1.5.0/harbor-offline-installer-v1.5.1.tgz (需要×××)
tar -zxvf harbor-offline-installer-v1.5.1.tgz
cd harbor
#Configure harbor.cfg
主要配置点：
hostname = xxx:8888
#如需要配置实现高可用，则需将registry 放到高可用存储节点， 以及adminserver 连接的mysql数据库、redis 配置为外置数据库。如不配置redis，则harbor节点异常，则docker client 需要进行重新登录。此处数据库均使用 ceph rbd 持久化存储。
修改完成后，执行install.sh ,由于官方默认是使用docker-compose 启动，执行脚本会默认检查相关组件，这里把如下3行注释，主要用install.sh 运行配置文件和加载镜像。
#docker-compose
#check_dockercompose
#check_docker
sh install.sh
执行完成后会在对应目录生成相关配置文件。看脚本你会发现，其实最终调用的是./prepare 脚本来生成配置文件等信息。我们不用compose，所以也可以直接使用prepare 生成配置文件。
修改docker-compose.yml文件，使用version 3 支持stack 部署。实例如下，这里去掉了log模块，直接使用docker service logs 查看日志更方便。
主要注意点：
为实现高可用，如下mysql、redis、registry 均使用的ceph rbd 存储。其他service使用mfs共享存储挂载。
version: '3'
services:
#log:
# image: vmware/harbor-log:v1.5.1
# volumes:
# - /var/log/:/var/log/docker/
# - /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/log/:/etc/logrotate.d/
# ports:
# - 1514:10514
# networks:
# - harbor
registry:
image: vmware/registry-photon:v2.6.2-v1.5.1
volumes:
- harbor_registry:/storage
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/registry/:/etc/registry/
networks:
- harbor
environment:
- GODEBUG=netdns=cgo
command:
["serve", "/etc/registry/config.yml"]
# depends_on:
# - log
# logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "registry"
mysql:
image: vmware/harbor-db:v1.5.1
volumes:
- harbor_mysql:/var/lib/mysql
networks:
- harbor
env_file:
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/db/env
# depends_on:
# - log
# logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "mysql"
adminserver:
image: vmware/harbor-adminserver:v1.5.1
env_file:
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/adminserver/env
volumes:
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/config/:/etc/adminserver/config/
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/secretkey:/etc/adminserver/key
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/data:/data/
networks:
- harbor
# depends_on:
# - log
# logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "adminserver"
ui:
image: vmware/harbor-ui:v1.5.1
env_file:
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/ui/env
volumes:
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/ui/app.conf:/etc/ui/app.conf
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/ui/private_key.pem:/etc/ui/private_key.pem
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/ui/certificates/:/etc/ui/certificates/
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/secretkey:/etc/ui/key
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/ca_download/:/etc/ui/ca/
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/psc/:/etc/ui/token/
networks:
- harbor
depends_on:
- adminserver
- registry
#logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "ui"
jobservice:
image: vmware/harbor-jobservice:v1.5.1
env_file:
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/jobservice/env
volumes:
- /mnt/mfs/docker/swarm_stack/stack/harbor/online/job_logs:/var/log/jobs
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/jobservice/config.yml:/etc/jobservice/config.yml
networks:
- harbor
depends_on:
- redis
- ui
- adminserver
#logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "jobservice"
redis:
image: vmware/redis-photon:v1.5.1
volumes:
- harbor_redis:/data
networks:
- harbor
#logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "redis"
proxy:
image: vmware/nginx-photon:v1.5.1
volumes:
- /mnt/mfs/docker/swarm_stack/stack/harbor/common/config/nginx:/etc/nginx
networks:
- harbor
ports:
- 8888:80
- 8889:443
- 4443:4443
depends_on:
- mysql
- registry
- ui
# logging:
# driver: "syslog"
# options:
# syslog-address: "tcp://log:10514"
# tag: "proxy"
networks:
harbor:
external: false
volumes:
harbor_registry:
driver: rbd
driver_opts:
size: 200000
harbor_mysql:
driver: rbd
driver_opts:
size: 100000
harbor_redis:
driver: rbd
driver_opts:
size: 10000
docker stack deploy -c docker-compose.yml harbor ，稍等数据库初始化，即可登录harbor使用。确保7个service 有7个稳定Running状态的容器。
[[email protected] harbor]# docker stack ps harbor |grep -i Runn
q6uw8um1tmwx harbor_proxy.1 vmware/nginx-photon:v1.5.1 docker27 Running Running 15 minutes ago
mkhg3eowzugn harbor_redis.1 vmware/redis-photon:v1.5.1 docker39 Running Running 16 minutes ago
v61q3gynk9c0 harbor_jobservice.1 vmware/harbor-jobservice:v1.5.1 docker75 Running Running 16 minutes ago
nry1l16apmpt harbor_ui.1 vmware/harbor-ui:v1.5.1 docker40 Running Running 15 minutes ago
f4qw8xfj4gxt harbor_adminserver.1 vmware/harbor-adminserver:v1.5.1 docker38 Running Running 15 minutes ago
wkjbcgd4evms harbor_mysql.1 vmware/harbor-db:v1.5.1 docker39 Running Running 16 minutes ago
1oyhjtv5d8b6 harbor_registry.1 vmware/registry-photon:v2.6.2-v1.5.1 docker27 Running Running 16 minutes ago
客户端首次使用registry需要做如下配置：
1、修改docker配置文件，增加 "insecure-registries":["10.211.121.26:8888"] 到 /etc/docker/daemon.json
systemctl reload docker
2、docker login 10.211.121.26:8888 -uadmin -pHarbor12345
登录后即可正常使用 docker push/pull 进行镜像推送和拉取。
高可用测试：
1、重启harbor_registry ，业务出现短暂失效，待容器重启成功后可继续使用。
2、重启harbor_proxy，业务出现短暂失效，待容器重启成功后可继续使用。
3、重启harbor_redis，业务出现短暂失效，重启成功后正常。(redis数据持久化在rbd中)
4、重启harbor_jobservice，无影响。
5、重启harbor_ui，无影响
6、重启harbor_adminserver，业务出现短暂失效，待容器重启成功后可继续使用。
7、重启harbor_mysql，业务出现短暂失效，待容器重启成功后可继续使用。
毁灭性测试：
docker stack rm harbor
sleep 10
docker stack deploy -c docker-compose.yml harbor
待2分钟左右所有业务容器启动成功，仓库恢复正常使用。
以上是关于swarm 部署高可用harbor的主要内容，如果未能解决你的问题，请参考以下文章