50-Docker-分布式仓库Harbor高可用
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了50-Docker-分布式仓库Harbor高可用相关的知识，希望对你有一定的参考价值。
Harbor 介绍
- 基于角色的访问控制: 用户与Docker镜像仓库通过“项目”进行组织管理，一个用户可以对多个镜像仓库在同一命名空间（project）里有不同的权限
- 镜像复制: 镜像可在多个Registry实例中复制（同步）。尤其适合于负载均衡，高可用，混合云和多云的场景
- 图形化用户界面: 用户可以通过浏览器来浏览，检索当前Docker镜像仓库，管理项目和命名空间
- AD/LDAP 支: Harbor可以集成企业内部已有的AD/LDAP，用于鉴权认证管理
- 审计管理: 所有针对镜像仓 库的操作都可以被记录追溯，用于审计管理
- 国际化: 已拥有英文、中文、德文、日文和俄文的本地化版本。更多的语言将会添加进来
- RESTful API: 提供给管理员对于Harbor更多的操控, 使得与其它管理软件集成变得更容易
- 部署简单: 提供在线和离线两种安装工具， 也可以安装到vSphere平台(OVA方式)虚拟设备
harbor 官方github 地址: https://github.com/vmware/harbor
安装 Harbor
#下载离线完整安装包
https://github.com/goharbor/harbor/releases/download/v2.7.0/harbor-offline-installer-v2.7.0.tgz
#安装前确保docker和docker-compose安装完毕
[root@ubuntu2204 ~]#docker-compose version
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
[root@ubuntu2204 ~]#docker version
Client:
Version: 20.10.12
API version: 1.41
Go version: go1.17.3
Git commit: 20.10.12-0ubuntu4
Built: Mon Mar 7 17:10:06 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.17.3
Git commit: 20.10.12-0ubuntu4
Built: Mon Mar 7 15:57:50 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.9-0ubuntu3.1
GitCommit:
runc:
Version: 1.1.0-0ubuntu1.1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
[root@ubuntu2204 ~]#ll har*
-rw-r--r-- 1 root root 789527572 1月 13 14:09 harbor-offline-installer-v2.7.0.tgz
#解压缩离线包
[root@ubuntu2204 ~]#mkdir /apps
[root@ubuntu2204 ~]#tar xvf harbor-offline-installer-v2.7.0.tgz -C /apps/
harbor/harbor.v2.7.0.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
#编辑 harbor 配置文件
[root@ubuntu2204 apps]#cd /apps/harbor/
[root@ubuntu2204 harbor]#ls
common.sh harbor.v2.7.0.tar.gz harbor.yml.tmpl install.sh LICENSE prepare
[root@ubuntu2204 harbor]#mv harbor.yml.tmpl harbor.yml
[root@ubuntu2204 harbor]#vim harbor.yml
[root@ubuntu2204 harbor]#cat harbor.yml |grep hostname
# The IP address or hostname to access admin UI and registry service.
hostname: 10.0.0.200
# And when it enabled the hostname will no longer used
# # endpoint: http://hostname:14268/api/traces
# # agent_host: hostname
# # endpoint: hostname:4318
[root@ubuntu2204 harbor]#cat harbor.yml |grep harbor_admin_password
harbor_admin_password: 123456
*harbor.yml禁用了https.后面用到了再开启
#运行 harbor 安装脚本
注意：如果脚本运行失败，可利用当前目录下生成的docker-compose文件卸载容器，排除错误再运行
[root@ubuntu2204 harbor]#./install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 20.10.12
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.29.2
[Step 2]: loading Harbor images ...
Loaded image: goharbor/prepare:v2.7.0
Loaded image: goharbor/harbor-db:v2.7.0
Loaded image: goharbor/harbor-core:v2.7.0
Loaded image: goharbor/harbor-log:v2.7.0
Loaded image: goharbor/harbor-exporter:v2.7.0
Loaded image: goharbor/nginx-photon:v2.7.0
Loaded image: goharbor/chartmuseum-photon:v2.7.0
Loaded image: goharbor/harbor-portal:v2.7.0
Loaded image: goharbor/harbor-jobservice:v2.7.0
Loaded image: goharbor/harbor-registryctl:v2.7.0
Loaded image: goharbor/registry-photon:v2.7.0
Loaded image: goharbor/notary-server-photon:v2.7.0
Loaded image: goharbor/redis-photon:v2.7.0
Loaded image: goharbor/notary-signer-photon:v2.7.0
Loaded image: goharbor/trivy-adapter-photon:v2.7.0
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /apps/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/core/env
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Note: stopping existing Harbor instance ...
Removing network harbor_harbor
WARNING: Network harbor_harbor not found.
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating registryctl ... done
Creating registry ... done
Creating redis ... done
Creating harbor-db ... done
Creating harbor-core ... done
Creating nginx ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----
[root@ubuntu2204 harbor]#ss -nltp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.1:1514 0.0.0.0:* users:(("docker-proxy",pid=6471,fd=4))
LISTEN 0 4096 0.0.0.0:80 0.0.0.0:* users:(("docker-proxy",pid=7150,fd=4))
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=720,fd=14))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=818,fd=3))
LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* users:(("sshd",pid=921,fd=7))
LISTEN 0 4096 127.0.0.1:39647 0.0.0.0:* users:(("containerd",pid=769,fd=13))
LISTEN 0 4096 [::]:80 [::]:* users:(("docker-proxy",pid=7155,fd=4))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=818,fd=4))
LISTEN 0 128 [::1]:6010 [::]:* users:(("sshd",pid=921,fd=5))
[root@ubuntu2204 harbor]#docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a47c64904b16 goharbor/harbor-jobservice:v2.7.0 "/harbor/entrypoint.…" 9 minutes ago Up 9 minutes (healthy) harbor-jobservice
15a7d3807dae goharbor/nginx-photon:v2.7.0 "nginx -g daemon of…" 9 minutes ago Up 9 minutes (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp nginx
660cf0009cfb goharbor/harbor-core:v2.7.0 "/harbor/entrypoint.…" 9 minutes ago Up 9 minutes (healthy) harbor-core
731c9af81973 goharbor/harbor-db:v2.7.0 "/docker-entrypoint.…" 9 minutes ago Up 9 minutes (healthy) harbor-db
8c5705c268bc goharbor/redis-photon:v2.7.0 "redis-server /etc/r…" 9 minutes ago Up 9 minutes (healthy) redis
0fc108dd6a9a goharbor/registry-photon:v2.7.0 "/home/harbor/entryp…" 9 minutes ago Up 9 minutes (healthy) registry
6cb7a7983283 goharbor/harbor-registryctl:v2.7.0 "/home/harbor/start.…" 9 minutes ago Up 9 minutes (healthy) registryctl
f7267bd1c057 goharbor/harbor-portal:v2.7.0 "nginx -g daemon of…" 9 minutes ago Up 9 minutes (healthy) harbor-portal
f1689c0378e7 goharbor/harbor-log:v2.7.0 "/bin/sh -c /usr/loc…" 9 minutes ago Up 9 minutes (healthy) 127.0.0.1:1514->10514/tcp harbor-log
#测试
#通过service文件实现开机启动服务
[root@ubuntu2204 harbor]#vim /lib/systemd/system/harbor.service
[root@ubuntu2204 harbor]#cat /lib/systemd/system/harbor.service
[Unit]
Descriptinotallow=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentatinotallow=http://github.com/vmware/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f /apps/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /apps/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
[root@ubuntu2204 harbor]#systemctl daemon-reload
[root@ubuntu2204 harbor]#systemctl enable harbor
Created symlink /etc/systemd/system/multi-user.target.wants/harbor.service → /lib/systemd/system/harbor.service.
#测试
[root@ubuntu2204 harbor]#systemctl status harbor
○ harbor.service - Harbor
Loaded: loaded (/lib/systemd/system/harbor.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: http://github.com/vmware/harbor
[root@ubuntu2204 harbor]#reboot
[root@ubuntu2204 ~]#systemctl status harbor
● harbor.service - Harbor
Loaded: loaded (/lib/systemd/system/harbor.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-01-13 14:53:54 CST; 42min ago
Docs: http://github.com/vmware/harbor
Main PID: 1960 (docker-compose)
Tasks: 13 (limit: 3402)
Memory: 27.9M
CPU: 13.296s
CGroup: /system.slice/harbor.service
└─1960 /usr/bin/python3 /usr/bin/docker-compose -f /apps/harbor/docker-compose.yml up
1月 13 15:36:00 ubuntu2204.wang.org docker-compose[1960]: registryctl | 172.19.0.5 - - [13/Jan/2023:07:36:00 +0000] "GET /api/health HTTP/1.1" 200 9
1月 13 15:36:00 ubuntu2204.wang.org docker-compose[1960]: harbor-portal | 172.19.0.5 - - [13/Jan/2023:07:36:00 +0000] "GET / HTTP/1.1" 200 785 "-" "Go-http-client/1.1"
1月 13 15:36:02 ubuntu2204.wang.org docker-compose[1960]: registry | 127.0.0.1 - - [13/Jan/2023:07:36:02 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.86.0"
1月 13 15:36:03 ubuntu2204.wang.org docker-compose[1960]: harbor-portal | 127.0.0.1 - - [13/Jan/2023:07:36:03 +0000] "GET / HTTP/1.1" 200 785 "-" "curl/7.86.0"
1月 13 15:36:10 ubuntu2204.wang.org docker-compose[1960]: registry | 172.19.0.5 - - [13/Jan/2023:07:36:10 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
1月 13 15:36:10 ubuntu2204.wang.org docker-compose[1960]: registryctl | 172.19.0.5 - - [13/Jan/2023:07:36:10 +0000] "GET /api/health HTTP/1.1" 200 9
1月 13 15:36:10 ubuntu2204.wang.org docker-compose[1960]: harbor-portal | 172.19.0.5 - - [13/Jan/2023:07:36:10 +0000] "GET / HTTP/1.1" 200 785 "-" "Go-http-client/1.1"
1月 13 15:36:13 ubuntu2204.wang.org docker-compose[1960]: registryctl | 127.0.0.1 - - [13/Jan/2023:07:36:13 +0000] "GET /api/health HTTP/1.1" 200 9
1月 13 15:36:18 ubuntu2204.wang.org docker-compose[1960]: nginx | 127.0.0.1 - "GET / HTTP/1.1" 200 785 "-" "curl/7.86.0" 0.000 0.001 .
1月 13 15:36:18 ubuntu2204.wang.org docker-compose[1960]: harbor-portal | 172.19.0.2 - - [13/Jan/2023:07:36:18 +0000] "GET / HTTP/1.1" 200 785 "-" "curl/7.86.0"
使用单主机 Harbor
建立项目
命令行登录 harbor
[root@ubuntu2204 ~]#vim /etc/docker/daemon.json
[root@ubuntu2204 ~]#cat /etc/docker/daemon.json
...
"insecure-registry": [ "10.0.0.200","10.0.0.202"]
[root@ubuntu2204 ~]#systemctl daemon-reload
[root@ubuntu2204 ~]#systemctl restart docker
[root@ubuntu2204 ~]#docker login 10.0.0.200
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@ubuntu2204 ~]#cat /root/.docker/config.json
"auths":
"10.0.0.200":
"auth": "YWRtaW46MTIzNDU2"
,
"10.0.0.202":
"auth": "YWRtaW46MTIzNDU2"
,
"127.0.0.1":
"auth": "YWRtaW46MTIzNDU2"
,
"https://index.docker.io/v1/":
"auth": "bW9vcmV5eGlhOm1vb3JleXhpYS44ODEw"
[root@ubuntu2204 ~]#echo -e YWRtaW46MTIzNDU2|base64 -d
admin:123456
给本地镜像打标签并上传到 Harbor
#修改 images 的名称，不修改成指定格式无法将镜像上传到 harbor 仓库
[root@ubuntu2204 ~]#docker tag nginx-alpine:1.16.1 10.0.0.200/mooreyxia-1/nginx-alpine:1.16.1-1
[root@ubuntu2204 ~]#docker push 10.0.0.200/mooreyxia-1/nginx-alpine:1.16.1-1
The push refers to repository [10.0.0.200/mooreyxia-1/nginx-alpine]
f54ca93f29a8: Pushed
0273e525dd0a: Pushed
daa344f0fb22: Pushed
5ccc4c24bcac: Pushed
b3dd37fd4cfa: Pushed
e2dc414ff3be: Pushed
9fa45c5f1089: Mounted from mooreyxia-200/alpine-base
ded7a220bb05: Mounted from mooreyxia-200/alpine-base
1.16.1-1: digest: sha256:b80de5bd851ed0a162f273947e6aff0122e1757bc61a3323ab3551bc167929d0 size: 1996
访问harbor网站验证上传镜像成功
下载 Harbor 的镜像
#下载前必须修改docker的service 文件，加入harbor服务器的地址才可以下载
[root@ubuntu2204 ~]#vim /etc/docker/daemon.json
[root@ubuntu2204 ~]#cat /etc/docker/daemon.json
...
"insecure-registries": ["10.0.0.200", "10.0.0.202"]
[root@ubuntu2204 ~]#systemctl daemon-reload
[root@ubuntu2204 ~]#systemctl restart docker
[root@ubuntu2204 ~]#docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 6
Running: 4
Paused: 0
Stopped: 2
Images: 16
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.0-57-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.896GiB
Name: ubuntu2204.wang.org
ID: 2PI7:R6DF:7UAL:2JP4:NSW4:HEWM:SKRS:WTFS:FFCY:OGZH:2EQN:GK3M
Docker Root Dir: /data/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
10.0.0.200
10.0.0.202
127.0.0.0/8
Registry Mirrors:
https://registry.docker-cn.com/
http://hub-mirror.c.163.com/
https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: true
#从harbor下载镜像
[root@ubuntu2204 ~]#docker pull 10.0.0.200/mooreyxia-1/nginx-alpine:1.16.1-1
1.16.1-1: Pulling from mooreyxia-1/nginx-alpine
c158987b0551: Pull complete
b24b5e1a85db: Pull complete
09d00cadef1a: Pull complete
9cd63c39ff06: Pull complete
81b6b70fa169: Pull complete
8cb247876251: Pull complete
894ae90a2895: Pull complete
c17c8b0dae99: Pull complete
Digest: sha256:b80de5bd851ed0a162f273947e6aff0122e1757bc61a3323ab3551bc167929d0
Status: Downloaded newer image for 10.0.0.200/mooreyxia-1/nginx-alpine:1.16.1-1
10.0.0.200/mooreyxia-1/nginx-alpine:1.16.1-1
[root@ubuntu2204 ~]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
10.0.0.200/mooreyxia-1/nginx-alpine 1.16.1-1 34b3eb9eef98 3 days ago 274MB
实现 Harbor 高可用
- Harbor支持基于策略的Docker镜像复制功能
在第二台主机上安装部署好harbor，并登录系统
参考第一台harbor服务器的项目名称，在第二台harbor服务器上新建与之同名的项目
第二台harbor上新建复制规则实现到第一台harbor的单向推送复制
测试 202上传镜像，200同步
#202
[root@ubuntu2204 ~]#docker tag busybox:latest 10.0.0.202/mooreyxia-1/busybox:latest-1
[root@ubuntu2204 ~]#docker push 10.0.0.202/mooreyxia-1/busybox:latest-1
The push refers to repository [10.0.0.202/mooreyxia-1/busybox]
b64792c17e4a: Preparing
unauthorized: unauthorized to access repository: mooreyxia-1/busybox, action: push: unauthorized to access repository: mooreyxia-1/busybox, action: push
[root@ubuntu2204 ~]#docker login 10.0.0.202
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@ubuntu2204 ~]#docker push 10.0.0.202/mooreyxia-1/busybox:latest-1
The push refers to repository [10.0.0.202/mooreyxia-1/busybox]
b64792c17e4a: Pushed
latest-1: digest: sha256:907ca53d7e2947e849b839b1cd258c98fd3916c60f2e6e70c30edbf741ab6754 size: 528
查看200
注意：以上操作，只是实现了从第二台harbor主机10.0.0.202到第一台harbor主机10.0.200的单向同步，在200的主机harbor上再执行相同的推送操作，才实现双向同步
测试镜像删除是否同步
- 200删除，202观察是否也同步删除
删除后
配置 Nginx 做为反向代理
#配置Nginx反向代理
[root@ubuntu2004 ~]#cat /etc/nginx/conf.d/harbor.mooreyxia.org.conf
upstream harbor
ip_hash;
server harbor1.mooreyxia.org:80;
server harbor2.mooreyxia.org:80;
server
listen 80;
server_name harbor.mooreyxia.org;
client_max_body_size 10g;
location /
proxy_pass http://harbor;
#客户端docker配置
[root@rocky8 ~]#cat /etc/docker/daemon.json
"registry-mirrors": ["https://si7y70hh.mirror.aliyuncs.com"],
"insecure-registries": ["harbor.mooreyxia.org"]
[root@rocky8 ~]#systemctl restart docker
#客户端docker配置名称解析
[root@rocky8 ~]#vim /etc/hosts
10.0.0.100 harbor.mooreyxia.org
#如果harbor配置中的hostname: 指定harbor1.mooreyxia.org和harbor2.mooreyxia.org名称，还需要加下面解析
10.0.0.101 harbor1.mooreyxia.org
10.0.0.102 harbor2.mooreyxia.org
我是moore,大家一起加油！
以上是关于50-Docker-分布式仓库Harbor高可用的主要内容，如果未能解决你的问题，请参考以下文章