harbor基于keepalive高可用部署

Posted 2020-12-10

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了harbor基于keepalive高可用部署相关的知识，希望对你有一定的参考价值。

1.基础设置
1.1.拓扑图
2 技术图片
Harbor 1.4.0版本开始提供了HA部署方式，和非HA的主要区别就是把有状态的服务分离出来，使用外部集群，而不是运行在本地的容器上。而无状态的服务则可以部署在多个节点上，通过配置上层Load Balancer构成HA。

1.2.主机角色
主机角色 IP地址操作系统备注
harbor01 10.7.132.243 linux7 安装docker-ce、redis、postgresql、mysql、keepalived
harbor02 10.7.132.219 linux7 安装docker-ce、redis、postgresql、mysql、keepalived
vip 10.7.132.253 harbor01是keepalived的master节点
harbor02是keepalived的slave节点

1.3.软件包

名称版本备注
linux centos7 操作系统
keepalived 1.3.5 用于vip
docker-ce 18.09.6 运行镜像基础环境
docker-compose 1.21.2 用于编译yml
mariadb 10.2.24 存储registry
redis 3.2.1 存储消息缓存
postgresql 9.6 存储镜像扫描，功能未开启
harbor 1.5.4 基础包

1.4.hosts解析
cat >> /etc/hosts <<-‘EOF‘
10.7.132.243 harbor01
10.7.132.219 harbor02
10.7.132.253 reg.xgmin.com

EOF

1.5.指定yum源
harbor01和harbor02两台主机都需要操作
rm -f /etc/yum.repos.d/*.repo
wget -O /etc/yum.repos.d/centos7.repo http://mirror.XXXXXXX.com/other/repo/centos7.repo
wget -O /etc/yum.repos.d/docker-ce.repo http://mirror.XXXXXXX.com/other/repo/docker-ce.repo
wget -O /etc/yum.repos.d/epel7.repo http://mirror.XXXXXXX.com/other/repo/epel7.repo

yum clean all && yum repolist
yum install vim tree -y

1.6.建立证书
1.生成CA证书，只在harbor01上操作
mkdir /root/cert && cd /root/cert
openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 3650 -out ca.crt
2.生成证书签名请求
openssl req -newkey rsa:4096 -nodes -sha256 -keyout reg.xgmin.com.key -out reg.xgmin.com.csr
3.生成注册证书
openssl x509 -req -days 3650 -in reg.xgmin.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out reg.xgmin.com.crt
4.拷贝harbor01生成的证书到harbor02
scp -r /root/cert root@10.7.132.219:/root

1.7.替换防火墙
此步骤必须操作
1.将centos7默认的防火墙关闭，harbor01和harbor02都操作
systemctl stop firewalld
systemctl disable firewalld
2.在centos7安装iptables防火墙，harbor01和harbor02都操作
yum install iptables iptables-services -y
systemctl start iptables
systemctl status iptables
systemctl enable iptables.service
3.添加允许规则,harbor01,harbor02都执行
iptables -I INPUT -p tcp --dport 6379 -j ACCEPT

iptables -I INPUT -p tcp --dport 5432 -j ACCEPT

iptables -I INPUT -p tcp --dport 20197 -j ACCEPT

iptables -I INPUT -p tcp --dport 3306 -j ACCEPT

iptables -I INPUT -p vrrp -j ACCEPT

iptables -I INPUT -p tcp --dport 80 -j ACCEPT
iptables -I INPUT -p tcp --dport 443 -j ACCEPT

service iptables save
1.8.网络权限开通
由于harbor01与harbor02节点为测试环境，青云s3对象存储在生产环境。需要开通如下跨区网络访问权限

2.Docker配置
2.1.Pip源搭建
1.在harbor01和harbor02上安装pip源
yum -y install python-pip
2.在harbor01上安装pypiserver
tar xf pip-pkg.tar.gz
cd pip-pkg
pip install pypiserver-1.2.1-py2.py3-none-any.whl
3.启动pypiserver
cat > /root/pip.sh<<-‘EOF‘
#!/bin/bash
/usr/bin/pypi-server -p 20197 /root/pip-pkg/ >/dev/null 2>&1 &
EOF

sh pip.sh
4.pip加速设置，harbor01使用本地的pip源，harbor02只需要执行此步骤
mkdir ~/.pip
cat > ~/.pip/pip.conf <<-‘EOF‘
[global]
index-url = http://10.7.132.243:20197/simple/
trusted-host = 10.7.132.243
EOF

pip install --upgrade pip
pip install docker-compose

2.2.Docker安装
1.本地安装docker，harbor01和harbor02都需要执行此步骤
yum install -y yum-utils device-mapper-persistent-data lvm2
yum -y install docker-ce
2.拷贝自定义的证书到dcker目录，如果以后在其它节点docker login 需要拷贝这个证书
mkdir -p /etc/docker/certs.d/reg.xgmin.com
cp -a /root/cert/reg.xgmin.com.crt /etc/docker/certs.d/reg.xgmin.com/
ll /etc/docker/certs.d/reg.xgmin.com/
3.启动docker并验证
systemctl restart docker
systemctl status docker
systemctl enable docker
docker info
3.Redis配置
3.1. harbor01节点
yum install redis -y
cat >/etc/redis.conf <<-‘EOF‘
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile /var/log/redis/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
EOF

systemctl restart redis
systemctl enable redis
3.2.harbor02节点
yum install redis -y
cat > /etc/redis.conf<<-‘EOF‘
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile /var/log/redis/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis
slaveof 10.7.132.243 6379
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

EOF
systemctl restart redis
systemctl enable redis

3.3.主从验证
1.在harbor01和harbor02上登录查看
redis-cli -h 127.0.0.1
info Replication
2.检查harbor01节点为

Replication

role:master
connected_slaves:1
slave0:ip=10.7.132.219,port=6379,state=online,offset=15,lag=0
master_repl_offset:15
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:14
3.检查harbor02节点为

Replication

role:slave
master_host:10.7.132.243
master_port:6379
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_repl_offset:43
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
4.harbor01节点创建keyharbor02节点查看
keys
set name xgmin
keys
4.MariaDB配置
4.1.harbor01节点
1.安装软件包
tar xf /root/mariadb-10.2.24-linux-systemd-x86_64.tar.gz -C /usr/local/
useradd -s /sbin/nologin -M mysql
ln -s /usr/local/mariadb-10.2.24-linux-systemd-x86_64 /usr/local/mariadb
mkdir -p /usr/local/mariadb/data
mkdir -p /usr/local/mariadb/data/mysqlbinlog
chown -R mysql:mysql /usr/local/mariadb /usr/local/mariadb-10.2.24-linux-systemd-x86_64

初始化

/usr/local/mariadb/scripts/mysql_install_db --basedir=/usr/local/mariadb
--datadir=/usr/local/mariadb/data
--user=mysql
cp /usr/local/mariadb/support-files/mysql.server /usr/lib/systemd/system/

vim /usr/lib/systemd/system/mysql.server

If you change base dir, you must also change datadir. These may get

overwritten by settings in the MariaDB configuration files.

basedir=/usr/local/mariadb
datadir=/usr/local/mariadb/data
2.配置数据库
cat > /etc/my.cnf <<-‘EOF‘

The following options will be passed to all MySQL clients

[client]
#password = your_password
port = 3306
socket = /tmp/mysql.sock

The MySQL server

[mysqld]
user = mysql
datadir = /usr/local/mariadb/data
port = 3306
pid-file = /tmp/mysql.pid
socket = /tmp/mysql.sock
log-bin = /usr/local/mariadb/data/mysqlbinlog
binlog_format = ROW
slow-query-log = 1
slow-query-log-file = /usr/local/mariadb/logs/mysql_slow.log
default_storage_engine = InnoDB
skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 1M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
thread_concurrency = 8

server-id = 1
innodb_log_file_size = 100M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
log-error=/usr/local/mariadb/logs/mysqld.log

4.2.harbor02节点
1.安装软件包
tar xf /root/mariadb-10.2.24-linux-systemd-x86_64.tar.gz -C /usr/local/
useradd -s /sbin/nologin -M mysql
ln -s /usr/local/mariadb-10.2.24-linux-systemd-x86_64 /usr/local/mariadb
mkdir -p /usr/local/mariadb/data
mkdir -p /usr/local/mariadb/data/mysqlbinlog
chown -R mysql:mysql /usr/local/mariadb /usr/local/mariadb-10.2.24-linux-systemd-x86_64

初始化

/usr/local/mariadb/scripts/mysql_install_db --basedir=/usr/local/mariadb
--datadir=/usr/local/mariadb/data
--user=mysql
cp /usr/local/mariadb/support-files/mysql.server /usr/lib/systemd/system/

sed -i ‘45s#basedir=#basedir=/usr/local/mariadb#g‘ /usr/lib/systemd/system/mysql.server
sed -i ‘46s#datadir=#datadir=/usr/local/mariadb/data#g‘ /usr/lib/systemd/system/mysql.server
2.配置数据库
cat > /etc/my.cnf <<-‘EOF‘

The following options will be passed to all MySQL clients

[client]
#password = your_password
port = 3306
socket = /tmp/mysql.sock

The MySQL server

server-id = 2
innodb_log_file_size = 100M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
log-error=/usr/local/mariadb/logs/mysqld.log

EOF
3.启动服务
systemctl daemon-reload
/usr/lib/systemd/system/mysql.server start
netstat -lntup | grep 3306
4.数据库初始化
/usr/local/mariadb/bin/mysql_secure_installation --basedir=/usr/local/mariadb/
ln -s /usr/local/mariadb/bin/mysql /usr/local/bin/mysql
5.给最大权限，并可以远程连接
mysql -uroot -p‘Wab1IJvdHurMbPUp‘
grant all privileges on . to root@"%" identified by "Wab1IJvdHurMbPUp";
6.在harbor02的mysql终端中输入master的信息
CHANGE MASTER TO
MASTER_HOST=‘10.7.132.243‘,
MASTER_PORT=3306,
MASTER_USER=‘rep‘,
MASTER_PASSWORD=‘UUqBT7LwgqULoq6g‘,
MASTER_LOG_FILE=‘mysqlbinlog.000001‘,
MASTER_LOG_POS=16281;
7.启动主从同步进程
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)
4.3.主从验证
查看主从同步状态
MariaDB [(none)]> show slave status G
1. row
Slave_IO_State: Waiting for master to send event
Master_Host: 10.7.132.243
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysqlbinlog.000001
Read_Master_Log_Pos: 16281
Relay_Log_File: mysql-relay-bin.000002
Relay_Log_Pos: 557
Relay_Master_Log_File: mysqlbinlog.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 16281
Relay_Log_Space: 866
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: conservative
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
1 row in set (0.00 sec)
5.PostgreSQL配置
5.1.harbor01节点
1.安装并启动并初始化
cd ~/postgresql96
rpm -ivh *.rpm
/usr/pgsql-9.6/bin/postgresql96-setup initdb
systemctl start postgresql-9.6
systemctl enable postgresql-9.6
2.设置密码，添加同步的用户
su - postgres
psql -U postgres
ALTER USER postgres with encrypted password ‘bXTCUL5BIz5a4liM‘;

CREATE ROLE replica login replication encrypted password ‘V7MqYaP9ogzgAfad‘;

#输入q并回车，退出数据库。
#输入exit并回车，退出用户。
3.修改postgresql.conf，监听任意端口
vim /var/lib/pgsql/9.6/data/postgresql.conf
listen_addresses = ‘*‘ # what IP address(es) to listen on

wal_level = hot_standby
max_wal_senders = 32
wal_keep_segments = 256
wal_sender_timeout = 60s
max_connections = 100
4.允许指定的远程访问
vim /var/lib/pgsql/9.6/data/pg_hba.conf

IPv4 local connections:

host all all 127.0.0.1/32 ident
host all all 0.0.0.0/0 md5
host replication replica 10.7.132.219/32 md5
5.重启master节点
systemctl restart postgresql-9.6.service
5.2.harbor02节点
1.安装并启动并初始化
cd ~/postgresql96
rpm -ivh *.rpm
/usr/pgsql-9.6/bin/postgresql96-setup initdb
systemctl start postgresql-9.6
systemctl enable postgresql-9.6
2.设置密码，添加同步的用户
su - postgres
psql -U postgres
ALTER USER postgres with encrypted password ‘bXTCUL5BIz5a4liM‘;

#输入q并回车，退出数据库。
#输入exit并回车，退出用户。
3.修改postgresql.conf，监听任意端口
vim /var/lib/pgsql/9.6/data/postgresql.conf
将wal_level，max_wal_senders与wal_keep_segments等参数注释
listen_addresses = ‘*‘ # what IP address(es) to listen on

#在备份的同时允许查询
hot_standby = on

#可选，流复制最大延迟
max_standby_streaming_delay = 30s

#可选，从向主报告状态的最大间隔时间
wal_receiver_status_interval = 10s

#可选，查询冲突时向主反馈
hot_standby_feedback = on

#默认参数，非主从配置相关参数，表示到数据库的连接数，一般从库做主要的读服务时，设置值需要高于主库
max_connections = 1000
4.允许指定的远程访问
vim /var/lib/pgsql/9.6/data/pg_hba.conf

IPv4 local connections:

host all all 127.0.0.1/32 ident
host all all 0.0.0.0/0 md5
5.重启master节点
systemctl restart postgresql-9.6.service
6.从库安装postgresql后，暂不初始化，如果从库已初始化，可以清空其data目录（默认安装是/ /var/lib/pgsql/9.6/data/目录）。
[root@harbor02 ~]# cd /var/lib/pgsql/9.6/data/
[root@harbor02 data]# rm -rf *
[root@harbor02 data]# pg_basebackup -h 10.7.132.243 -p 5432 -U replica -F p -P -D /var/lib/pgsql/9.6/data/
口令: V7MqYaP9ogzgAfad
22853/22853 kB (100%), 1/1 表空间
NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup

#-h，主库主机，-p，主库服务端口；
#-U，复制用户；
#-F，p是默认输出格式，输出数据目录和表空间相同的布局，t表示tar格式输出；
#-P，同--progress，显示进度；
#-D，输出到指定目录；
#因为主库采用的是md5认证，这里需要密码认证。

chown -R postgres:postgres /var/lib/pgsql/9.6/data/
7.恢复文件recovery.conf
cp /usr/pgsql-9.6/share/recovery.conf.sample /var/lib/pgsql/9.6/data/recovery.conf
chown postgres:postgres /var/lib/pgsql/9.6/data/recovery.conf

vim /var/lib/pgsql/9.6/data/recovery.conf
#指明从库身份
standby_mode = on

#连接到主库信息
primary_conninfo = ‘host=10.7.132.243 port=5432 user=replica password=V7MqYaP9ogzgAfad‘

#同步到最新数据
recovery_target_timeline = ‘latest‘

5.3.主从验证
1.主库sender进程
[root@harbor01 ~]# ps -ef | grep postgres
root 19734 4368 0 21:50 pts/0 00:00:00 su - postgres
postgres 19735 19734 0 21:50 pts/0 00:00:00 -bash
postgres 19777 19735 0 21:51 pts/0 00:00:00 psql -U postgres
postgres 25118 1 0 22:11 ? 00:00:00 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres 25120 25118 0 22:11 ? 00:00:00 postgres: logger process
postgres 25122 25118 0 22:11 ? 00:00:00 postgres: checkpointer process
postgres 25123 25118 0 22:11 ? 00:00:00 postgres: writer process
postgres 25124 25118 0 22:11 ? 00:00:00 postgres: wal writer process
postgres 25125 25118 0 22:11 ? 00:00:00 postgres: autovacuum launcher process
postgres 25126 25118 0 22:11 ? 00:00:00 postgres: stats collector process
postgres 26927 25118 0 22:17 ? 00:00:00 postgres: wal sender process replica 10.7.132.219(45796) streaming 0/50003A8
root 27277 25001 0 22:18 pts/1 00:00:00 grep --color=auto postgres
2.从库receiver过程
[root@harbor02 ~]# ps -ef | grep postgres
postgres 31852 1 0 22:17 ? 00:00:00 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres 31854 31852 0 22:17 ? 00:00:00 postgres: logger process
postgres 31855 31852 0 22:17 ? 00:00:00 postgres: startup process recovering 000000010000000000000005
postgres 31856 31852 0 22:17 ? 00:00:00 postgres: wal receiver process streaming 0/50003A8
postgres 31857 31852 0 22:17 ? 00:00:00 postgres: checkpointer process
postgres 31858 31852 0 22:17 ? 00:00:00 postgres: writer process
postgres 31859 31852 0 22:17 ? 00:00:00 postgres: stats collector process
root 32440 26384 0 22:19 pts/0 00:00:00 grep --color=auto postgres
3.查看复制状态（主库）
[root@harbor01 ~]# su - postgres
上一次登录：四 8月 1 21:50:58 CST 2019pts/0 上
-bash-4.2$ psql -U postgres
psql (9.6.14)
Type "help" for help.

以上是关于harbor基于keepalive高可用部署的主要内容，如果未能解决你的问题，请参考以下文章

Harbor高可用集群设计及部署（实操+视频），基于离线安装方式

基于共享存储的Harbor高可用-Docker部署方案

swarm 部署高可用harbor

镜像仓库Harbor私服高可用策略分析及部署

k8s部署-19-harbor高可用部署