OpenShift 容器平台社区版 OKD 4.10.0部署

Posted 2022-04-08 willblog

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了OpenShift 容器平台社区版 OKD 4.10.0部署相关的知识，希望对你有一定的参考价值。

OpenShift简介

RedHat OpenShift 是一个领先的企业级 Kubernetes 容器平台，它为本地、混合和多云部署提供了基础。通过自动化运营和简化的生命周期管理，OpenShift 使开发团队能够构建和部署新的应用程序，并帮助运营团队配置、管理和扩展 Kubernetes 平台，OpenShift 还提供了一个CLI，该CLI支持Kubernetes CLI提供的操作的超集。

OpenShift有多个版本，两个主要版本：

红帽OpenShift的开源社区版本称为OKD（The Origin Community Distribution of Kubernetes，或OpenShift Kubernetes Distribution的缩写，原名OpenShiftOrigin），是 Red Hat OpenShift Container Platform (OCP) 的上游和社区支持版本。
红帽OpenShift的企业版本称为OCP（Red Hat OpenShift Container Platform ），OpenShift 的私有云产品，不购买订阅也可以安装使用，只是不提供技术支持。

OpenShift安装方式分为以下两种：

IPI(Installer Provisioned Infrastructure)方式：安装程序配置的基础架构集群，基础架构引导和配置委托给安装程序，而不是自己进行。安装程序会创建支持集群所需的所有网络、机器和操作系统。
UPI(User Provisioned Infrastructure)方式：用户配置的基础架构集群，必须由用户自行提供所有集群基础架构和资源，包括引导节点、网络、负载均衡、存储和集群的每个节点。

本文基于VMware vSphere7.0.3环境创建多个虚拟机，并在虚拟机上使用UPI模式手动部署OpenShift OKD 4.10版本集群，即官方介绍的Bare Metal (UPI)模式。

安装架构示意图:

安装流程示意图：

OKD社区版安装

官方文档参考：https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html

备注：本篇文章大多内容出自官方文档示例。

集群基本信息

集群名称：okd4
基本域名：example.com
集群规格：3个maste节点，2个worker节点

节点配置清单：

前期只需创建一个bastion节点，在bastion节点准备就绪后，其他节点需要逐个手动引导启动，无需提前创建。

Hostname	FQDN	IPaddress	NodeType	CPU	Mem	Disk	OS
bastion	bastion.okd4.example.com	192.168.72.20	基础节点	2C	4G	100G	Ubuntu 20.04.4 LTS
bootstrap	bootstrap.okd4.example.com	192.168.72.21	引导节点	4C	16G	100G	Fedora CoreOS 35
master0	master0.okd4.example.com	192.168.72.22	主控节点	4C	16G	100G	Fedora CoreOS 35
master1	master1.okd4.example.com	192.168.72.23	主控节点	4C	16G	100G	Fedora CoreOS 35
master2	master2.okd4.example.com	192.168.72.24	主控节点	4C	16G	100G	Fedora CoreOS 35
worker0	worker0.okd4.example.com	192.168.72.25	工作节点	2C	8G	100G	Fedora CoreOS 35
worker1	worker1.okd4.example.com	192.168.72.26	工作节点	2C	8G	100G	Fedora CoreOS 35
api server	api.okd4.example.com	192.168.72.20	Kubernetes API
api-int	api-int.okd4.example.com	192.168.72.20	Kubernetes API
apps	*.apps.okd4.example.com	192.168.72.20	Apps
registry	registry.example.com	192.168.72.20	镜像仓库

节点类型介绍：

Bastion节点，基础节点或堡垒机节点，提供http服务和registry的本地安装仓库服务，同时所有的ign点火文件，coreos所需要的ssh-rsa密钥等都由这个节点生成，OS类型可以任意。
Bootstrap节点，引导节点，引导工作完成后续可以删除，OS类型必须为Fedora CoreOS
Master节点，openshift的管理节点，操作系统必须为Fedora CoreOS
Worker节点，openshift的工作节点，操作系统可以在 Fedora CoreOS、Fedora 8.4 或 Fedora 8.5 之间进行选择。

bastion节点需要安装以下组件：

组件名称	组件说明
Docker	容器环境
Bind9	DNS服务器
Haproxy	负载均衡服务器
nginx	Web服务器
Harbor	容器镜像仓库
OpenShift CLI	oc命令行客户端
OpenShift-Install	openshift安装程序

部署完成后的基础资源信息：

部署完成后的openshift节点信息：

Bastion环境准备

首先创建一台Bastion 节点，配置静态IP地址，作为基础部署节点，操作系统类型没有要求，这里使用ubuntu，无特殊说明以下所有操作在该节点执行。

1、修改主机名

hostnamectl set-hostname bastion.okd4.example.com

2、安装docker

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
systemctl status docker
docker version

3、查看节点ip信息

root@bastion:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:99:0d:57 brd ff:ff:ff:ff:ff:ff
    inet 192.168.72.20/24 brd 192.168.72.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe99:d57/64 scope link 
       valid_lft forever preferred_lft forever

4、查看节点OS信息

root@bastion:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Bind安装

在 OKD 部署中，以下组件需要 DNS 名称解析：

Kubernetes API
OKD 应用访问入口
引导节点、控制平面和计算节点

Kubernetes API、引导机器、控制平面机器和计算节点也需要反向 DNS 解析。DNS A/AAAA 或 CNAME 记录用于名称解析，PTR 记录用于反向名称解析。反向记录很重要，因为 Fedora CoreOS (FCOS) 使用反向记录来设置所有节点的主机名，除非主机名由 DHCP 提供。此外，反向记录用于生成 OKD 需要操作的证书签名请求 (CSR)。

在每条记录中，<cluster_name>是集群名称，并且<base_domain>是在install-config.yaml文件中指定的基本域。完整的 DNS 记录采用以下形式：<component>.<cluster_name>.<base_domain>..

1、创建bind配置文件目录

mkdir -p /etc/bind
mkdir -p /var/lib/bind
mkdir -p /var/cache/bind

2、创建bind主配置文件

options 
        directory "/var/cache/bind";
        listen-on  any; ;
        listen-on-v6  any; ;
        allow-query  any; ;
        allow-query-cache  any; ;
        recursion yes;
        allow-recursion  any; ;
        allow-transfer  none; ;
        allow-update  none; ;
        auth-nxdomain no;
        dnssec-validation no;
        forward first;
        forwarders 
          114.114.114.114;
          8.8.8.8;
        ;
;

zone "example.com" IN 
  type master;
  file "/var/lib/bind/example.com.zone";
;

zone "72.168.192.in-addr.arpa" IN 
  type master;
  file "/var/lib/bind/72.168.192.in-addr.arpa";
;
EOF

4、创建正向解析配置文件

cat >/var/lib/bind/example.com.zone<<'EOF'
$TTL 1W
@   IN    SOA    ns1.example.com.    root (
                 2019070700        ; serial
                 3H                ; refresh (3 hours)
                 30M               ; retry (30 minutes)
                 2W                ; expiry (2 weeks)
                 1W )              ; minimum (1 week)
    IN    NS     ns1.example.com.
    IN    MX 10  smtp.example.com.
;
ns1.example.com.            IN A 192.168.72.20
smtp.example.com.           IN A 192.168.72.20
;
registry.example.com.       IN A 192.168.72.20
api.okd4.example.com.       IN A 192.168.72.20
api-int.okd4.example.com.   IN A 192.168.72.20
;
*.apps.okd4.example.com.    IN A 192.168.72.20
;
bastion.okd4.example.com.   IN A 192.168.72.20
bootstrap.okd4.example.com. IN A 192.168.72.21
;
master0.okd4.example.com.   IN A 192.168.72.22
master1.okd4.example.com.   IN A 192.168.72.23
master2.okd4.example.com.   IN A 192.168.72.24
;
worker0.okd4.example.com.   IN A 192.168.72.25
worker1.okd4.example.com.   IN A 192.168.72.26
EOF

5、创建反向解析配置文件

cat >/var/lib/bind/72.168.192.in-addr.arpa<<'EOF'
$TTL 1W
@   IN    SOA      ns1.example.com.     root (
                   2019070700        ; serial
                   3H                ; refresh (3 hours)
                   30M               ; retry (30 minutes)
                   2W                ; expiry (2 weeks)
                   1W )              ; minimum (1 week)
    IN    NS       ns1.example.com.
;
20.72.168.192.in-addr.arpa. IN PTR api.okd4.example.com.
20.72.168.192.in-addr.arpa. IN PTR api-int.okd4.example.com.
;
20.72.168.192.in-addr.arpa. IN PTR bastion.okd4.example.com.

21.72.168.192.in-addr.arpa. IN PTR bootstrap.okd4.example.com.
;
22.72.168.192.in-addr.arpa. IN PTR master0.okd4.example.com.
23.72.168.192.in-addr.arpa. IN PTR master1.okd4.example.com.
24.72.168.192.in-addr.arpa. IN PTR master2.okd4.example.com.
;
25.72.168.192.in-addr.arpa. IN PTR worker0.okd4.example.com.
26.72.168.192.in-addr.arpa. IN PTR worker1.okd4.example.com.
EOF

配置文件权限，允许容器有读写权限

chmod -R a+rwx /etc/bind
chmod -R a+rwx /var/lib/bind/
chmod -R a+rwx /var/cache/bind/

6、ubuntu中的dns由systemd-resolved管理，修改以下配置项，指定dns为本地DNS：

root@ubuntu:~# cat /etc/systemd/resolved.conf 
[Resolve]
DNS=192.168.72.20

重启systemd-resolved服务

systemctl restart systemd-resolved.service

创建到resolv.conf的链接：

ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

查看resolv.conf配置，确认输出内容如下：

root@ubuntu:~# cat /etc/resolv.conf 
......
# operation for /etc/resolv.conf.

nameserver 192.168.72.20
nameserver 114.114.114.114

7、以容器方式启动bind服务，注意绑定到本机IP，以免与ubuntu默认dns服务53端口冲突：

docker run -d --name bind9 \\
  --restart always \\
  --name=bind9 \\
  -e TZ=Asia/Shanghai \\
  --publish 192.168.72.20:53:53/udp \\
  --publish 192.168.72.20:53:53/tcp \\
  --publish 192.168.72.20:953:953/tcp \\
  --volume /etc/bind:/etc/bind \\
  --volume /var/cache/bind:/var/cache/bind \\
  --volume /var/lib/bind:/var/lib/bind \\
  --volume /var/log/bind:/var/log \\
  internetsystemsconsortium/bind9:9.18

8、使用dig命令来验证正向域名解析

dig +noall +answer @192.168.72.20 registry.example.com
dig +noall +answer @192.168.72.20 api.okd4.example.com
dig +noall +answer @192.168.72.20 api-int.okd4.example.com
dig +noall +answer @192.168.72.20 console-openshift-console.apps.okd4.example.com
dig +noall +answer @192.168.72.20 bootstrap.okd4.example.com
dig +noall +answer @192.168.72.20 master0.okd4.example.com
dig +noall +answer @192.168.72.20 master1.okd4.example.com
dig +noall +answer @192.168.72.20 master2.okd4.example.com
dig +noall +answer @192.168.72.20 worker0.okd4.example.com
dig +noall +answer @192.168.72.20 worker1.okd4.example.com

正向解析结果如下，确认每一项都能够正常解析

root@bastion:~# dig +noall +answer @192.168.72.20 registry.example.com
registry.example.com.   604800  IN      A       192.168.72.20
root@bastion:~# dig +noall +answer @192.168.72.20 api.okd4.example.com
api.okd4.example.com.   604800  IN      A       192.168.72.20
root@bastion:~# dig +noall +answer @192.168.72.20 api-int.okd4.example.com
api-int.okd4.example.com. 604800 IN     A       192.168.72.20
root@bastion:~# dig +noall +answer @192.168.72.20 console-openshift-console.apps.okd4.example.com
console-openshift-console.apps.okd4.example.com. 604800 IN A 192.168.72.20
root@bastion:~# dig +noall +answer @192.168.72.20 bootstrap.okd4.example.com
bootstrap.okd4.example.com. 604800 IN   A       192.168.72.21
root@bastion:~# dig +noall +answer @192.168.72.20 master0.okd4.example.com
master0.okd4.example.com. 604800 IN     A       192.168.72.22
root@bastion:~# dig +noall +answer @192.168.72.20 master1.okd4.example.com
master1.okd4.example.com. 604800 IN     A       192.168.72.23
root@bastion:~# dig +noall +answer @192.168.72.20 master2.okd4.example.com
master2.okd4.example.com. 604800 IN     A       192.168.72.24
root@bastion:~# dig +noall +answer @192.168.72.20 worker0.okd4.example.com
worker0.okd4.example.com. 604800 IN     A       192.168.72.25
root@bastion:~# dig +noall +answer @192.168.72.20 worker1.okd4.example.com
worker1.okd4.example.com. 604800 IN     A       192.168.72.26

验证反向域名解析

dig +noall +answer @192.168.72.20 -x 192.168.72.21
dig +noall +answer @192.168.72.20 -x 192.168.72.22
dig +noall +answer @192.168.72.20 -x 192.168.72.23
dig +noall +answer @192.168.72.20 -x 192.168.72.24
dig +noall +answer @192.168.72.20 -x 192.168.72.25
dig +noall +answer @192.168.72.20 -x 192.168.72.26

反向解析结果如下，同样需要确认每一项都能够正常解析

root@bastion:~# dig +noall +answer @192.168.72.20 -x 192.168.72.21
21.72.168.192.in-addr.arpa. 604800 IN   PTR     bootstrap.okd4.example.com.
root@bastion:~# dig +noall +answer @192.168.72.20 -x 192.168.72.22
22.72.168.192.in-addr.arpa. 604800 IN   PTR     master0.okd4.example.com.
root@bastion:~# dig +noall +answer @192.168.72.20 -x 192.168.72.23
23.72.168.192.in-addr.arpa. 604800 IN   PTR     master1.okd4.example.com.
root@bastion:~# dig +noall +answer @192.168.72.20 -x 192.168.72.24
24.72.168.192.in-addr.arpa. 604800 IN   PTR     master2.okd4.example.com.
root@bastion:~# dig +noall +answer @192.168.72.20 -x 192.168.72.25
25.72.168.192.in-addr.arpa. 604800 IN   PTR     worker0.okd4.example.com.
root@bastion:~# dig +noall +answer @192.168.72.20 -x 192.168.72.26
26.72.168.192.in-addr.arpa. 604800 IN   PTR     worker1.okd4.example.com.

安装Haproxy

使用haproxy创建负载均衡器，负载machine-config、kube-apiserver和集群ingress controller。

1、创建haproxy配置目录

mkdir -p /etc/haproxy

2、创建haproxy配置文件

cat >/etc/haproxy/haproxy.cfg<<EOF
global
  log         127.0.0.1 local2
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000
frontend stats
  bind *:1936
  mode            http
  log             global
  maxconn 10
  stats enable
  stats hide-version
  stats refresh 30s
  stats show-node
  stats show-desc Stats for openshift cluster 
  stats auth admin:openshift
  stats uri /stats

frontend openshift-api-server
    bind *:6443
    default_backend openshift-api-server
    mode tcp
    option tcplog
backend openshift-api-server
    balance source
    mode tcp
    server bootstrap 192.168.72.21:6443 check 
    server master0 192.168.72.22:6443 check
    server master1 192.168.72.23:6443 check
    server master2 192.168.72.24:6443 check
frontend machine-config-server
    bind *:22623
    default_backend machine-config-server
    mode tcp
    option tcplog
backend machine-config-server
    balance source
    mode tcp
    server bootstrap 192.168.72.21:22623 check
    server master0 192.168.72.22:22623 check
    server master1 192.168.72.23:22623 check
    server master2 192.168.72.24:22623 check
frontend ingress-http
    bind *:80
    default_backend ingress-http
    mode tcp
    option tcplog
backend ingress-http
    balance source
    mode tcp
    server worker0 192.168.72.25:80 check
    server worker1 192.168.72.26:80 check
frontend ingress-https
    bind *:443
    default_backend ingress-https
    mode tcp
    option tcplog
backend ingress-https
    balance source
    mode tcp
    server worker0 192.168.72.25:443 check
    server worker1 192.168.72.26:443 check
EOF

以容器方式启动haproxy服务

docker run -d --name haproxy \\
  --restart always \\
  -p 1936:1936 \\
  -p 6443:6443 \\
  -p 22623:22623 \\
  -p 80:80 -p 443:443 \\
  --sysctl net.ipv4.ip_unprivileged_port_start=0 \\
  -v /etc/haproxy/:/usr/local/etc/haproxy:ro \\
  haproxy:2.5.5-alpine3.15

安装Nginx

OpenShift 集群部署时需要从 web服务器下载 CoreOS Image 和 Ignition 文件，这里使用nginx提供文件下载。

1、创建nginx相关目录

mkdir -p /etc/nginx/templates
mkdir -p /usr/share/nginx/html/ignition,install

2、创建nginx配置文件，打开目录浏览功能（可选）

cat >/etc/nginx/templates/default.conf.template<<EOF
server 
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    location / 
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        autoindex on;
        autoindex_exact_size off;
        autoindex_format html;
        autoindex_localtime on;
    

    error_page   500 502 503 504  /50x.html;
    location = /50x.html 
        root   /usr/share/nginx/html;
    

EOF

修改文件权限，允许容器内部读写

chmod -R a+rwx /etc/nginx/
chmod -R a+rwx /usr/share/nginx/

3、以容器方式启动nginx服务，注意修改为以下端口以免冲突

docker run -d --name nginx-okd \\
  --restart always \\
  -p 8088:80 \\
  -v /etc/nginx/templates:/etc/nginx/templates \\
  -v /usr/share/nginx/html:/usr/share/nginx/html:ro \\
  nginx:1.21.6-alpine

浏览器访问验证：

安装OpenShift CLI

OpenShift CLI ( oc) 用于从命令行界面与 OKD 交互，可以在 Linux、Windows 或 macOS 上安装oc。

下载地址：https://github.com/openshift/okd/releases

1、下载openshift-client到本地，如果网络不好可以使用浏览器下载后在上传到bastion节点

wget https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-03-07-131213/openshift-client-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz

2、解压到/usr/local/bin目录下

tar -zxvf openshift-client-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz
cp oc /usr/local/bin/
cp kubectl /usr/local/bin/

3、检查版本，后续拉取镜像需要该版本信息

[root@bastion ~]# oc version
Client Version: 4.10.0-0.okd-2022-03-07-131213

安装OpenShift安装程序

openshift-install是OpenShift 4.x cluster的安装程序，是openshift集群的安装部署工具。

下载地址：https://github.com/openshift/okd/releases

1、下载openshift-install到本地，版本与openshift CLI要一致：

wget https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-03-07-131213/openshift-install-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz

2、解压到/usr/local/bin目录下

tar -zxvf openshift-install-linux-4.10.0-0.okd-2022-03-07-131213.tar.gz 
cp openshift-install /usr/local/bin/

3、检查版本

[root@bastion ~]# openshift-install version
openshift-install 4.10.0-0.okd-2022-03-07-131213
built from commit 3b701903d96b6375f6c3852a02b4b70fea01d694
release image quay.io/openshift/okd@sha256:2eee0db9818e22deb4fa99737eb87d6e9afcf68b4e455f42bdc3424c0b0d0896
release architecture amd64

安装harbor镜像仓库

使用harbor作为openshift镜像仓库，提前将对应版本镜像同步到本地仓库，加快后续安装过程。

1、安装docker-compose

curl -L "https://get.daocloud.io/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose version

2、下载harbor并解压

curl -L https://github.com/goharbor/harbor/releases/download/v2.4.2/harbor-offline-installer-v2.4.2.tgz -o ./harbor-offline-installer-v2.4.2.tgz
tar -zxf harbor-offline-installer-v2.4.2.tgz -C /opt/

如果下载较慢，可以考虑使用国内清华源地址：
https://mirrors.tuna.tsinghua.edu.cn/github-release/goharbor/harbor/v2.4.2/harbor-offline-installer-v2.4.2.tgz

3、生成harbor https证书，注意修改域名信息，参考自harbor官方文档

mkdir -p /opt/harbor/cert
cd /opt/harbor/cert

openssl genrsa -out ca.key 4096

openssl req -x509 -new -nodes -sha512 -days 3650 \\
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.example.com" \\
 -key ca.key \\
 -out ca.crt

openssl genrsa -out registry.example.com.key 4096

openssl req -sha512 -new \\
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.example.com" \\
    -key registry.example.com.key \\
    -out registry.example.com.csr
	
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=registry.example.com
DNS.2=registry.example
DNS.3=registry
EOF

openssl x509 -req -sha512 -days 3650 \\
    -extfile v3.ext \\
    -CA ca.crt -CAkey ca.key -CAcreateserial \\
    -in registry.example.com.csr \\
    -out registry.example.com.crt

查看生成的证书

root@bastion:/opt/harbor/cert#  ll
total 28
drwxr-xr-x 2 root root  158 Apr  3 21:51 ./
drwxr-xr-x 3 root root   19 Apr  3 21:39 ../
-rw-r--r-- 1 root root 2069 Apr  3 21:49 ca.crt
-rw------- 1 root root 3243 Apr  3 21:49 ca.key
-rw-r--r-- 1 root root   41 Apr  3 21:51 ca.srl
-rw-r--r-- 1 root root 2151 Apr  3 21:51 registry.example.com.crt
-rw-r--r-- 1 root root 1716 Apr  3 21:50 registry.example.com.csr
-rw------- 1 root root 3243 Apr  3 21:50 registry.example.com.key
-rw-r--r-- 1 root root  277 Apr  3 21:50 v3.ext

复制证书到操作系统目录

cp ca.crt registry.example.com.crt /usr/local/share/ca-certificates/
update-ca-certificates

复制证书到harbor运行目录

mkdir -p /data/cert/
cp registry.example.com.crt /data/cert/
cp registry.example.com.key /data/cert/

将证书提供给docker

openssl x509 -inform PEM -in registry.example.com.crt -out registry.example.com.cert

mkdir -p /etc/docker/certs.d/registry.example.com:8443
cp registry.example.com.cert /etc/docker/certs.d/registry.example.com:8443/
cp registry.example.com.key /etc/docker/certs.d/registry.example.com:8443/
cp ca.crt /etc/docker/certs.d/registry.example.com:8443/

4、修改harbor配置文件，调整以下内容，注意修改为以下端口，以免与haproxy冲突

cd /opt/harbor
cp harbor.yml.tmpl harbor.yml

# vi harbor.yml
hostname: registry.example.com
http:
  port: 8080
https:
  port: 8443
  certificate: /data/cert/registry.example.com.crt
  private_key: /data/cert/registry.example.com.key

5、安装并启动harbor

./install.sh

配置harbor开机自启动

cat >/etc/systemd/system/harbor.service<<EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/goharbor/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

systemctl enable harbor

确认harbor运行状态正常

root@bastion:/opt/harbor# docker-compose ps
      Name                     Command                  State                                             Ports                                       
------------------------------------------------------------------------------------------------------------------------------------------------------
harbor-core         /harbor/entrypoint.sh            Up (healthy)                                                                                     
harbor-db           /docker-entrypoint.sh 96 13      Up (healthy)                                                                                     
harbor-jobservice   /harbor/entrypoint.sh            Up (healthy)                                                                                     
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp                                                         
harbor-portal       nginx -g daemon off;             Up (healthy)                                                                                     
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:8080->8080/tcp,:::8080->8080/tcp, 0.0.0.0:8443->8443/tcp,:::8443->8443/tcp
redis               redis-server /etc/redis.conf     Up (healthy)                                                                                     
registry            /home/harbor/entrypoint.sh       Up (healthy)                                                                                     
registryctl         /home/harbor/start.sh            Up (healthy)

验证登录harbor，用户名为admin，默认密码为Harbor12345

docker login registry.example.com:8443

浏览器访问Harbor，注意，本地配置好hosts解析或指定dns服务器

https://registry.example.com:8443

手动创建一个项目名为openshift

同步okd镜像到harbor仓库

harbor镜像仓库准备就绪后，开始将quay.io中的openshit okd容器镜像同步到本地。

1、创建一个openshift临时安装目录

mkdir -p /opt/okd-install/4.10.0/
cd /opt/okd-install/4.10.0/

2、在红帽网站注册账号，下载pull-secret：https://console.redhat.com/openshift/install/pull-secret，（理论上可选，以下pull-secret内容仅为演示不可用，需要自行下载）

root@bastion:/opt/okd-install/4.10.0# cat pull-secret.txt
"auths":"cloud.openshift.com":"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfZjBjYmJiMDgyN2QyNGI0NDhjM2NkYjFiNTg0Y2M5MTY6VVBPRjZVTFRUQUpDTVhMSzFaNElNQkxWRUQwVjQ0VUFQOFVBSzZIR0pQWVNONUtZUDdETk1YMlZWWkw4M1A3TQ==","email":"willzhmic@outlook.com","quay.io":"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2NsdfasdfsafMDgyN2QyNGI0NDhjM2NkYjFiNTg0Y2M5MTY6VVBPRjZVTFRUQUpDTVhMSzFaNElNQkxWRUQwVjQ0VUFQOFVBSzZIR0pQWVNONUtZUDdETk1YMlZWWkw4M1A3TQ==","email":"willzhmic@outlook.com","registry.connect.redhat.com":"auth":"fHVoYy1wb29sLWExMWJiZjQ5LWExMzktNDBhNC1hZjM3LTViMWU1MzkyNjg2MzpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSXlaREUzTkdWall6WXpaalUwTkRZMkasdfsadfsafUzWkNKOS5oOUN2ZXNrbGJfaTM1TXVfMXhtWWdwMW91YzhYNHQxM2lNQW1ESk40M0dlQ1lKQlpBVXFQYVZGN09aeXNuSHBYaHprNWpYZWc4MzAwUUFRaTNBVnBNNFIxZ2VaeElVQjQ5S2lDbzVTWXRibHVPWGhRc2xXcUdFRklzbnJMSkxwS0hPNXUzX1BaWVNRNWxGNDFIWks5MERPdlo4YVAwMmpIMmZmZ0x4MXFmZFBkUXIxdTNDc3pCYTlWcFBzQjRMWlNzQkQwQWVCYmZibHNpUkRXbDBUSWx4bnFiTWItSFMydlBnSHZQbWs0ZENub2Q5TDdWek5FMVpJNkdPOC1LM0NCb1NacFVNTm9JdlFFWHpBZVJoRnZVZlF2RlJYRDJUblJQSHg2UzdOemM4SnRja3R1OE1FSmJmV0ZvQ2NyeE9uejNsaXpsSkVYYzdjLTA2V0NpVmZndVpfTXpOU1dmSlhWOW9QMlBxQV94eFhELS1hWE84OUhUczdmNWQ4VXgtTEhqczQybWhSMU9jeWYyT2ZYbUloSkRGaVRIaENqUGVSTHBDbEFSeDI1ZzN4NVBXSzdWYXdRMEFodlgtSlhvRXJLMlNreGcxdGxFclVfS3NSblpMYkIyTm82Q3IySVN0aUZ3WUxMSHl6LUp3QWdsel92VnBzeVF2eExtNFNCcy1BdUpPdjFqRjVGQnc4VzNCSG9sX0ZrUEVVZnZTQWRqaG5odXRXMVM3TlFYZ3FJR1lkRGEzTERFRWxta3FwYlVELWZCSzBQa0MwQTA4a3FscGROUmhyNG5PSGZCRFJVdjJMamppUEtSbFU5d012WjhuNGRkcVFLOXNEMGpYaW1UV3hpLTduanlvOEJKaTBrTDF6VGVzMmh4d0RweS1lTFdWSlJYZFdlRE9OZEVucw==","email":"willzhmic@outlook.com","registry.redhat.io":"auth":"fHVoYy1wb29sLWExMWJiZjQ5LWExMzktNDBhNC1hZjM3LTViMWU1MzkyNjg2MzpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSXlaREUzTkdWall6WXpaalUwTkRZMk9EZasdfsadfasdfS5oOUN2ZXNrbGJfaTM1TXVfMXhtWWdwMW91YzhYNHQxM2lNQW1ESk40M0dlQ1lKQlpBVXFQYVZGN09aeXNuSHBYaHprNWpYZWc4MzAwUUFRaTNBVnBNNFIxZ2VaeElVQjQ5S2lDbzVTWXRibHVPWGhRc2xXcUdFRklzbnJMSkxwS0hPNXUzX1BaWVNRNWxGNDFIWks5MERPdlo4YVAwMmpIMmZmZ0x4MXFmZFBkUXIxdTNDc3pCYTlWcFBzQjRMWlNzQkQwQWVCYmZibHNpUkRXbDBUSWx4bnFiTWItSFMydlBnSHZQbWs0ZENub2Q5TDdWek5FMVpJNkdPOC1LM0NCb1NacFVNTm9JdlFFWHpBZVJoRnZVZlF2RlJYRDJUblJQSHg2UzdOemM4SnRja3R1OE1FSmJmV0ZvQ2NyeE9uejNsaXpsSkVYYzdjLTA2V0NpVmZndVpfTXpOU1dmSlhWOW9QMlBxQV94eFhELS1hWE84OUhUczdmNWQ4VXgtTEhqczQybWhSMU9jeWYyT2ZYbUloSkRGaVRIaENqUGVSTHBDbEFSeDI1ZzN4NVBXSzdWYXdRMEFodlgtSlhvRXJLMlNreGcxdGxFclVfS3NSblpMYkIyTm82Q3IySVN0aUZ3WUxMSHl6LUp3QWdsel92VnBzeVF2eExtNFNCcy1BdUpPdjFqRjVGQnc4VzNCSG9sX0ZrUEVVZnZTQWRqaG5odXRXMVM3TlFYZ3FJR1lkRGEzTERFRWxta3FwYlVELWZCSzBQa0MwQTA4a3FscGROUmhyNG5PSGZCRFJVdjJMamppUEtSbFU5d012WjhuNGRkcVFLOXNEMGpYaW1UV3hpLTduanlvOEJKaTBrTDF6VGVzMmh4d0RweS1lTFdWSlJYZFdlRE9OZEVucw==","email":"willzhmic@outlook.com"

转换为json格式

apt install -y jq
cat ./pull-secret.txt | jq . > pull-secret.json

生成本地harbor镜像仓库base64位的加密口令

echo -n 'admin:Harbor12345' | base64 -w0

创建harbor镜像仓库登录文件

cat >pull-secret-local.json<<EOF

    "auths":
        "registry.example.com:8443":
            "auth":"YWRtaW46SGFyYm9yMTIzNDU=",
            "email":""
        
    

EOF

将harbor镜像仓库登录文件内容追加到pull-secret.json中，最终示例如下:

root@bastion:~# cat pull-secret.json

  "auths": 
    "cloud.openshift.com": 
      "auth": "b3BlbnNoaWZ0LXJasdfasd3NfZjBjYmJiMDgyN2QyNGI0NDhjM2NkYjFiNTg0Y2M5MTY6VVBPRjZVTFRUQUpDTVasfdasdafkxWRUQwVjQ0VUFQOFVBSzZIR0pQWVNONUtZUDdETk1YMlZWWkw4M1A3TQ==",
      "email": "example@outlook.com"
    ,
    "quay.io": 
      "auth": "b3BlbnNoaWZ0LXJlbGVhc2UtZGV2KasfdassadjM2NkYjFiNTg0Y2M5MTY6VVBPRjZVTFRUQUpDTasfdxWRUQwVjQ0VUFQOFVBSzZIR0pQWVNONUtZUDdETk1YMlZWWkw4M1A3TQ==",
      "email": "example@outlook.com"
    ,
    "registry.connect.redhat.com": 
      "auth": "fHVoYy1wb29sLWExMWJiZjQ5LWExMzktNDBhNC1hZjM3LTViMWU1MzkyNjg2MzpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSXlaREUzTkdWall6WXpaalUwTkRZMk9EZ3dOekZrWVRjME9UYzRPR0UzWkNKOS5oOUN2ZXNrbGJfaTM1TXVfMXhtWWdwMW91YzhYNHQxM2lNQW1ESk40M0dlQ1lKQlpBVXFQYVZGasdfasdfsadfaVaeElVQjQ5S2lDbzVTWXRibHVPWGhRc2xXcUdFRklzbnJMSkxwS0hPNXUzX1BaWVNRNWxGNDFIWks5MERPdlo4YVAwMmpIMmZmZ0x4MXFmZFBkUXIxdTNDc3pCYTlWcFBzQjRMasdfasfbHNpUkRXbDBUSWx4bnFiTWItSFMydlBnSHZQbWs0ZENub2Q5TDdWek5FMVpJNkdPOC1LM0NCb1NacFVNTm9JdlFFWHpBZVJoRnZVZlF2RlJYRDJUblJQSHg2UzdOemM4SnRja3R1OE1FSmJmV0ZvQ2NyeE9uejNsaXpsSkVYYzdjLTA2V0NpVmZndVpfTXpOU1dmSlhWOW9QMlBxQV94eFhELS1hWE84OUhUczdmNWQ4VXgtTEhqczQybWhSMU9jeWYyT2ZYbUloSkRGaVRIaENqUGVSTHBDbEFSeDI1ZzN4NVBXSzdWYXdRMEFodlgtSlhvRXJLMlNreGcxdGxFclVfS3NSblpMYkIyTm82Q3IySVN0aUZ3WUxMSHl6LUp3QWdsel92VnBzeVF2eExtNFNCcy1BdUpPdjFqRjVGQnc4VzNCSG9sX0ZrUEVVZnZTQWRqaG5odXRXMVM3TlFYZ3FJR1lkRGEzTERFRWxta3FwYlVELWZCSzBQa0MwQTA4a3FscGROUmhyNG5PSGZCRFJVdjJMamppUEtSbFU5d012WjhuNGRkcVFLOXNEMGpYaW1UV3hpLTduanlvOEJKaTBrTDF6VGVzMmh4d0RweS1lTFdWSlJYZFdlRE9OZEVucw==",
      "email": "example@outlook.com"
    ,
    "registry.redhat.io": 
      "auth": "fHVoYy1wb29sLWExMWJiZjQ5LWExMzktNDBhNC1hZjM3LTViMWU1MzkyNjg2MzpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSXlaREUzTkdWall6WXpaalUwTkRZMk9EZ3dOekZrWVRjME9UYzRPR0UzWkNKOS5oOUN2ZXNrbGJfaTM1TXVfMXhtWWdwMW91YzhYNHQxM2lNQW1ESk40M0dlQ1lKQlpBVXFQYVZGN09aeXNuSHBYaHprNWpYZWc4MzAwUUFRaTNBVnBNNFIxZ2VaeElVQjQ5S2lDbzVTWXRibHVPWGhRc2xXcUdFRklzbnJMSkxwS0hPNXUzX1BaWVNRNWxGNDFIWks5MERPdlo4YVAwMmpIMmZmZ0x4MXFmZFBkUXIxdTNDc3pCYTlWcFBzQjRMWlNzQkQwQWVCYmZibHNpUkRXbDBUSWx4bnFiTWItSFMydlBnSHZQbWs0ZENub2Q5TDdWek5FMVpJNkdPOC1LM0NCb1NacFVNTm9JdlFFWHpBZVJoRnZVZlF2RlJYRDJUblJQSHg2UzdOemM4SnRja3R1OE1FSmJmV0ZvQ2NyeE9uejNsaXpsSkVYYzdjLTA2V0NpVmZndVpfTXpOU1dmSlhWOW9QMlBxQV94eFhELS1hWE84OUhUczdmNWQ4VXgtTEhqczQybWhSMU9jeWYyT2ZYbUloSkRGaVRIaENqUGVSTHBDbEFSeDI1ZzN4NVBXSzdWYXdRMEFodlgtSlhvRXJLMlNreGcxdGxFclVfS3NSblpMYkIyTm82Q3IySVN0aUZ3WUxMSHl6LUp3QWdsel92VnBzeVF2eExtNFNCcy1BdUpPdjFqRjVGQnc4VzNCSG9sX0ZrUEVVZnZTQWRqaG5odXRXMVM3TlFYZ3FJR1lkRGEzTERFRWxta3FwYlVELWZCSzBQa0MwQTA4a3FscGROUmhyNG5PSGZCRFJVdjJMamppUEtSbFU5d012WjhuNGRkcVFLOXNEMGpYaW1UV3hpLTduanlvOEJKaTBrTDF6VGVzMmh4d0RweS1lTFdWSlJYZFdlRE9OZEVucw==",
      "email": "example@outlook.com"
    ,
    "registry.example.com:8443": 
      "auth": "YWRtaW46SGFyYm9yMTIzNDU=",
      "email": ""

3、查看oc版本号

root@bastion:/opt/okd-install/4.10.0# oc version
Client Version: 4.10.0-0.okd-2022-03-07-131213

配置以下变量

export OKD_RELEASE="4.10.0-0.okd-2022-03-07-131213"
export LOCAL_REGISTRY='registry.example.com:8443'
export LOCAL_REPOSITORY='openshift/okd'
export PRODUCT_REPO='openshift'
export LOCAL_SECRET_JSON='/opt/okd-install/4.10.0/pull-secret.json'
以上是关于OpenShift 容器平台社区版 OKD 4.10.0部署的主要内容，如果未能解决你的问题，请参考以下文章 
 在裸机上部署openshift4.3
 关于OpenShift(OKD)网络ServiceRoutes的一些认识
 OpenShift-OKD3.10部署
 [openshift]openshfit OKD的安装
 OpenShift-安装部署
 OpenShift+VMware：新的容器架构