linux12Devops -->10Jenkins流水线容器化+Harbor私有仓库

Posted FikL-09-19

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了linux12Devops -->10Jenkins流水线容器化+Harbor私有仓库相关的知识,希望对你有一定的参考价值。

Jenkins流水线容器化+Harbor私有仓库

将Jenkins的编译环境迁移至k8s中

Jenkins中自动构建项目的类型

Jenkins中自动构建项目的类型有很多,常用的有以下三种:

自由风格软件项目( FreeStyle Project)
Maven 项目(Maven Project)
流水线项目( Pipeline Project)
# 每种类型的构建其实都可以完成一样的构建过程与结果,只是在操作方式、灵活度等方面有所区别,在实际开发中可以根据自己的需求和习惯来选择。(PS:个人推荐使用流水线类型,因为灵活度非常高)

Harbor

官网: https://goharbor.io/

部署私有镜像仓库(Harbor)

作为一个企业级私有 Registry 服务器,Harbor 提供了更好的性能和安全。提升用户使用 Registry 构建和运行环境传输镜像的效率。Harbor 支持安装在多个 Registry 节点的镜像资源复制,镜像全部保存在私有 Registry 中, 确保数据和知识产权在公司内部网络中管控。

另外,Harbor 也提供了高级的安全特性,诸如用户管理,访问控制和活动审计等

配置HTTPS

默认情况下,Harbor不附带证书,可以在没有安全性的情况下部署Harbor,以便您可以通过HTTP连接到它。但是,只有在没有连接到外部Internet的空白测试或开发环境中,才可以使用HTTP。在没有空隙的环境中使用HTTP会使您遭受中间人攻击;要配置HTTPS,必须创建SSL证书。

您可以使用由受信任的第三方CA签名的证书,也可以使用自签名证书。本节介绍如何使用 OpenSSL创建CA,以及如何使用CA签署服务器证书和客户端证书。

一、生成证书

将Harbor私有镜像仓库装在了k8m-m-01上,也可以安装一台测试机

  • 在生产环境中,您应该从CA获得证书。在测试或开发环境中,您可以生成自己的CA。要生成CA证书,请运行以下命令。

1.生成CA证书私钥

mkdir /opt/cert
cd /opt/cert
openssl genrsa -out ca.key 4096
2.生成CA证书
openssl req -x509 -new -nodes -sha512 -days 3650 \\
-subj "/C=CN/ST=ShangHai/L=ShangHai/O=Oldboy/OU=Linux/CN=192.168.15.130" \\
-key ca.key \\
-out ca.crt

3.生成服务器证书

openssl genrsa -out 192.168.15.130.key 4096

4.生成证书签名请求

openssl req -sha512 -new \\
-subj "/C=CN/ST=ShangHai/L=ShangHai/O=Oldboy/OU=Linux/CN=192.168.15.130" \\
-key 192.168.15.130.key \\
-out 192.168.15.130.csr

5.生成一个x509 v3扩展文件

  • 域名版(企业用内,此处没用到)
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=yourdomain.com
DNS.2=yourdomain
DNS.3=hostname
EOF
  • IP版(此处使用)
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:192.168.15.130
EOF

6.使用该v3.ext文件生成证书

openssl x509 -req -sha512 -days 3650 \\
-extfile v3.ext \\
-CA ca.crt -CAkey ca.key -CAcreateserial \\
-in 192.168.15.130.csr \\
-out 192.168.15.130.crt

7.安装Docker

[root@test ~]# vim docker.aliyun.sh 
# step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 &&\\
# Step 2: 添加软件源信息
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo &&\\
# Step 3
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo &&\\
# Step 4: 更新并安装Docker-CE
sudo yum makecache fast &&\\
sudo yum -y install docker-ce &&\\
# Step 4: 开启Docker服务
systemctl enable --now docker.service &&\\
#  Step 5: Docker加速优化服务
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://k7eoap03.mirror.aliyuncs.com"]
}
EOF

二、颁发证书

颁发证书给Harbor和Docker来用

生成后ca.crt,192.168.15.130.crt和192.168.15.130.key文件,必须将它们提供给Harbor和Docker,和重新配置港使用它们

1.分发证书

  • 将服务器证书和密钥复制到Harbor主机上的certficates文件夹中
/opt/cert/

openssl x509 -inform PEM -in 192.168.15.130.crt -out 192.168.15.130.cert

mkdir -pv /etc/docker/certs.d/192.168.15.130/

cp 192.168.15.130.{cert,key} ca.crt /etc/docker/certs.d/192.168.15.130/

# 如果nginx端口默认部署443和80,后面port则指定为端口号
/etc/docker/certs.d/192.168.15.130:port
/etc/docker/certs.d/192.168.15.130:port

# 复制Harbor证书(当前测试执行此步)
mkdir -p /data/cert 
cp 192.168.15.130.crt  /data/cert
cp 192.168.15.130.key /data/cert

# 重启docker加载证书
systemctl restart docker

2.安装Harbor

链接:https://github.com/goharbor/harbor

# 进入下载目录
cd /opt

# 下载
wget https://github.com/goharbor/harbor/releases/download/v2.3.1/harbor-offline-installer-v2.3.1.tgz
# 解压
tar xf harbor-offline-installer-v2.3.1.tgz

# 进入harbor目录
cd harbor/

# 下载docker-composrt
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# 打印出上方地址,复制到浏览器下载
curl -L https://raw.githubusercontent.com/docker/compose/1.29.2/contrib/completion/bash/dock
er-compose > /etc/bash_completion.d/docker-compose

# 设置可执行权限
chmod +x /usr/local/bin/docker-compose 

# 测试docker-composer
docker-compose -v

# 修改harbor.yaml以下三处
cp harbor.yml.tmpl harbor.yml
vim harbor.yml
hostname: 192.168.15.130
certificate: /data/cert/192.168.15.130.crt
private_key: /data/cert/192.168.15.130.key

# 生成配置文件,其实就是导入压缩包内的镜像	# 
docker load < harbor.v2.3.1.tar.gz
./prepare 

# 清空相关容器
docker-compose down

# 安装并启动
./install.sh
# 耐心等待...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db     ... done
Creating harbor-portal ... done
Creating registry      ... done
Creating redis         ... done
Creating registryctl   ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

# 若报错,则查看是否端口被占用,结束相关端口即刻,一般为nginx的80端口
# 修改端口可改为8089或其他,两处都要修改,再次执行脚本即可
vim harbor.yaml
hostname: 192.168.15.130:8089
port: 8089
# 先执行prepare,否则配置不生效
./prepare     # 生成harbor仓库
./install.sh  # 安装harbor

访问ip测试:https://192.168.15.130:8089  #默认是https://192.168.15.130/
用户名:admin
密码  :Harbor12345
#密码配置
[root@test harbor]# cat harbor.yml |grep 12345
harbor_admin_password: Harbor12345

3.测试Harbor

# 测试登录
docker login 192.168.15.130 --username=admin --password=Harbor12345

# 测试打包、推送到私有仓库
[root@k8m-m-01 harbor]# docker pull nginx
[root@k8m-m-01 harbor]# docker tag 62d49f9bab67 192.168.15.130/test/nginx:v1
[root@k8m-m-01 harbor]# docker push 192.168.15.130/test/nginx:v1

# 拉取
[root@k8m-m-01 harbor]# docker pull 192.168.15.130/test/nginx:v1

# k8s集群所有节点都必须给docker做证书认证
for i in n1 n2;
do
ssh $i "mkdir -pv /etc/docker/certs.d/192.169.12.11/"
scp 192.168.15.130.cert 192.168.15.130.key ca.crt root@$i:/etc/docker/certs.d/192.168.15.130/
ssh $i "systemctl restart docker"
done
# n1、n2访问成功

Jenkins流水线实现容器化

最近我们构建和部署服务的方式与原来相比简直就是突飞猛进,像那种笨拙的、单一的、用于构建单体式应用程序的方式已经是过去式了。

现在的应用为了提供更好的拓展性和可维护性,都会去拆解成各种相互依赖小、解耦性强的微服务,这些服务有各自的依赖和进度。这跟我们的Kubernetes不谋而合。

1.用JenKins连接K8S

1)创建admin-csr.json(kubernetes)
mkdir /opt/cert && cd /opt/cert

cat > admin-csr.json << EOF
{
    "CN":"admin",
    "key":{
        "algo":"rsa",
        "size":2048
    },
    "names":[
        {
            "C":"CN",
            "L":"BeiJing",
            "ST":"BeiJing",
            "O":"system:masters",
            "OU":"System"
        }
    ]
}
EOF
2)安装证书生成工具(kubernetes)
# 下载
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

# 设置执行权限
chmod +x cfssljson_linux-amd64 cfssl_linux-amd64

# 移动到/usr/local/bin
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl_linux-amd64 /usr/local/bin/cfssl
3)创建证书私钥(kubernetes)
cfssl gencert -ca=/etc/kubernetes/pki/ca.crt -ca-key=/etc/kubernetes/pki/ca.key --profile=kubernetes admin-csr.json | cfssljson -bare admin
4)配置证书(kubernetes)
openssl pkcs12 -export -out ./jenkins-admin.pfx -inkey ./admin-key.pem -in ./admin.pem -passout pass:123456
5)配置jenkins链接k8s(Jenkins)
系统管理 ---> 系统配置 ---> cloud



2.链接K8s测试

[root@k8m-m-01 ~]# kubectl create secret generic kubeconfig --from-file=.kube/config 
secret/kubeconfig created
pipeline {
  agent {
    kubernetes{
      cloud "${KUBERNETES_NAME}"  #集群名字
      slaveConnectTimeout 1200  #连接超时时间
      yaml '''
apiVersion: v1
kind: pod  #metadata 流水线默认创建,此处不许设置
spec:
  containers:
    - name: jnlp
      image: jenkins/inbound-agent:4.7-1-jdk11
      args: [\\'$(JENKINS_SECRET)\\', \\'$(JENKINS_NAME)\\']
      imagePullPolicy: IfNotPresent  #镜像拉取策略:不存在则拉取
      volumeMounts:  #挂载时区,用一个系统时间
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
    - name: docker  #与流水线中容器名称必须一致container(name: 'docker', shell: 'echo')
      image: docker:19.03.15-git  #docker容器镜像必须19版本
      imagePullPolicy: IfNotPresent
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/var/run/docker.sock"  
          name: "volume-docker"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
    - name: kubectl  #与流水线中容器名称必须一致 container(name: 'kubectl', shell: 'echo')
      image: bitnami/kubectl:1.20.2  #kubectl版本要与k8s本版一致
      imagePullPolicy: IfNotPresent
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-docker"
          readOnly: false
        - mountPath: "/root/.kube"
          name: "kubeconfig"
          readOnly: false
    - name: golang  #业务容器
      image: golang:1.16.3
      imagePullPolicy: IfNotPresent
      tty: true
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      command:  #默认指令,不执行相当于占位符
        - "cat"
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
    - name: maven  #java所需容器镜像
      image: maven:3.6.3-openjdk-8
      tty: true
      command:
        - "cat"
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/root/.m2/repository"
          name: "volume-maven-repo"
          readOnly: false
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
  volumes:
    - name: volume-maven-repo
      emptyDir: {}
    - name: volume-2
      hostPath:
        path: "/usr/share/zoneinfo/Asia/Shanghai"
    - name: kubeconfig
      secret:
        secretName: kubeconfig
        items:
          - key: config
            path: config
    - name: volume-docker
      hostPath:
        path: "/var/run/docker.sock"
    - name: volume-hosts
      hostPath:
        path: /etc/hosts


'''
    }
  }
  stages {
    stage('源代码管理') {
      parallel {
        stage('拉取代码') {
          steps {
            git(url: '${GIT_REPOSITORY_URL}', branch: '${GIT_TAG}', changelog: true, credentialsId: '${CREDENTIALS_ID}')
          }
        }

        stage('系统检查及初始化') {
          steps {
            sh """

            echo "Check Sysyem Env"

"""
          }
        }

      }
    }

    stage('编译及构建') {
      parallel {
        stage('编译代码') {
          steps {
            container(name: 'golang', shell: 'echo') {
              sh """

                go build main.go

"""
            }

          }
        }

        stage('初始化操作系统及kubernetes环境') {
          steps {
            script {
              CommitID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
              CommitMessage = sh(returnStdout: true, script: "git log -1 --pretty=format:'%h : %an  %s'").trim()
              def curDate = sh(script: "date '+%Y%m%d-%H%M%S'", returnStdout: true).trim()
              TAG = curDate[0..14] + "-" + CommitID + "-master"
            }
          }
        }

      }
    }

    stage('构建镜像及检查kubernetes环境') {
      parallel {
        stage('构建镜像') {
          steps {
            withCredentials([usernamePassword(credentialsId: '${DOCKER_REPOSITORY_CREDENTIAL_ID}', passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
              container(name: 'docker', shell: 'echo') {
                sh """
                  docker build -t ${HARBOR_HOST}/${NAMESPACE_NAME}/${REPOSITORY_NAME}:${TAG} .
                  docker login ${HARBOR_HOST} --username=${USERNAME} --password=${PASSWORD}
                  docker push ${HARBOR_HOST}/${NAMESPACE_NAME}/${REPOSITORY_NAME}:${TAG}
                """
              }
            }
          }
        }

        stage('Check Kubernetes ENV') {
          steps {
            container(name: 'kubectl', shell: 'echo') {
              sh 'sleep 10'
            }

          }
        }

      }
    }

    stage('Deploy Image to Kubernetes') {
      steps {
        container(name: 'kubectl', shell: 'echo') {
          sh 'sleep 10'
        }
      }
    }

    stage('Test Service') {
      steps {
        sh 'sleep 10'
      }
    }

    stage('Send Email to Admin') {
      steps {
        sh 'sleep 10'
      }
    }

  }
}
    stage('Check Kubernetes ENV') {
      steps {
        container(name: 'kubectl', shell: 'echo') {
          sh 'sleep 10'
        }

      }
    }

  }
}

stage('Deploy Image to Kubernetes') {
  steps {
    container(name: 'kubectl', shell: 'echo') {
      sh 'sleep 10'
    }
  }
}

stage('Test Service') {
  steps {
    sh 'sleep 10'
  }
}

stage('Send Email to Admin') {
  steps {
    sh 'sleep 10'
  }
}
   }
  }

以上是关于linux12Devops -->10Jenkins流水线容器化+Harbor私有仓库的主要内容,如果未能解决你的问题,请参考以下文章

linux12Devops -->02安装GitLab

linux12Devops -->04Jenkins参数化构建

linux12Devops -->06Jenkins部署tomcat

linux12Devops -->07jenkins部署python

linux12Devops -->08jenkins部署Golang

linux12Devops -->05 Jenkins部署php实战