jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - k8s系列

Posted 2022-04-02 93年的老男孩

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - k8s系列相关的知识，希望对你有一定的参考价值。

环境背景

已实现 k8s系列(一)-使用kubeadm安装kubernetes(k8s)
km - 2cpu - 4g内存 - ip - 192.168.23.39
node1 - 2cpu - 2G内存 - ip - 192.168.23.40
node1 - 2cpu - 2G内存 - ip - 192.168.23.41
示例语言 - 因GO语言支持跨平台编译，对容器化部署非常友好，所以示例使用GO语言
git 仓库地址（CI/CD所需文件及代码） https://gitee.com/lnamp/k8s.git tag为:master-20220331-00
所有操作基于官方文档说明

GO语言安装

wget https://golang.google.cn/dl/go1.18.linux-amd64.tar.gz

tar -xvzf go1.18.linux-amd64.tar.gz

cp -a go /usr/local/go-1.18

ln -s /usr/local/go-1.18/* /bin/

搭建本地仓库

1.搭建私有镜像仓库

#拉取所需镜像
docker pull registry

#启动私有镜像仓库
docker run -itd -e REGISTRY_STORAGE_DELETE_ENABLED=true -p 5000:5000 -v /www/wwwroot/private_registry:/var/lib/registry --name docker_registry registry

#私有镜像仓库可视化web仓库
docker pull konradkleine/docker-registry-frontend:v2

#启动私有镜像仓库可视化web
docker run -d --restart=always -e ENV_DOCKER_REGISTRY_HOST=192.168.23.39 -e ENV_DOCKER_REGISTRY_PORT=5000 -p 9011:80 konradkleine/docker-registry-frontend:v2

2.修改仓库源(三台机器都需要修改)

vim /etc/docker/daemon.json

#增加如下代码
"insecure-registries": [
    "192.168.23.39:5000"
],

3.常用命令

#查看所有仓库
curl -XGET http://192.168.23.39:5000/v2/_catalog

#查看某个仓库的tag
curl -XGET http://192.168.23.39:5000/v2/k8s/tags/list

#查看某个仓库tag的sha256码
curl --header "Accept: application/vnd.docker.distribution.manifest.v2+json" -I -X HEAD http://192.168.23.39:5000/v2/k8s/manifests/tagname

#删除某个Tag
curl -v -X DELETE http://192.168.23.39:5000/v2/k8s/manifests/sha256:sha256code

安装jenkins

1.安装jenkins - 基于官方文档

wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo --no-check-certificate

rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

yum install -y java-11-openjdk

yum install -y jenkins

systemctl start jenkins

systemctl daemon-reload

#查看admin密码
cat /var/lib/jenkins/secrets/initialAdminPassword

2.初始化jenkins

访问:http://192.168.23.39:8080/
账号:admin 密码:cat /var/lib/jenkins/secrets/initialAdminPassword
登录后选择安装推荐的插件 ( 如果因为网络原因部分失败，请点击重试 )
安装插件 - Manage Jenkins -> Manage Plugin
- 搜索 Authorization 安装 Role-based Authorization Strategy 和 Authorize Project - 权限管理
- 搜索 kubernetes 安装 kubernetes
- 搜索 Git Parameter 安装 Git Parameter

创建CI/CD

1.准备工作

#拉取环境镜像
docker pull alpine:latest

#安装git
yum install -y git

#将jenkins添加到docker用户组 - 重要
gpasswd -a jenkins docker

#重启jenkins
systemctl restart jenkins

2.创建Pipeline

创建工作任务 - 输入任务名称 - 选择Pipeline

3.Pipeline流水线配置推荐

pipeline 

    agent any

    stages 
        stage(clone) 
            steps 
                echo clone // 拉取代码
            
        
        stage(build go) 
            steps 
                echo build go // 编译可执行文件
            
        
        stage(make image) 
            steps 
                echo make image // 制作代码镜像
            
        
        stage(push image) 
            steps 
                echo push image // 推送到私有仓库
            
        
        stage(deploy) 
            steps 
                echo deploy // 部署代码

4.使用流水线语法工具

选择工作任务点击设置，拉到最下面，点击流水线语法

1).生成git语法 - 填充到 clone

示例步骤 - 选择git
输入仓库url: https://gitee.com/lnamp/k8s.git
如需账号密码则需要添加凭据选择 user_name with password 填充确定
填充其他信息
生成流水线脚本填充到 clone

git credentialsId: gitee_userid, url: https://gitee.com/lnamp/k8s.git

2).生成kubernetes(k8s)语法 - 填充到 deploy

准备工作


#kubernetes(k8s) 相关配置
cat /root/.kube/config

#分别保存 certificate-authority-data/client-certificate-data/client-key-data的值为对应的txt文件
cat /root/.kube/config|grep certificate-authority-data |awk -F : print $2 > certificate-authority-data.txt
cat /root/.kube/config|grep client-certificate-data |awk -F : print $2 > client-certificate-data.txt
cat /root/.kube/config|grep client-key-data |awk -F : print $2 > client-key-data.txt

#生成秘钥文件
cat certificate-authority-data.txt |base64 -d > ca.crt
cat client-certificate-data.txt |base64 -d > client.crt
cat client-key-data.txt |base64 -d > client.key
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
#Enter Export Password:输入自定义密码
#Verifying - Enter Export Password:再次输入自定义密码

#查看kubernetes(k8s)集群信息
kubectl cluster-info

- 示例步骤 - 选择kubeconfig
- 输入服务端点: kubectl cluster-info 可查看 https://192.168.23.39:6443
- 填充 Certificate of certificate authority - ca.crt文件内容
- 添加凭据类型选择Certificate
  - 上传cert.pfx文件
  - 输入密码为上面 自定义密码
  - 填充其他信息
- 生成流水线脚本 填充到 deploy
~~~shell
kubeconfig(caCertificate: -----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIyMDMxOTA4MTAwM1oXDTMyMDMxNjA4MTAwM1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOMy
ekzc4rWutBV/5ATcWLirMI8mXPycEW8WpGsZF28nlFBHy50UlmcD0tViV/JDjc2e
mj9/DEWy/H81os1a8jTcyGLI8p+TKi3avDibQGe4etUDF+eJavEn5zqWEoP98ohA
HfTJijBfIaI7n/qCiHVMnwvS+2yakaMcBoNhgOCDCN9gHpNqa1xBrXIr6o1HMrv7
RQ88t26yss3e/wq3XqNApEBdk1nIkDHy/ZuFO+kTMtPPO67QBNa9LaFhKRU2/VdE
/HAGk3n5JQ604Bn8hLZmDrf11p2dDbN6K9NtKbGuOLXbN0PsLnoBMCFdiRC/Ol1t
RneY86zkPgk6FQEv15ECAwEAAaNZMFcwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFLiwKGVC7HEf3goVSrzybGlpvWdSMBUGA1UdEQQO
MAyCCmt1YmVybmV0ZXMwDQYJKoZIhvcNAQELBQADggEBABcXtzTpR+Oee8VzVQte
gqvy3DQyTOcne5CS0q+kliyY36Tfsh9mieGhhHKRlD2esWVyl25qo6D0zgDhQOem
y/QKS7/Wka3i49ygw4dxC/mJEAzMRsrCxsEFqptHXM6IMaXmsLbsil16kmMx2fLV
4g46+TKqrRmdRSYMtUTieZtl+IJU9zmQJSYxCUFLXPysknxqenGOmbqnPat9FcZ1
BkrZf7yD8fCbq50zcFtSP3d6hxiN9rbPGsNWGjkj+WXZ0EymxWVukKwX/BRuNlLP
jDVgDCeR1rMCoGMkTgm1WqjynOAn/V4dl757E5da2NeOPgCfWBIuKdqOMPo4eWUV
Mls=
-----END CERTIFICATE-----, credentialsId: kubernetes-pfx, serverUrl: https://192.168.23.39:6443) 
    --TODO

5.相关 Deployment与Service配置在git仓库内go_app.yaml文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-app
  labels:
    app: goweb
spec:
  selector:
    matchLabels:
      app: goweb
  replicas: 10
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: goweb
    spec:
      containers:
        - name: go-app
          image: 192.168.23.39:5000/k8s:tag_name
          imagePullPolicy: Always
          ports:
          - name: http
            containerPort: 80
          livenessProbe:
            httpGet:
              port: 80
              path: /ping
            initialDelaySeconds: 2
            periodSeconds: 60
            timeoutSeconds: 3
--- 
apiVersion: v1
kind: Service
metadata:
  name: go-app
spec:
  type: ClusterIP
  selector:
    app: goweb
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80

6.最终流水线脚本

pipeline 

    agent any

    stages 
        stage(clone) 
            steps 
                echo clone // 拉取代码
                git credentialsId: gitee_userid, url: https://gitee.com/lnamp/k8s.git
                sh "git checkout master-20220331-00" //切换本期对应的代码分支
            
        
        stage(build go) 
            steps 
                echo build go // 编译可执行文件
                sh "export GOPROXY=https://goproxy.cn && go mod tidy && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./run run.go"
                sh "mkdir -p work && cp -a run ./work/ && cp -a static ./work/"
            
        
        stage(make image) 
            steps 
                echo make image // 制作代码镜像
                sh "docker build -f Dockerfile -t 192.168.23.39:5000/k8s:tag_name ." //相关Dockerfile在git上
            
        
        stage(push image) 
            steps 
                echo push image // 推送到私有仓库
                sh "docker push 192.168.23.39:5000/k8s:tag_name"
            
        
        stage(deploy) 
            steps 
                echo deploy // 部署代码
                kubeconfig(caCertificate: -----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIyMDMyOTAzMzcyOVoXDTMyMDMyNjAzMzcyOVowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOZO
NeeVjwkKy0BoGA0QetynFk89/3Zlh2EUmbbBXMygwDkbiJ0wRGyIDXiYqQ4HniC5
q0I8DwTB7WPkwhaHBGrQItXcuoo68hxnR3sRewwfuUB4uivkkhqIyeMk2KgSvbLW
41dnX6QamWSYJLGkLMmLru+BaKkEPGe1SZQH6ognATDm19Kt5vtu70kZH5O7qH38
TOguX+inuboREDB2RpMBm0Qp5NdXm3QXFlwHkryvYYIR5JkexHnUi9jjLy3V4qdQ
uDJhrcS6/w286IHeMzZ5dOaKcQ4vp7/wyk4soD+5MHnaBmttRkUIGwaTwmWWQjUp
votUOjK3CAkD/EsysqMCAwEAAaNZMFcwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFObVS5S7UkExTHiRIaJzKXtAfuiYMBUGA1UdEQQO
MAyCCmt1YmVybmV0ZXMwDQYJKoZIhvcNAQELBQADggEBAEN2sVIkQhuCGaLMuxTh
e+91J7LYnMEw6U/RTM3jyxm/6L0iSOx4jYh3MES7Nl7r42IRF0QxOV8cPy3AzMby
MZs+KEt67EusJFNucEkRbmcJuYmokbzc3U+hxQi4rA2AHnOLUPvH9cz6A1uaRlJ5
+PqGatgLSKXeZIwYtlI+JV5a6s4Ra2fBNbvcGSm0n8IJp0Jf5kgqmF7Gy5pBKDuP
2ifb45U/ntkc7hIzXd+wJd6369W031NUjAOMCE1xovv4RGlW2BAa0PEX+XBuZ3bW
UVTd0XVUVuhjoStKI+3jm7GF96d/a5Igj+JwkjeEq3POC131LMlt5gd2ohMTbcr2
MyM=
-----END CERTIFICATE-----, credentialsId: kubernetes-key, serverUrl: https://192.168.23.39:6443) 
                    sh "kubectl apply -f go_app.yaml"

7.执行jenkins -> Build now - 结束

以上是关于jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - k8s系列的主要内容，如果未能解决你的问题，请参考以下文章

（集群外）jenkins连接K8S集群

jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - k8s系列

基于Kubernetes集群的Jenkins CI/CD版本上线流程部署

Jenkins知识概括

Kubernetes和Jenkins——基于Kubernetes构建Jenkins持续集成平台