jenkins流水线部署K8S应用

Posted 2023-02-02

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了jenkins流水线部署K8S应用相关的知识，希望对你有一定的参考价值。

通过独立的jenkins部署K8S应用

PS：这种方式是jenkins独立，然后slave已容器方式运行，master是独立出来的，减少运维成本，还能提高效率。

master部署https://blog.csdn.net/Hlroliu/article/details/118902124

如果使用master部署需要把master加入k8s集群

jenkins安装不赘述

创建证书秘钥

在k8s-master节点上生成对应的证书

#1、查看kubernetes的config文件
cat ~/.kube/config
#2、根据配置文件生成证书.替换引号内部的信息为config内相关value
echo "certificate-authority-data" | base64 -d > ca.crt
echo "client-certificate-data" | base64 -d > client.crt
echo "client-key-data" | base64 -d > client.key
#3、生成jenkins使用的cert.pfx，此处需要设置一个4位数以上的密码
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
ls
ca.crt cert.pfx client.key client.crt

添加凭据

在jenkins里添加凭据

jenkins流水线部署K8S应用_jenkins

凭据类型选Certificate

上传cert.pfx，输入刚才设置的密码即可

jenkins流水线部署K8S应用_k8s_02

安装插件

jenkins的地址为http://192.168.1.22:8080/

安装完毕后安装kubernetes插件重启

jenkins流水线部署K8S应用_jenkins_03

打开代理

manage jenkins ->configure global security -> 代理

jenkins流水线部署K8S应用_k8s_04

制作jenkins-slave镜像

FROM jenkins/jnlp-slave:latest-jdk11
USER root

#修改镜像时区
RUN ln -sf /usr/share/zoneinfo/Asia/ShangHai /etc/localtime && \\
    echo "Asia/Shanghai" > /etc/timezone && \\
    dpkg-reconfigure -f noninteractive tzdata    #重新配置tzdata软件包,使得时区设置生效

#修改debia源
RUN echo "deb http://mirrors.163.com/debian/ buster main" > /etc/apt/sources.list && \\
    echo "deb http://mirrors.163.com/debian/ buster-updates main non-free contrib" >> /etc/apt/sources.list && \\
    echo "deb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib" >> /etc/apt/sources.list
#安装docker相关依赖
RUN apt-get update && \\
    apt-get -y install apt-transport-https ca-certificates curl gnupg2 apt-utils lsb-release software-properties-common && \\
    curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/debian/gpg | apt-key add - && \\
    add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/debian $(lsb_release -cs) stable" && \\
    apt-get -y update && \\
    apt-get -y install docker-ce
#安装kubectl
RUN apt-get update && apt-get install -y apt-transport-https && \\
    curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - && \\
    echo "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list && \\
    apt-get update && \\
    apt-get install kubectl

docker build -t harbor.test.com/jenkins/jenkins-slave:v3 .

docker push harbor.test.com/jenkins/jenkins-slave:v3

配置K8S集群信息

重启后点击manage jenkins->manage nodes and clouds->configure clouds -> add a new cloud

或者直接http://192.168.1.22:8080/manage/configureClouds/

jenkins流水线部署K8S应用_jenkins_05

jenkins流水线部署K8S应用_k8s_06

填写集群信息

填写k8s集群api-server的url，kubernetes服务证书为刚刚生成的ca.crt，凭据为刚刚添加的，填写Jenkins地址，点击连接测试验证

jenkins流水线部署K8S应用_k8s_07

jenkins流水线部署K8S应用_jenkins_08

连接测试

jenkins流水线部署K8S应用_k8s_09

填写jenkins信息

配置jenkins地址

jenkins流水线部署K8S应用_jenkins_10

添加POD模板

添加 Pod 模板------>Kubernetes Pod Template--->按如下配置

jenkins流水线部署K8S应用_jenkins_11

往下拉有个sa信息，填写刚才的sa

jenkins流水线部署K8S应用_k8s_12

添加卷

jenkins流水线部署K8S应用_k8s_13

添加容器模板

添加容器------>Container Template------>按如下配置------>

容器填写刚才制作的镜像

jenkins流水线部署K8S应用_k8s_14

完成。

jenkins流水线部署K8S应用_jenkins_15

通过容器版jenkins部署K8S

创建名称空间

kubectl create namespace jenkins-k8s

创建 pv

apiVersion: v1
kind: PersistentVolume
metadata:
 name: jenkins-k8s-pv
spec:
 capacity:
 storage: 10Gi
 accessModes:
 - ReadWriteMany
 nfs:
 server: 192.168.1.63
 path: /data/v1

kubectl apply -f pv.yaml

创建 pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: jenkins-k8s-pvc
 namespace: jenkins-k8s
spec:
 resources:
 requests:
 storage: 10Gi
 accessModes:
 - ReadWriteMany

kubectl apply -f pvc.yaml

查看 pvc 和 pv 绑定是否成功

kubectl get pvc -n jenkins-k8s

jenkins流水线部署K8S应用_jenkins_16

创建一个 sa 账号

kubectl create sa jenkins-k8s-sa -n jenkins-k8s

把上面的 sa 账号做 rbac 授权

kubectl create clusterrolebinding jenkins-k8s-sa-cluster --clusterrole=cluster-admin --serviceaccount=jenkins-k8s:jenkins-k8s-sa

通过 deployment 部署 jenkins

docker pull jenkins/jenkins:latest

kind: Deployment
apiVersion: apps/v1
metadata:
  name: jenkins
  namespace: jenkins-k8s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccount: jenkins-k8s-sa
      containers:
      - name: jenkins
        image:  jenkins/jenkins:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkins-volume
          subPath: jenkins-home
          mountPath: /var/jenkins_home
      volumes:
      - name: jenkins-volume
        persistentVolumeClaim:
          claimName: jenkins-k8s-pvc

kubectl apply -f jenkins-deployment.yaml

kubectl get pods -n jenkins-k8s

jenkins流水线部署K8S应用_jenkins_17

注：如果看到CrashLoopBackOff，解决方法如下：

#查看 jenkins-675b586fdb-5mzmk 日志

[root@xuegod63 ~]# kubectl logs jenkins-675b586fdb-5mzmk -n jenkins-k8s

touch: cannot touch /var/jenkins_home/copy_reference_file.log: Permission denied

#上面问题是因为/data/v1 目录权限问题，按如下方法解决：

chown -R 1000.1000 /data/v1/

重新部署deployment即可

把 jenkins 前端加上 service，提供外部网络访问

apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: jenkins-k8s
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30002
  - name: agent
    port: 50000
    targetPort: agent

kubectl apply -f jenkins-service.yaml

kubectl get svc -n jenkins-k8s

jenkins流水线部署K8S应用_k8s_18

配置 Jenkins

在浏览器访问 jenkins 的 web 界面：

http://192.168.1.5:30002/

初始化不赘述。

安装kubernetes插件

配置 jenkins 连接到我们存在的 k8s 集群

jenkins流水线部署K8S应用_jenkins_19

jenkins地址填http://jenkins-service.jenkins-k8s.svc.cluster.local:8080/

jenkins流水线部署K8S应用_jenkins_20

POD和容器配置和上半部分一样即可。

流水线脚本

通用

node(testhan) 
    stage(Clone) 
        // sh "git config --global user.name root"
        // sh "git config --global user.email admin@example.com"

        // echo "1.Clone Stage"
        // git url: "git@192.168.1.20:test/jenkins-test.git"
        // script 
        //     build_tag = sh(returnStdout: true, script: git rev-parse --short HEAD).trim()
        // 
        deleteDir()
        git branch: main, credentialsId: gitlab, url: http://192.168.1.20/test/jenkins-test.git
    
    stage(Test) 
        echo "2.Test Stage"

    
    stage(Build) 
        echo "3.Build Docker Image Stage"
        sh "docker build -t harbor.test.com/test-image/jenkins-demo:$build_tag ."
    
    stage(Push) 
        echo "4.Push Docker Image Stage"
        withCredentials([usernamePassword(credentialsId: dockerhub, passwordVariable: dockerHubPassword, usernameVariable: dockerHubUser)]) 
            sh "docker login harbor.test.com -u $dockerHubUser -p $dockerHubPassword"
            sh "docker push harbor.test.com/test-image/jenkins-demo:$build_tag"
        
    
    stage(Deploy to dev) 
        echo "5. Deploy DEV"
        sh "sed -i s/<BUILD_TAG>/$build_tag/ k8s-dev.yaml"
        sh "sed -i s/<BRANCH_NAME>/$env.BRANCH_NAME/ k8s-dev.yaml"
        //        sh "bash running-devlopment.sh"
        sh "kubectl apply -f k8s-dev.yaml  --validate=false"
      
    stage(Promote to qa)   
        def userInput = input(
            id: userInput,

            message: Promote to qa?,
            parameters: [
                [
                    $class: ChoiceParameterDefinition,
                    choices: "YES\\nNO",
                    name: Env
                ]
            ]
        )
        echo "This is a deploy step to $userInput"
        if (userInput == "YES") 
            sh "sed -i s/<BUILD_TAG>/$build_tag/ k8s-qa.yaml"
            sh "sed -i s/<BRANCH_NAME>/$env.BRANCH_NAME/ k8s-qa.yaml"
            //            sh "bash running-qa.sh"
            sh "kubectl apply -f k8s-qa.yaml --validate=false"
            sh "sleep 6"
            sh "kubectl get pods -n qatest"
         else 
            //exit
        
  
    stage(Promote to pro)   
        def userInput = input(

            id: userInput,
            message: Promote to pro?,
            parameters: [
                [
                    $class: ChoiceParameterDefinition,
                    choices: "YES\\nNO",
                    name: Env
                ]
            ]
        )
        echo "This is a deploy step to $userInput"
        if (userInput == "YES") 
            sh "sed -i s/<BUILD_TAG>/$build_tag/ k8s-prod.yaml"
            sh "sed -i s/<BRANCH_NAME>/$env.BRANCH_NAME/ k8s-prod.yaml"
            //            sh "bash running-production.sh"
            sh "cat k8s-prod.yaml"
            sh "kubectl apply -f k8s-prod.yaml --record --validate=false"

以上是关于jenkins流水线部署K8S应用的主要内容，如果未能解决你的问题，请参考以下文章

使用jenkins流水线完成自动部署服务到K8s

再见 Jenkins！几行脚本搞定自动化部署，这款神器有点厉害

持续集成工具篇：Jenkins 与流水线管理转

自动化集成：Pipeline整合Docker+K8S

Jenkins——Jenkins+harbor+gitlab(git) 部署maven项目

再见Jenkins，从Gitlab代码提交到k8s服务持续交付只需七毛三(走过路过不要错过)