Jenkins 基于k8s应用

Posted 2021-08-31 凌许冬

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Jenkins 基于k8s应用相关的知识，希望对你有一定的参考价值。

一. 简介

当前k8s平台已趋于普及，传统的主机部署jenkins面临以下问题：

主 Master 发生单点故障时，整个流程都不可用了
每个 Slave 的配置环境不一样，来完成不同语言的编译打包等操作，但是这些差异化的配置导致管理起来非常不方便，维护起来也是比较费劲
资源分配不均衡，有的 Slave 要运行的 job 出现排队等待，而有的 Slave 处于空闲状态
资源有浪费，每台 Slave 可能是物理机或者虚拟机，当 Slave 处于空闲状态时，也不会完全释放掉资源。

正因为上面的这些种种痛点，我们渴望一种更高效更可靠的方式来完成这个 CI/CD 流程，而 Docker 虚拟化容器技术能很好的解决这个痛点，又特别是在 Kubernetes 集群环境下面能够更好来解决上面的问题，下图是基于 Kubernetes 搭建 Jenkins 集群的简单示意图：

从图上可以看到 Jenkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。
这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。

这种方式部署给我们带来如下好处：

服务高可用，当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master 容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用。
动态伸缩，合理使用资源，每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave 自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配 Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。
扩展性好，当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个 Kubernetes Node 到集群中，从而实现扩展。

二. 部署jenkins

创建jenkins的持久化存储pv、pvc


cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name:  jenkins
spec:
capacity: 
storage: 3Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
nfs:
path: /opt/k8s-pv
server: 172.18.227.128

```bash
cat pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins
  namespace: devops
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2G

因为jenkins需要动态创建pod，需要操作k8s的api，因此需要创建授权

[root@jumpesrv-SZ-Testing jenkins]# cat rbac.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-sa
  namespace: devops

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: jenkins-cr
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get","list","watch"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins-crd
roleRef:
  kind: ClusterRole
  name: jenkins-cr
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: jenkins-sa
  namespace: devops

创建jenkins服务deployment

[root@jumpesrv-SZ-Testing jenkins]# cat dep.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: \'2\'
creationTimestamp: \'2021-08-24T01:08:06Z\'
generation: 2
name: jenkins
namespace: devops
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
  app: jenkins
strategy:
rollingUpdate:
  maxSurge: 25%
  maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
  labels:
    app: jenkins
spec:
  containers:
    - env:
        - name: JAVA_OPTS
          value: >-
            -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0
            -Dhudson.slaves.NodeProvisioner.MARGIN=50
            -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
            -Duser.timezone=Asia/Shanghai
      image: \'jenkins/jenkins:lts\'
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 12
        httpGet:
          path: /login
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      name: jenkins
      ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
      readinessProbe:
        failureThreshold: 12
        httpGet:
          path: /login
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 60
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      resources:
        limits:
          cpu: \'2\'
          memory: 1Gi
        requests:
          cpu: \'2\'
          memory: 512Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkinshome
          subPathExpr: jenkins
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
  serviceAccount: jenkins-sa
  serviceAccountName: jenkins-sa
  terminationGracePeriodSeconds: 10
  volumes:
    - name: jenkinshome
      persistentVolumeClaim:
        claimName: jenkins
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: devops
spec:
ports:
- name: jenkins-8080-8080
  port: 80
  protocol: TCP
  targetPort: 8080
- name: jenkins-50000-50000
  port: 50000
  protocol: TCP
  targetPort: 50000
selector:
app: jenkins
sessionAffinity: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/service-weight: \'\'
generation: 1
name: jenkins
namespace: devops
spec:
rules:
- host: jenkins.eebbk.net
  http:
    paths:
      - backend:
          serviceName: jenkins
          servicePort: 80
        path: /
        pathType: ImplementationSpecific

启动如果报如下错误（因为我们容器里是以jenkins用户启动，而我们NFS服务器上是root启动，所以没有权限）：

touch: cannot touch \'/var/jenkins_home/copy_reference_file.log\': Permission denied
Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?

然后给我们NFS服务器上的目录授权即可：

chown -R 1000 /opt/k8s-pv/jenkins

然后登录网站，我们直接在浏览器用这个端口访问：

密码可以通过如下命令获得：

cat /opt/k8s-pv/jenkins/secrets/initialAdminPassword

三. 配置动态jenkins-slave

安装插件kubernetes
填写Kubernetes和Jenkins的配置信息
系统管理-->节点管理-->Configure Clouds

四. k8s部署gitlab

参考：k8s部署gitlab

五. 在slave中运行Pipeline

脚本如下：

pipeline {
  agent {
    node {
      label \'jenkins-slave\'
    }
  }
  stages {
      stage(\'Init\') {
          steps{
              script {
                  println "welcome to lingxudong"
                  println env.WORKSPACE
              }
          }
      }
      stage(\'Git Source\') {
          steps {
              script {
                  dir("${env.WORKSPACE}/workplace") {
                      git credentialsId: \'2f0f81b2-d268-4031-aec5-e9f86c85a990\', url: \'http://172.21.5.245/test/solo.git\'
                  }
              }
          }
      }
      stage(\'Maven Build\') {
          steps {
              script {
                  container(\'maven\') {
                      dir("${env.WORKSPACE}/workplace") {
                        sh \'mvn clean package\'  
                      }
                  }
              }
          }

      }
      stage(\'Docker Build\') {
          steps {
              script {
                  dir("${env.WORKSPACE}/workplace") {
                    container(\'docker\') {
                        sh \'docker login -u \' + HARBOR_CREDS_USR  + \' -p \' + HARBOR_CREDS_PSW + \' \' + HARBOR_HOST
                        sh "docker build --build-arg JAR_FILE=`ls workplace/target/*.jar |cut -d \'/\' -f2` -t " + HARBOR_HOST + "/" + DOCKER_IMAGE + ":" + GIT_TAG + " --no-cache ."
                        sh "docker push " + HARBOR_HOST + "/" + DOCKER_IMAGE + ":" + GIT_TAG
                    }                      
                  }
              }
          }
      }
      stage(\'K8s Deploy\') {
          steps {
              script {
                  dir("${env.WORKSPACE}/workplace") {
                      container(\'jnlp\') {
                          sh "sed -e \'s#{IMAGE_URL}#" + HARBOR_HOST + "/" + DOCKER_IMAGE + "#g;s#{IMAGE_TAG}#" + GIT_TAG + "#g;s#{APP_NAME}#" + APP_NAME + "#g\' k8s-deployment.tpl > k8s-deployment.yml"
                          sh "sed -e \'s#{APP_NAME}#" + APP_NAME + "#g;s#{NODE_PORT}#" + NODE_PORT + "#g\' k8s-deployment-svc.tpl > k8s-deployment-svc.yml"
                          sh "kubectl apply -f k8s-deployment.yml --namespace=" + K8S_NAMESPACE
                          sh "kubectl apply -f k8s-deployment-svc.yml --namespace=" + K8S_NAMESPACE
                      }
                  }
              }
          }
      }

  }
  environment {
      HARBOR_CREDS = credentials(\'fa459e9b-59bb-4677-b304-a87cb0f2fa24\')
      GIT_TAG = sh(returnStdout: true,script: \'echo \' + BUILD_NUMBER ).trim()
  }
  parameters {
      string(name: \'HARBOR_HOST\', defaultValue: \'registry.cn-shenzhen.aliyuncs.com\', description: \'harbor仓库地址\')
      string(name: \'DOCKER_IMAGE\', defaultValue: \'pro-bbk/jenkins-slave\', description: \'docker镜像名\')
      string(name: \'APP_NAME\', defaultValue: \'jenkinsdemo\', description: \'k8s中标签名\')
      string(name: \'NODE_PORT\', defaultValue: \'30000\', description: \'Service端口\')
      string(name: \'K8S_NAMESPACE\', defaultValue: \'default\', description: \'k8s的namespace名称\')
  }

}

脚本二：

node(\'jenkins-slave\') {
    stage(\'Clone\') {
      echo "1.Clone Stage"
      git credentialsId: \'2f0f81b2-d268-4031-aec5-e9f86c85a990\', url: \'http://172.21.5.245/test/h5-test.git\'
    }
    stage(\'Test\') {
      echo "2.Test Stage"
    }
    stage(\'docker-build\') {
      container(\'docker\') {
        echo "3. docker-build"
        sh \'\'\'
        tag=$(date +%Y%m%d%H%M%S)
        docker login registry.cn-shenzhen.aliyuncs.com  -u xxx -p xxx
        docker build -t registry.cn-shenzhen.aliyuncs.com/pro-bbk/jenkins-slave:$tag .
        docker push registry.cn-shenzhen.aliyuncs.com/pro-bbk/jenkins-slave:$tag
        sed -i "s/TAG/$tag/g" deployment.yaml
        \'\'\'
      }
    }
    stage(\'deploy\') {
      container(\'jnlp\') {
        echo "4. Deploy Stage"
        sh \'\'\'
        kubectl apply -f deployment.yaml
        \'\'\'
      }
    }
}

以上是关于Jenkins 基于k8s应用的主要内容，如果未能解决你的问题，请参考以下文章

在k8s+jenkins+github+dockerhub环境中用pipeline部署应用

K8S基于Docker+K8S+GitLab/SVN+Jenkins+Harbor搭建持续集成交付环境（环境搭建篇）

基于k8s构建企业jenkins CICD

基于 k8s 的 Jenkins 构建集群实践

Gitlab+Jenkins+Docker+Harbor+K8s集群搭建CICD平台