Jenkins容器自动化部署与持续集成（Kubernetes）

Posted 张志翔 ̮

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Jenkins容器自动化部署与持续集成（Kubernetes）相关的知识，希望对你有一定的参考价值。

         本文是SpringCloud实战（九）-Docker自动化部署与持续集成，若要关注前文，请点击传送门：

        SpringCloud实战（八）-断路器监控与集群监控(Hystrix Dashboard)

        前文我们介绍了Hystrix的集群监控。本文主要讲述SpringCloud构建镜像和远端部署。

一、Docker自动化部署简介

        随着业务的增长，需求也开始增多，每个需求的大小，开发周期，发布时间都不一致。基于微服务的系统架构，功能的叠加，对应的服务的数量也在增加，大小功能的快速迭代，更加要求部署的快速化，智能化。因此，传统的人工部署已经心有余而力不足。

        持续集成，持续部署，持续交互对于微服务开发来说，是提高团队整体效率不可或缺的一环。合理的使用CI,CD能够极大的提高了生产效率，也提高了产品的交互质量。

二、准备工作

        操作系统是Centos7，需要大家提前搭建好Gitlab、Jenkins，并安装好Maven、Git、Jdk1.8、Docker、Nexus3等，我已经将这些工具的安装和部署教程都写好了，下面直接带上传送门：

        Centos7安装Gitlab

        Centos7安装Jenkins

        Centos7安装Maven

        Centos7安装Jdk1.8

        Centos7安装Git

        Centos7安装Docker

        Centos7安装Nexus3

        搭建好上述环境后，我们就可以开始配置Jenkins来自动化部署了，相关架构图如下：


        首先用户通过账号名密码登录到Jenkins，然后选择需要构建的服务模块，点击构建，此时Client端会发送一条请求到Jenkins，Jenkins会将该服务模块代码从Gitlab上进行拉取，然后执行之前已经配置好的Shell脚本，Shell脚本首先会将工程打成Jar包，然后将Jar包构建成镜像，之后会将镜像上传到Nexus镜像仓库，Jenkins在完成上传镜像的动作之后会向Sonar发起代码质量检测请求，Sonar接到请求后会从Gitlab上拉取代码，然后将拉取到的代码进行Bug、坏味道检测，Sonar会将检测结果展现在监控平台上，完成质量检测这一步后，Jenkins会通过SSH连接192.168.3.206节点并远端执行拉取镜像，集群部署命令，用户可以通过Portainerio来查看和管理集群。 

三、配置Jenkins

        jenkins需要配置的细节比较多，大家千万别遗漏下任何细节哈。

        1、jenkins插件安装

        首先我们来安装一些需要用到的插件，Jenkins在进行页面安装的时候会自动给你安装一些插件，不过我们还需要用到其他的一些插件，我们先登录jenkins找到插件管理，访问 http://127.0.0.1:8371/，如图所示：

        找到并点击可选插件tab，从过滤输入框中输入插件名就可以准确的找到该插件，然后安装就可以了，这里我列一个需要安装的插件列表：

GitLab Plugin
Localization: Chinese (Simplified)
SSH plugin
Maven Integration plugin
JDK Tool Plugin
Git plugin

        目测应该是全的，如果下面的步骤中发现页面和我的不一样，可能是因为插件漏装了，大家直接在评论区提醒我，我立刻补全。

        安装好插件之后我们找到全局工具配置（在系统管理页面上），如图所示：

        点击进去，进入全局工具配置页面，如图所示：

        我们在这里对Maven、Jdk、Git、Docker进行配置的，如图所示：

        这里Jdk、Maven需要在环境变量中配置一下，配置详情关注上文的准备工作，然后我们的全局工具配置就配好了。

        下面我们去配置系统设置，如图所示：

        系统设置这里我们配置两个地方就好了，一个是SSH，一个是Gitlab，SSH的配置是为了我们能够连接到远端主机进行镜像拉取和部署，Gitlab的配置是为了和Jenkins建立关联，如图所示：

        这里SSH的凭证用远端机器的登录用户名和密码，Gitlab凭证用Gitlab登录后的token，如图所示：

        到这里系统设置就配置完成了。

四、新建任务

        Jenkins配置完之后，我们来新建一个任务。

        我们新建一个Maven风格的项目，然后点击确定，进入到项目配置页面，这里我们需要配置Gitlab项目所在的地址，并且需要用token认证，Gitlab项目所在地址如图所示：

        然后我们还需要写一下构建中和构建后执行的脚本。

        1、Jenkins参数化构建

        在开始写构建中和构建后执行脚本之前，需要参照以下文章搭建安装相关插件和学习参数配置，传送门如下：

        Jenkins实现参数化构建

        2、Docker脚本配置方式

        Build阶段脚本如下：

  clean install dockerfile:build dockerfile:push

        Post Steps本地执行脚本如下：

  docker rmi 192.168.3.202:8088/oascloud/eureka-server | true

        Post Steps远端执行脚本如下：

docker stop eureka-server | true
docker rm -v eureka-server | true
docker rmi 192.168.3.202:8088/oascloud/eureka-server | true
docker pull 192.168.3.202:8088/oascloud/eureka-server
docker run -itd -p 8761:8761 -v  /usr/local/logs:/usr/local/logs --name eureka-server 192.168.3.202:8088/oascloud/eureka-server

        详细配置，图示如下： 

        3、Kubernetes脚本配置方式

        Build阶段脚本如下：

        Root POM：

  pom.xml

        Goals and options：

  clean package dockerfile:build dockerfile:push -Dmaven.test.skip=true

        补充：

        如果工程有多个模块，在Jenkins构建的时候只需要构建其中某一个模块的镜像，此时 Goals and options 如下所示：

  clean test package install dockerfile:build dockerfile:push -pl $module -am -Dmaven.test.skip=true

        maven非常智能，只要父工程的pom文件中关联了子工程，可以不需要指定目录进行打包、构建。

        Post Steps阶段脚本如下：

        Execute shell：

  docker rmi 10.12.1.202:8088/oascloud/openailab-data-center-service | true

        Execute shell script on remote host using ssh（Master节点执行）：

#!/bin/bash
kubectl delete deploy/openailab-data-center-service --namespace=oas-$ENV|true
kubectl delete svc/openailab-data-center-service --namespace=oas-$ENV|true
kubectl apply -f /home/oas/openailab-k8s-yaml/$ENV/openailab-data-center-service/deploy.yaml --record
kubectl create -f /home/oas/openailab-k8s-yaml/$ENV/openailab-data-center-service/svc.yaml

        Execute shell script on remote host using ssh（Slave节点执行）：

  docker rmi $(docker images | grep '<none>' | awk 'print $3') | true

        Execute shell script on remote host using ssh（Slave节点执行）：

  docker rmi $(docker images | grep '<none>' | awk 'print $3') | true

        详细配置，图示如下： 

        对照着上面的图就更清楚了。

五、eureka-server工程改造

        然后我们需要创建一个springboot工程，这里我直接只用本专栏前面的eureka-server工程，需要做修改的地方是需要在工程的pom文件中增加docker插件配置

<build>
    <plugins>
        <plugin>
            <groupId>com.spotify</groupId>
            <artifactId>dockerfile-maven-plugin</artifactId>
            <version>1.3.6</version>
            <configuration>
                <repository>$docker.image.url/$docker.image.prefix/$project.artifactId</repository>
                <useMavenSettingsForAuth>true</useMavenSettingsForAuth>
                <buildArgs>
                    <JAR_FILE>target/$project.build.finalName.jar</JAR_FILE>
                </buildArgs>
            </configuration>
        </plugin>
    </plugins>
</build>

        如果需要将eureka-server上传到远端仓库，则需要在pom文件中增加Nexus远端仓库配置（这里的id需要与maven setting中的<servers><server><id>保持一致，通过相同的id进行配置关联）

<!--定义snapshots库和releases库的nexus地址-->
<distributionManagement>
    <repository>
        <id>nexus-releases</id>
        <url>
            http://192.168.3.202:8083/repository/maven-releases/
        </url>
    </repository>
    <snapshotRepository>
        <id>nexus-snapshots</id>
        <url>
            http://192.168.3.202:8083/repository/maven-snapshots/
        </url>
    </snapshotRepository>
</distributionManagement>

        并且需要在pom文件的同级目录新建一个Dockerfile文件，内容如下：

FROM frolvlad/alpine-java:jdk8-slim
VOLUME /tmp
ARG JAR_FILE
ADD $JAR_FILE app.jar
RUN sh -c 'touch /app.jar'zz
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
ENV JAVA_OPTS="-server -Xms512m -Xmx512m -Xss512k"
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]

        我们的SpringBoot工程就改造好了。

六、maven setting配置

        我们需要修改maven的setting文件，需要增加我们的nexus上的镜像仓库地址，所以需要我们提前在nexus中新建一个docker仓库并获取到他的镜像仓库暴露端口，图示如下

        然后我们拿到docker仓库的访问路径之后，去修改maven setting中的docker仓库配置。

        setting配置内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<!--
 | This is the configuration file for Maven. It can be specified at two levels:
 |
 |  1. User Level. This settings.xml file provides configuration for a single user,
 |                 and is normally provided in $user.home/.m2/settings.xml.
 |
 |                 NOTE: This location can be overridden with the CLI option:
 |
 |                 -s /path/to/user/settings.xml
 |
 |  2. Global Level. This settings.xml file provides configuration for all Maven
 |                 users on a machine (assuming they're all using the same Maven
 |                 installation). It's normally provided in
 |                 $maven.conf/settings.xml.
 |
 |                 NOTE: This location can be overridden with the CLI option:
 |
 |                 -gs /path/to/global/settings.xml
 |
 | The sections in this sample file are intended to give you a running start at
 | getting the most out of your Maven installation. Where appropriate, the default
 | values (values used when the setting is not specified) are provided.
 |
 |-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <!-- localRepository
   | The path to the local repository maven will use to store artifacts.
   |
   | Default: $user.home/.m2/repository
  <localRepository>/path/to/local/repo</localRepository>
  -->
<localRepository>C:\\Users\\Lenovo\\.m2\\repo\\</localRepository>

  <!-- interactiveMode
   | This will determine whether maven prompts you when it needs input. If set to false,
   | maven will use a sensible default value, perhaps based on some other setting, for
   | the parameter in question.
   |
   | Default: true
  <interactiveMode>true</interactiveMode>
  -->

  <!-- offline
   | Determines whether maven should attempt to connect to the network when executing a build.
   | This will have an effect on artifact downloads, artifact deployment, and others.
   |
   | Default: false
  <offline>false</offline>
  -->

  <!-- pluginGroups
   | This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.
   | when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers
   | "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.
   |-->
  <pluginGroups>
      <pluginGroup>com.spotify</pluginGroup>
  </pluginGroups>

  <!-- proxies
   | This is a list of proxies which can be used on this machine to connect to the network.
   | Unless otherwise specified (by system property or command-line switch), the first proxy
   | specification in this list marked as active will be used.
   |-->
  <proxies>
    <!-- proxy
     | Specification for one proxy, to be used in connecting to the network.
     |
    <proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <username>proxyuser</username>
      <password>proxypass</password>
      <host>proxy.host.net</host>
      <port>80</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
    -->
  </proxies>

  <!-- servers
   | This is a list of authentication profiles, keyed by the server-id used within the system.
   | Authentication profiles can be used whenever maven must make a connection to a remote server.
   |-->
  <servers>
    <!-- server
     | Specifies the authentication information to use when connecting to a particular server, identified by
     | a unique name within the system (referred to by the 'id' attribute below).
     |
     | NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are
     |       used together.
     |
    <server>
      <id>deploymentRepo</id>
      <username>repouser</username>
      <password>repopwd</password>
    </server>
    -->

    <!-- Another sample, using keys to authenticate.
    <server>
      <id>siteServer</id>
      <privateKey>/path/to/private/key</privateKey>
      <passphrase>optional; leave empty if not used.</passphrase>
    </server>
    -->
	<server>
      <id>nexus-releases</id>
      <username>admin</username>
      <password>admin123</password>
    </server>
	<server>
      <id>nexus-snapshots</id>
      <username>admin</username>
      <password>admin123</password>
    </server>
	<server>
        <id>192.168.3.202:8088</id>
        <username>admin</username>
        <password>admin123</password>
    </server>
  </servers>

 <mirrors>
    <mirror>
        <id>nexus-aliyun</id>
        <mirrorOf>central</mirrorOf>
        <name>Nexus aliyun</name>
        <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
	<mirror>
        <id>maven-public</id>
        <mirrorOf>*</mirrorOf>
        <url>http://192.168.3.202:8083/repository/maven-public/</url>
    </mirror>
</mirrors>

  <!-- profiles
   | This is a list of profiles which can be activated in a variety of ways, and which can modify
   | the build process. Profiles provided in the settings.xml are intended to provide local machine-
   | specific paths and repository locations which allow the build to work in the local environment.
   |
   | For example, if you have an integration testing plugin - like cactus - that needs to know where
   | your Tomcat instance is installed, you can provide a variable here such that the variable is
   | dereferenced during the build process to configure the cactus plugin.
   |
   | As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles
   | section of this document (settings.xml) - will be discussed later. Another way essentially
   | relies on the detection of a system property, either matching a particular value for the property,
   | or merely testing its existence. Profiles can also be activated by JDK version prefix, where a
   | value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.
   | Finally, the list of active profiles can be specified directly from the command line.
   |
   | NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact
   |       repositories, plugin repositories, and free-form properties to be used as configuration
   |       variables for plugins in the POM.
   |
   |-->
  <profiles>
    <!-- profile
     | Specifies a set of introductions to the build process, to be activated using one or more of the
     | mechanisms described above. For inheritance purposes, and to activate profiles via <activatedProfiles/>
     | or the command line, profiles have to have an ID that is unique.
     |
     | An encouraged best practice for profile identification is to use a consistent naming convention
     | for profiles, such as 'env-dev', 'env-test', 'env-production', 'user-jdcasey', 'user-brett', etc.
     | This will make it more intuitive to understand what the set of introduced profiles is attempting
     | to accomplish, particularly when you only have a list of profile id's for debug.
     |
     | This profile example uses the JDK version to trigger activation, and provides a JDK-specific repo.
    <profile>
      <id>jdk-1.4</id>

      <activation>
        <jdk>1.4</jdk>
      </activation>

      <repositories>
        <repository>
          <id>jdk14</id>
          <name>Repository for JDK 1.4 builds</name>
          <url>http://www.myhost.com/maven/jdk14</url>
          <layout>default</layout>
          <snapshotPolicy>always</snapshotPolicy>
        </repository>
      </repositories>
    </profile>
    -->

    <!--
     | Here is another profile, activated by the system property 'target-env' with a value of 'dev',
     | which provides a specific path to the Tomcat instance. To use this, your plugin configuration
     | might hypothetically look like:
     |
     | ...
     | <plugin>
     |   <groupId>org.myco.myplugins</groupId>
     |   <artifactId>myplugin</artifactId>
     |
     |   <configuration>
     |     <tomcatLocation>$tomcatPath</tomcatLocation>
     |   </configuration>
     | </plugin>
     | ...
     |
     | NOTE: If you just wanted to inject this configuration whenever someone set 'target-env' to
     |       anything, you could just leave off the <value/> inside the activation-property.
     |
    <profile>
      <id>env-dev</id>

      <activation>
        <property>
          <name>target-env</name>
          <value>dev</value>
        </property>
      </activation>

      <properties>
        <tomcatPath>/path/to/tomcat/instance</tomcatPath>
      </properties>
    </profile>
    -->
      <profile>
        <id>sonar</id>
        <activation>
          <activeByDefault>true</activeByDefault>
        </activation>
        <properties>
		  <sonar.jdbc.url>
            jdbc:mysql://192.168.3.203:3306/sonar
          </sonar.jdbc.url>
          <sonar.jdbc.driver>com.mysql.jdbc.Driver</sonar.jdbc.driver>
          <sonar.jdbc.username>root</sonar.jdbc.username>
          <sonar.jdbc.password>aidclouddb@123!@#</sonar.jdbc.password>
          <sonar.host.url>http://192.168.3.202:9000</sonar.host.url>
        </properties>
      </profile>
	<profile>
		<id>nexus</id>
		<repositories>
			<repository>
				<id>nexus-public</id>
				<url>http://192.168.3.202:8083/repository/maven-public/</url>
				<releases>
					<enabled>true</enabled>
					<updatePolicy>always</updatePolicy>
				</releases>
				<snapshots>
					<enabled>true</enabled>
					<updatePolicy>always</updatePolicy>
				</snapshots>
			</repository>
		</repositories>
		<pluginRepositories>
			<pluginRepository>
				<id>nexus-public</id>
				<url>http://192.168.3.202:8083/repository/maven-public/</url>
				<releases>
					<enabled>true</enabled>
					<updatePolicy>always</updatePolicy>
				</releases>
				<snapshots>
					<enabled>true</enabled>
					<updatePolicy>always</updatePolicy>
				</snapshots>
			</pluginRepository>
		</pluginRepositories>
	</profile>
  </profiles>

  <activeProfiles>
  	<activeProfile>nexus</activeProfile>
  	<activeProfile>sonar</activeProfile>
  </activeProfiles>

  <!-- activeProfiles
   | List of profiles that are active for all builds.
   |
  <activeProfiles>
    <activeProfile>alwaysActiveProfile</activeProfile>
    <activeProfile>anotherAlwaysActiveProfile</activeProfile>
  </activeProfiles>
  -->
</settings>

        我们同时还需要修改远端主机上的docker配置文件，需要指定镜像仓库为nexus的地址，所以我们先登陆到远端主机，通过下面的命令找到配置文件

    $ vim /etc/docker/daemon.json

        添加如下配置，将以下配置写入到文件中，保存并退出。

 
 
    "insecure-registries":["192.168.3.202:8088"], 
    "registry-mirrors": ["http://hub-mirror.c.163.com"]

        其中192.168.3.202:8088就是我们nexus的docker仓库暴露出来的端口地址。

        修改完之后我们就可以在Jenkins中执行构建，如图所示：

        然后我们就成功构建了Springboot工程，访问远端主机的服务路径，我这里是 http://192.168.3.206:8761/，之后就成功显示出eureka服务端监控平台，说明我们已经成功自动化部署了一个eureka-server项目。

        到此自动化部署和持续集成配置完成，我已经很尽力的在描述了，如果各位有不太清楚的地方，请在下方评论区留言，之后的文章我会讲自动化集群部署和管理。

以上是关于Jenkins容器自动化部署与持续集成（Kubernetes）的主要内容，如果未能解决你的问题，请参考以下文章

Jenkins与Gitlib实现自动化部署与持续构建

Jenkins与gitlib实现自动化部署与持续构建

持续集成与自动化部署 - jenkins与sonargitlab集成

Jenkins践行持续集成与持续部署实战之DevOps详解

持续集成与持续部署宝典Part 3：创建集成环境

使用阿里云容器服务Jenkins实现持续集成和Docker镜像构建