这些年我们一起搞过的持续集成~Jenkins+Perl and Shell script

Posted 2020-09-17 gavanwanggw

这些年我们一起搞过的持续集成~Jenkins+Perl and Shell script

##本文同一时候发表在http://www.cnblogs.com/wade-xu/p/4378224.html ##转载注明出处

 

部门用持续集成已经非常久了，但事实上使用起来还是非常麻烦的。每当要给一个新项目set up持续集成的环境，尽管是Copy一些现有的jobs, 可是很多參数，变量须要去改，然后还有调试，少说3,4天搞一下，非常不方便。

近期比較空，就把现有的持续集成系统升级改造下，job用一套模板，所有參数化，仅仅要改动配置文件，就能够为新项目配置好环境。

 

本文的重点是一些经验，想法的分享，并非一篇手把手教你搭建持续集成环境的教程，并且阅读本文须要一定的Jenkins, perl, shell脚本的基础知识， 所用到的知识有：Jenkins+Maven+Git+Sonar+Perl+Shell script

下图非常好的说明了我们现有的Job流， 以往是每一个项目都有一套这种job流，如今我改成这个Template_service job流，供全部的项目使用，项目名也是作为參数传递进来的。每一个job都是用Jenkins Execute shell这个组件来实现功能的，像check out code，Email Notification, Publish Junit test results, code coverage report 等Jenkins自带组件都没有使用。

 

执行环境：  apache-maven-3.0.5, jdk1.7.0_15, git 1.8.2.1, SonarQube 4.0, perl v5.8.8

 ##转载注明出处：http://www.cnblogs.com/wade-xu/p/4378224.html 

 

接下来简单的说明一下各个Job的功能，以及贴出具体的脚本供大家參考。

除了上图里面的job, 另一个Config job, export一些环境变量

0)Template_Service_Config (one time run)

用来Export 环境变量的，比方Project list, project git 地址， 

export PROJECT_LIST=PROJECTA,PROJECTB
export PROJECTA_GIT_PROJECT_URL=ssh://[e-mail]:7999/Project/ProjectA.git
export PROJECTA_BRANCH=development
export PROJECTA_SERVICE_NAME=China_Template_Service
export PROJECTA_EMAIL_LIST=[e-mail],[e-mail]
export PROJECTA_EMAIL_CC_LIST=[e-mail]

......

export PROJECTB_GIT_PROJECT_URL=ssh://[e-mail]:7999/abc/ProjectB.git
export PROJECTB_BRANCH=perf
export PROJECTB_SERVICE_NAME=China_ProjectB_DEV 
export PROJECTB_EMAIL_LIST=[e-mail],[e-mail] 
export PROJECTB_EMAIL_CC_LIST=[e-mail]

......

 

Build 这个job 的时候须要Parameters

 

代码例如以下：主要作用就是将输入的Parameters写入到一个文件。以便兴许全部job 一上来source一下

TEST=${WORKSPACE}/test
CONFIG_FILE=Project_Config.txt

rm -rf ${TEST}
git clone ssh://[e-mail]:7999/project/test.git
cd ${TEST}

cd Config
if [ ! -f "${CONFIG_FILE}" ]; then  
touch "${CONFIG_FILE}"
fi

##########Rewrite the config file base on the project list###########
CURRENT_TIME=`date +"%m/%d/%Y/%T"`
echo -e "export CONFIG_TIME=${CURRENT_TIME}\\n
export PROJECT_LIST=${PROJECT_LIST}\\n 
export SHARED_FOLDER=~/jenkins/common\\n" > ${CONFIG_FILE}

###########List the project1 parameters###########
echo -e "${PROJECTA_PARAMETERS}" >> ${CONFIG_FILE}

###########List the project2 parameters###########
echo -e "${PROJECTB_PARAMETERS}" >> ${CONFIG_FILE}

git add ${CONFIG_FILE}
git commit -m "config file"
git push origin master

cp ${CONFIG_FILE} ~/jenkins/common/Config
cd ..
cp Scripts/* ~/jenkins/common/Scripts
chmod 755 ~/jenkins/common/Scripts/*

还有就是将一些perl写的脚本从git clone下来放到某一个公共文件夹下。以便兴许job使用， 有例如以下脚本，比方发送email, Unit test analyzer

部分脚本像gridService, packageService是操作公司一个网格云计算平台。安装rpm包，重新启动service用的，不方便贴出来。

[wadexu@vm10226 common]$ cd Scripts/
[wadexu@vm10226 Scripts]$ ll
total 404
-rwxr-xr-x 1 wadexu wadexu  37853 Mar 27 09:12 gridService
-rwxr-xr-x 1 wadexu wadexu   4991 Mar 27 09:12 packageService
-rwxr-xr-x 1 wadexu wadexu   1628 Mar 27 09:12 queryDB.pl
-rwxr-xr-x 1 wadexu wadexu 317183 Mar 27 09:12 Reporter.jar
-rwxr-xr-x 1 wadexu wadexu   2190 Mar 27 09:12 report.pl
-rwxr-xr-x 1 wadexu wadexu   3415 Mar 27 09:12 sendEmail.pl
-rwxr-xr-x 1 wadexu wadexu   1210 Mar 27 09:12 ServiceInfoProvider.pl
-rwxr-xr-x 1 wadexu wadexu   3418 Mar 27 09:12 UnitTestAnalyzer.pl

 

 1)Template_Service_Initial

遍历Project list 列出的项目， 使用curl 调用下一个job (Commit_Auto_Build) 的Rest API， Build job with parameter接口

每一个项目的Job流都是通过Project name + Initial job的Build number 来贯穿的, 这点非常重要，多项目一起执行时各项目之间不会错乱，比方写Log文件。分析Log文件生成report

rm -rf ${WORKSPACE}/*

####### Import config file #######
cd ~/jenkins/common/Config
. Project_Config.txt
cd ~/jenkins/common/Temp

####### Get the Project list and traverse the list ######    
          id=1
          list_size=`echo $PROJECT_LIST | awk -F "," \'{print NF;}\'`
          let list_size=list_size+1
          while [ $id -lt $list_size ]
            do 
                projectName=`echo "${PROJECT_LIST}"|awk -v id=$id -F "," \'{print $id}\'`      
                if [ "$projectName " != \' \' ];  then            
                curl -X POST --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Commit_Auto_Build/build --data json=\'{"parameter": [{"name":"PROJECT_NAME","value":"\'$projectName\'"},{"name":"PIPELINE_NUM","value":"\'$BUILD_NUMBER\'"}]}\'                        
                    id=`expr $id + 1`
                 else
                    break
                fi
            done     
   

 

2)Template_Service_Commit_Auto_Build

 这个Job的作用是用来比較代码版本号有没有更新，假设有更新（或者是第一次run这个job）就会调用兴许的-- Build job

还有就是列出这次Build和上次Build之间修改过代码的作者。

假设将Initial job 设置成每几分钟run一下，事实上这个job就起到了 每当有人提交代码，则自己主动Build的作用了。

### Import config file ###
cd ~/jenkins/common/Config
. Project_Config.txt

### Initial parameters ###
GIT_PROJECT_URL=${PROJECT_NAME}"_GIT_PROJECT_URL" 
GIT_PROJECT_URL=$(eval echo \\${$GIT_PROJECT_URL})

BRANCH=${PROJECT_NAME}"_BRANCH"
BRANCH=$(eval echo \\${$BRANCH})

LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"
BUILD_LOG_FOLDER=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}
BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAME

mkdir $BUILD_LOG_FOLDER

GIT_PROJECT_NAME=${PROJECT_NAME}"_GIT_PROJECT_NAME"
GIT_PROJECT_NAME=$(eval echo \\${$GIT_PROJECT_NAME})

#### setup ####
export CURRENT_RESOURCE_DIR=${WORKSPACE}/${GIT_PROJECT_NAME}
export CURRENT_MD5_FILE=${WORKSPACE}/${PROJECT_NAME}"_Code_Current"
export BASE_MD5_FILE=${WORKSPACE}/${PROJECT_NAME}"_Code_Base"
export CURRENT_AUTHOR_LIST=${WORKSPACE}/${PROJECT_NAME}"_Author_Current"
export BASE_AUTHOR_LIST=${WORKSPACE}/${PROJECT_NAME}"_Author_Base"

#### cd resource,get log commit MD5 ####
cd ${WORKSPACE}
rm -rf ${CURRENT_RESOURCE_DIR}
git clone ${GIT_PROJECT_URL}
cd ${CURRENT_RESOURCE_DIR}
git checkout -b ci origin/${BRANCH}

if [ ! -f "${BASE_AUTHOR_LIST}" ]; then  
  touch "${BASE_AUTHOR_LIST}"
fi

if [ ! -f "${BASE_MD5_FILE}" ]; then  
  touch "${BASE_MD5_FILE}"

  ########call build job######  
  curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Build/build --data json=\'{"parameter":[{"name":"PROJECT_NAME","value":"\'${PROJECT_NAME}\'"},{"name":"PIPELINE_NUM","value":"\'${PIPELINE_NUM}\'"}]}\'

  git log --pretty=format:"Author: %an" > ${BASE_AUTHOR_LIST}
  sort -u  ${BASE_AUTHOR_LIST}  >  ${BUILD_LOG}

  git log --max-count=1 --pretty=format:"%H" > ${BASE_MD5_FILE}
  exit 0
else
  git log --max-count=1 --pretty=format:"%H" > ${CURRENT_MD5_FILE}
  git log --pretty=format:"Author: %an" > ${CURRENT_AUTHOR_LIST}
fi

#### compare the md5 file, and run the job.####
## -a means && ##
    if cmp -s ${CURRENT_MD5_FILE} ${BASE_MD5_FILE}
       then
          echo "same,no changes!"
     else
      mv ${CURRENT_MD5_FILE} ${BASE_MD5_FILE}
      diff -b $CURRENT_AUTHOR_LIST $BASE_AUTHOR_LIST | grep "<" | sed \'s/^< //g\' > Update_Author.txt
      mv ${CURRENT_AUTHOR_LIST} ${BASE_AUTHOR_LIST}
      sort -u Update_Author.txt > ${BUILD_LOG}
        
########call build job###### 
      curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_service_Build/build --data json=\'{"parameter": [{"name":"PROJECT_NAME","value":"\'${PROJECT_NAME}\'"},{"name":"PIPELINE_NUM","value":"\'${PIPELINE_NUM}\'"}]}\'
    fi

这里用到了--pretty=format 这个參数， %H就是打印出哈希字串， %an是打印出Author， --max-count=1 取近期的一条log 记录

 

##转载注明出处：http://www.cnblogs.com/wade-xu/p/4378224.html 

 

3) Template_Service_Build

这个job是最核心的job。 执行rpm build, 将rpm包丢到公司内部网格云计算平台的Repository下， 分析单元測试结果， 执行Sonar, 从Sonar的首页上拿到这个项目的一些信息，比方Sonar上 blocker, critical, major 等issues, 还有code coverage，所有打印到log里， 以便兴许的脚本生成report, 另外就是Sonar issue, code coverage 不达标。则发送对应的email 给对应的project email list

rm -rf ${WORKSPACE}/*

###### Import config file ######
cd ~/jenkins/common/Config
. Project_Config.txt

###### Initial parameters ######
GIT_PROJECT_NAME=${PROJECT_NAME}"_GIT_PROJECT_NAME"
GIT_PROJECT_NAME=$(eval echo \\${$GIT_PROJECT_NAME})

LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"
BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAME

BRANCH=${PROJECT_NAME}"_BRANCH"
BRANCH=$(eval echo \\${$BRANCH})

GIT_PROJECT_URL=${PROJECT_NAME}"_GIT_PROJECT_URL" 
GIT_PROJECT_URL=$(eval echo \\${$GIT_PROJECT_URL})

REPOSITORY=${PROJECT_NAME}"_REPOSITORY" 
REPOSITORY=$(eval echo \\${$REPOSITORY})

INSTANCES=${PROJECT_NAME}"_INSTANCES"
INSTANCES=$(eval echo \\${$INSTANCES})

COVERAGE_URL=${PROJECT_NAME}"_COVERAGE_URL"
COVERAGE_URL=$(eval echo \\${$COVERAGE_URL})

SONAR_MIN_COVERAGE=${PROJECT_NAME}"_SONAR_MIN_COVERAGE"
SONAR_MIN_COVERAGE=$(eval echo \\${$SONAR_MIN_COVERAGE})

MAX_MAJOR_ISSUE_NUM=${PROJECT_NAME}"_MAX_MAJOR_ISSUE_NUM"
MAX_MAJOR_ISSUE_NUM=$(eval echo \\${$MAX_MAJOR_ISSUE_NUM})

MAX_CRITICAL_ISSUE_NUM=${PROJECT_NAME}"_MAX_CRITICAL_ISSUE_NUM"
MAX_CRITICAL_ISSUE_NUM=$(eval echo \\${$MAX_CRITICAL_ISSUE_NUM})

SERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"
SERVICE_NAME=$(eval echo \\${$SERVICE_NAME})

EMAIL_LIST=${PROJECT_NAME}"_EMAIL_LIST"
EMAIL_LIST=$(eval echo \\${$EMAIL_LIST})

author_list=`cat $BUILD_LOG | grep "Author"` || true

######### SET ENVIRONMENT VARIABLE ##############
export BASE_DIR=${WORKSPACE}/${GIT_PROJECT_NAME}
export SCRIPTS_HOME=${SHARED_FOLDER}/Scripts
cd ${SCRIPTS_HOME}
chmod 755 *
export PATH=$PATH:`pwd`

####### Build Job ######
cd ${WORKSPACE}

START_TIME=`date +"%m/%d/%Y %T"`
echo "overview->startTime: ${START_TIME}" >> ${BUILD_LOG}
echo "overview->logURL: ${BUILD_URL}" >> ${BUILD_LOG}
echo "overview->serviceType: ${PROJECT_NAME} Service" >> ${BUILD_LOG}

git clone ${GIT_PROJECT_URL}

BUILD_FILE=rpmbuild.sh
cd ${BASE_DIR}
git checkout origin/${BRANCH}

let BUILD_NUMBER=100+${BUILD_NUMBER}
sed -i -r "s/^RELEASE.+/RELEASE=${BUILD_NUMBER}_Dev/g" ${BUILD_FILE}
sh ${BUILD_FILE}

cd ${WORKSPACE}
rm -rf *.rpm
mv `find . -type f -name "*.rpm" | egrep -v \'(\\.src\\.)\' | egrep -v \'BUILD\'` .
RPM=`ls *.rpm`

packageService --addPackages --repository ${REPOSITORY} --packages "${RPM}"

############# Unit test analyze  ###############
UnitTestAnalyzer.pl --path "${BASE_DIR}/.rpm/BUILD" --sufix xml >> ${BUILD_LOG}

################Run Sonar################
cd ${BASE_DIR}
mvn clean install -f ${BASE_DIR}/pom.xml -DPASSWORD=$ENV{PASSWORD} -DUSERNAME=$ENV{USERNAME} -e -B sonar:sonar

################Get project in Sonar ################
sonarProjectName=`sed -ne \'/name/{s/.*<name>\\(.*\\)<\\/name>.*/\\1/p;q;}\' pom.xml`
lowerProjectName=`echo $sonarProjectName | awk \'{print tolower($0)}\'`

sonarId=`queryDB.pl --query="select * from resource_index where kee = \'$lowerProjectName\' order by resource_id;"`
echo $sonarId
sonar_path=http://vm10686.global.xxx.net:9000/dashboard/index/${sonarId}
echo "SonarPath: ${sonar_path}" >> ${BUILD_LOG}

################Function for get Sonar home page info################
##Key for what you want to find
##column to print

function get_html_value() {
html=$1
key=$2
column=$3
echo `cat $html | sed -n \'/\'$key\'/p\' | awk -v column=$column -F "[><]" \'{print $column}\'`
}

##############Sonar issues analyze ###############
curl ${sonar_path} > html.txt
blocker_num=`get_html_value html.txt m_blocker_violations 3`
critical_num=`get_html_value html.txt m_critical_violations 3`
major_num=`get_html_value html.txt m_major_violations 3`
minor_num=`get_html_value html.txt m_minor_violations 3`
info_num=`get_html_value html.txt m_info_violations 3`

echo "SonarIssues->Blocker: ${blocker_num}" >> ${BUILD_LOG}
echo "SonarIssues->Critical: ${critical_num}" >> ${BUILD_LOG}
echo "SonarIssues->Major: ${major_num}" >> ${BUILD_LOG}
echo "SonarIssues->Minor: ${minor_num}" >> ${BUILD_LOG}
echo "SonarIssues->Info: ${info_num}" >> ${BUILD_LOG}

##############Sonar Code Coverage analyze ###############
code_coverage=`get_html_value html.txt m_coverage 7`
code_coverage_integer=`echo $code_coverage | sed -n \'s/%//p\'`

line_coverage=`get_html_value html.txt m_line_coverage 5`
branch_coverage=`get_html_value html.txt m_branch_coverage 5`

echo "CoverageInfo->UnitTestsCoverage: ${code_coverage}" >> ${BUILD_LOG}
echo "CoverageInfo->LineCoverage: ${line_coverage}" >> ${BUILD_LOG}
echo "CoverageInfo->BranchCoverage: ${branch_coverage}" >> ${BUILD_LOG}
echo "CoverageInfo->CoverageThreshold: ${SONAR_MIN_COVERAGE}%"  >> ${BUILD_LOG}

################Send Email Code Coverage < Threshold ################
if (( $(echo "$code_coverage_integer < $SONAR_MIN_COVERAGE" | bc -l) ))
then
mail_subject=${PROJECT_NAME}"_Service_Build Job - Build # "${BUILD_NUMBER}" Failure"\'!\'
 
mail_body="Hi, ${PROJECT_NAME}"" project member"$\'\\n\'$\'\\n\'"Alerts : Unit Tests Coverage: ${code_coverage} < CoverageThreshold: ${SONAR_MIN_COVERAGE}%, the latest build package cannot be allowed to install to service - ${SERVICE_NAME} [$INSTANCES]"\'!\'$\'\\n\'$\'\\n\'"For details please refer to SonarQube - ${sonar_path}"$\'\\n\'$\'\\n\'"The latest code author list as below:"$\'\\n\'"${author_list}"

sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --mailbody "$mail_body" --msg_type "text"
exit 0
fi

################Send Email when Sonar has major above issue################
if [ ${blocker_num} -gt 0 ] || [ ${critical_num} -gt ${MAX_CRITICAL_ISSUE_NUM} ] || [ ${major_num} -gt ${MAX_MAJOR_ISSUE_NUM} ]
then
mail_subject=${PROJECT_NAME}"_Service_Build Job - Build # "${BUILD_NUMBER}" Failure"\'!\'
 
mail_body="Hi, ${PROJECT_NAME}"" project member"$\'\\n\'$\'\\n\'"Alerts : Sonar major above issues != 0, the latest build package cannot be allowed to install to service - ${SERVICE_NAME} [$INSTANCES]"\'!\'$\'\\n\'$\'\\n\'"Blocker issues: ${blocker_num}"$\'\\n\'"Critical issues: ${critical_num}"$\'\\n\'"Major issues: ${major_num}"$\'\\n\'"Minor issues: ${minor_num}"$\'\\n\'"Info issues: ${info_num}"$\'\\n\'$\'\\n\'"For details please refer to SonarQube - ${sonar_path}"$\'\\n\'$\'\\n\'"The latest code author list as below:"$\'\\n\'"${author_list}"

sendEmail.pl --subject "${mail_subject}" --emaillist "${EMAIL_LIST}" --mailbody "$mail_body" --msg_type "text"
exit 0
fi

####### Call Deploy Job ######
curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Deploy/build --data json=\'{"parameter": [{"name":"PROJECT_NAME","value":"\'${PROJECT_NAME}\'"},{"name":"PIPELINE_NUM","value":"\'${PIPELINE_NUM}\'"}]}\'

mvn clean install -f ${BASE_DIR}/pom.xml -DPASSWORD=$ENV{PASSWORD} -DUSERNAME=$ENV{USERNAME} -e -B sonar:sonar

执行Sonar 的一些数据库配置信息，是放在maven/conf/settings.xml文件中的。

另外一点：每一个项目在Sonar里的名称是 项目pom文件的<name></name>标签里的值。依据这个值去数据库里搜出id, 有了这个id就能够直接訪问到Sonar上的这个项目 比方http://vm10686.global.xxx.net:9000/dashboard/index/1234

 

4)Template_Service_Deploy

Build完之后 就是要Deploy了。 将rpm包装到对应Service的instance上。然后重新启动service

rm -rf ${WORKSPACE}/*
    
###### Import config file ######
cd ~/jenkins/common/Config
. Project_Config.txt

###### Initial parameters ######
SERVICE_NAME=${PROJECT_NAME}"_SERVICE_NAME"
SERVICE_NAME=$(eval echo \\${$SERVICE_NAME})

INSTANCES=${PROJECT_NAME}"_INSTANCES"
INSTANCES=$(eval echo \\${$INSTANCES})

PATTERN=${PROJECT_NAME}"_RPM_GREP_PATTERN"
PATTERN=$(eval echo \\${$PATTERN})

LOG_NAME=${PROJECT_NAME}"_BUILD_"${PIPELINE_NUM}".log"
BUILD_LOG=${SHARED_FOLDER}/Log/${PROJECT_NAME}"_"${PIPELINE_NUM}/$LOG_NAME

##########SET ENVIRONMENT VARIABLE##############
SCRIPTS_HOME=~/jenkins/common/Scripts
cd ${SCRIPTS_HOME}
chmod 755 *
export PATH=$PATH:`pwd`

InstallTime=`date +"%m/%d/%Y %T"`
echo "Installation->InstallTime: ${InstallTime}" >> ${BUILD_LOG}

############ Deploy ###############
gridService --add-service --service ${SERVICE_NAME}  --instance ${INSTANCES}
sleep 60
gridService --start-service --service ${SERVICE_NAME}

Endpoint="${INSTANCES}:8080"
echo "Installation->Endpoint: ${Endpoint}" >> ${BUILD_LOG}

echo "Installation->LogURL: ${BUILD_URL}" >> ${BUILD_LOG}

UpdateTime=`date +"%m/%d/%Y %T"`
echo "ServiceInfo->UpdateTime: ${UpdateTime}" >> ${BUILD_LOG}

ServiceInfoProvider.pl --service ${SERVICE_NAME} --pattern ${PATTERN} >> ${BUILD_LOG}

###### Call Polling Job ######
curl -d --user "jadmin:71103407" -s http://vm10686.global.xxx.net:8080/view/Template/job/Template_Service_Polling/build --data json=\'{"parameter": [{"name":"PROJECT_NAM
以上是关于这些年我们一起搞过的持续集成~Jenkins+Perl and Shell script的主要内容，如果未能解决你的问题，请参考以下文章
 
 基于Jenkins持续集成CI
 那些年，加班搞过的需求...
 持续集成高级篇之Jenkins Pipeline 集成sonarqube
 利用Jenkins CI进行持续集成
 如何在Jenkins，Git和Gradle的持续集成环境中处理用户名/密码？ [关闭]
 持续集成开篇之代码发布流程