Build and install Ambari 2.7.5

Posted 2021-03-06 weavepub

源码编译安装 Ambari 2.7.5，记录步骤如下

1.1 安装依赖

官方Build过程：https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development

# 先安装下面依赖
JDK 8
maven 3.3.9 及以上，并配置setting.xml为wmqhealth内网仓库地址
python 2.7

# 安装python setuptools
wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea
sh setuptools-0.6c11-py2.7.egg

# yum安装
yum install -y rpm-build         //
yum install -y gcc-c++           //ambari-logsearch-web模块会报错

1.2 下载和编译

参考：https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.7.5

# 下载
mkdir -p /opt/ambari && cd /opt/ambari
wget https://www-eu.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz
tar xfvz apache-ambari-2.7.5-src.tar.gz
cd apache-ambari-2.7.5-src
mvn versions:set -DnewVersion=2.7.5.0.0

pushd ambari-metrics
mvn versions:set -DnewVersion=2.7.5.0.0
popd

# 编译
mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6"

参数说明

# -B参数：该参数表示让Maven使用批处理模式构建项目，能够避免一些需要人工参与交互而造成的挂起状态。

# 如果第一次编译报错之后,重新编译有问题，则在命令中加上 -Drat.skip=true
# -X：如果看详细debug信息可加上该参数
#[root@master apache-ambari-2.7.5-src]# mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X

#因为会报错，重复执行mvn，把clean去掉可以快不少
#[root@master apache-ambari-2.7.5-src]# mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X

#从某个点开始恢复，就不用把之前build好的再重新来一次了
#如果报错末提示如：mvn <args> -rf :ambari-metrics-storm-sink-legacy
#[root@master apache-ambari-2.7.5-src]# mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X -rf :ambari-web

1.3 编译报错解决

1）ambari-web 报错1：

mvn -rf :ambari-web

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:install-node-and-yarn (install node and yarn) on project ambari-web: Could not extract the Yarn archive: Could not extract archive: '/root/.m2/repository/com/github/eirslett/yarn/0.23.2/yarn-0.23.2./yarn-v0.23.2.tar.gz': EOFException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:install-node-and-yarn (install node and yarn) on project ambari-web: Could not extract the Yarn archive: Could not extract archive: '/root/.m2/repository/com/github/eirslett/yarn/0.23.2/yarn-0.23.2./yarn-v0.23.2.tar.gz'

# 有可能是因为网络问题包没有下载完整，我遇到了几个包都不是完整的，所以直接去外面下载然后覆盖到仓库中
# 删除yarn-v0.23.2.tar.gz手动下载并放于/root/.m2/repository/com/github/eirslett/yarn/0.23.2/yarn-0.23.2./目录下：
https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz

# 重新编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X -rf :ambari-web

2）ambari-web 报错2：

mvn -rf :ambari-web

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:yarn (yarn install) on project ambari-web: Failed to run task: 'yarn install --ignore-engines --pure-lockfile' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

# 这个问题最容易出现，主要是nodejs会通过 yarn去加载依赖包，解决方式是保证网络环境好。
# 重试即可
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-web

3）ambari-web 卡住：

mvn -rf :ambari-web

[INFO] Downloading https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2
[INFO] Saving to /tmp/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2
[INFO] Receiving...

# 卡在这一步半小时，进该目录发现还下载了 0Kb，手动下载该文件并存于/tmp/phantomjs/目录下。
# 继续编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-web

4）ambari-metrics-timelineservice 报错：

mvn -rf :ambari-metrics-timelineservice

[ERROR] Failed to execute goal on project ambari-metrics-timelineservice: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-timelineservice:jar:2.7.5.0.0: Failed to collect dependencies at org.apache.phoenix:phoenix-core:jar:5.0.0.3.1.4.0-315 -> org.apache.hbase:hbase-mapreduce:jar:2.0.2.3.1.4.0-315 -> org.apache.hbase:hbase-server:jar:2.0.2.3.1.4.0-315 -> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 -> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Failed to read artifact descriptor for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to thirdparty (http://nexus.wmqhealth.com/repository/thirdparty/): Transfer failed for http://nexus.wmqhealth.com/repository/thirdparty/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom 400 Repository version policy: RELEASE does not allow version: 3.0.1-b06-SNAPSHOT -> [Help 1]

# 解决
# 增加华为的maven镜像仓库，修改setting.xml
<settings>
  <mirrors>
    <mirror>
      <id>huaweicloud</id>
      <mirrorOf>*</mirrorOf>
      <url>https://mirrors.huaweicloud.com/repository/maven/</url>
    </mirror>
  </mirrors>
</settings>

# 删除
find ~/.m2/repository/ -name "*.lastUpdated" -exec rm -rf {} ;

# 重新编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-metrics-timelineservice

5）手动下载hbase，phoenix：

Download HBase:
    [mkdir] Created dir: /opt/ambari/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded
      [get] Getting: https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz
      [get] To: /opt/ambari/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/hbase.tar.gz


# 下载hbase.tar.gz特别慢，手动下载并存于/opt/ambari/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/目录下。
mv hbase-2.0.2.3.1.4.0-315-bin.tar.gz hbase.tar.gz

# 继续编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-metrics-timelineservice

Download Phoenix:
      [get] Getting: https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/phoenix/phoenix-5.0.0.3.1.4.0-315.tar.gz
      [get] To: /opt/ambari/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/phoenix.tar.gz

# 下载慢，手动下载，并重命名
mv phoenix-5.0.0.3.1.4.0-315.tar.gz phoenix.tar.gz
# 继续编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-metrics-timelineservice

6）手动下载hadoop：

[INFO] -------------< org.apache.ambari:ambari-metrics-assembly >--------------
[INFO] Building Ambari Metrics Assembly 2.7.5.0.0                        [5/29]

Downloading Hadoop:
    [mkdir] Created dir: /opt/ambari/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-assembly/target/embedded
      [get] Getting: https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz
      [get] To: /opt/ambari/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-assembly/target/embedded/hadoop.tar.gz

# 下载慢，手动下载，并重命名
mv hadoop-3.1.1.3.1.4.0-315.tar.gz hadoop.tar.gz
# 继续编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-metrics-assembly

7）ambari-logsearch-web报错：

[INFO] --- frontend-maven-plugin:1.6:yarn (yarn install) @ ambari-logsearch-web ---
[INFO] Running 'yarn install --ignore-engines --pure-lockfile' in /opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build
[INFO] yarn install v1.1.0
[INFO] [1/4] Resolving packages...
[INFO] [2/4] Fetching packages...
[INFO] info fsevents@1.1.1: The platform "linux" is incompatible with this module.
[INFO] info "fsevents@1.1.1" is an optional dependency and failed compatibility check. Excluding it from installation.
[INFO] [3/4] Linking dependencies...
[ERROR] warning "@angular/animations@4.4.6" has incorrect peer dependency "@angular/core@4.4.6".
[ERROR] warning "@angular/core@4.4.3" has unmet peer dependency "zone.js@^0.8.4".
[ERROR] warning "angular2-notifications@0.4.46" has incorrect peer dependency "@angular/core@^2.1.2".
[ERROR] warning "angular2-notifications@0.4.46" has incorrect peer dependency "@angular/common@^2.1.2".
[ERROR] warning "angular2-notifications@0.4.46" has incorrect peer dependency "@angular/platform-browser@^2.1.1".
[ERROR] warning "angular2-notifications@0.4.46" has incorrect peer dependency "rxjs@5.0.0-beta.12".
[ERROR] warning "license-webpack-plugin@1.0.0" has unmet peer dependency "webpack-sources@>= 1.0.0".
[INFO] [4/4] Building fresh packages...

# 卡在这里半小时了也没动静。
# 进入 /opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build 目录手动执行 yarn install --ignore-engines --pure-lockfile 命令

# 会输出报错信息如下：
[4/4] Building fresh packages...
[-/3] ? waiting...
[2/3] ? node-sass
warning Error running install script for optional dependency: "/opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-sass: Command failed.
Exit code: 1
Command: node scripts/build.js
Arguments: 
Directory: /opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-sass
Output:
Building: /usr/bin/node /opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
gyp info it worked if it ends with ok
gyp verb cli [ '/usr/bin/node',
gyp verb cli   '/opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-gyp/bin/node-gyp.js',
gyp verb cli   'rebuild',
gyp verb cli   '--verbose',
gyp verb cli   '--libsass_ext=',
gyp verb cli   '--libsass_cflags=',
gyp verb cli   '--libsass_ldflags=',
gyp verb cli   '--libsass_library=' ]
gyp info using node-gyp@3.6.2
gyp info using node@10.16.3 | linux | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb `which` succeeded python2 /bin/python2
gyp verb check python version `/bin/python2 -c "import platform; print(platform.python_version());"` returned: "2.7.5\n"
gyp verb get node dir no --target version specified, falling back to host node version: 10.16.3
gyp verb command install [ '10.16.3' ]
gyp verb install input version string "10.16.3"
gyp verb install installing version: 10.16.3
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version not already installed, continuing with install 10.16.3
gyp verb ensuring nodedir is created /root/.node-gyp/10.16.3
gyp verb created nodedir /root/.node-gyp/10.16.3
gyp http GET https://nodejs.org/download/release/v10.16.3/node-v10.16.3-headers.tar.gz
gyp WARN install got an error, rolling back install
gyp verb command remove [ '10.16.3' ]
gyp verb remove using node-gyp dir: /root/.node-gyp
gyp verb remove removing target version: 10.16.3
gyp verb remove removing development files for version: 10.16.3
gyp ERR! configure error 
gyp ERR! stack Error: This is most likely not a problem with node-gyp or the package itself and
gyp ERR! stack is related to network connectivity. In most cases you are behind a proxy or have bad 
gyp ERR! stack network settings.
gyp ERR! stack     at Request.<anonymous> (/opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-gyp/lib/install.js:193:21)
gyp ERR! stack     at Request.emit (events.js:198:13)
gyp ERR! stack     at Request.onRequestError (/opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/request/request.js:884:8)
gyp ERR! stack     at ClientRequest.emit (events.js:198:13)
gyp ERR! stack     at TLSSocket.socketErrorListener (_http_client.js:392:9)
gyp ERR! stack     at TLSSocket.emit (events.js:198:13)
gyp ERR! stack     at emitErrorNT (internal/streams/destroy.js:91:8)
gyp ERR! stack     at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
gyp ERR! stack     at process._tickCallback (internal/process/next_tick.js:63:19)
gyp ERR! System Linux 3.10.0-957.21.3.el7.x86_64
gyp ERR! command "/usr/bin/node" "/opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
gyp ERR! cwd /opt/ambari/apache-ambari-2.7.5-src/ambari-logsearch/ambari-logsearch-web/target/webapp-build/node_modules/node-sass
gyp ERR! node -v v10.16.3
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok 
[-/3] ? waiting...
[-/3] ? waiting...
[3/3] ? phantomjs-prebuilt

# 提示的是 node-gyp configure 报错
# 解决：移除 $HOME/.node-gyp，然后重新安装最新的 node-gyp 包
mv /root/.node-gyp /root/.node-gyp-bk
npm i node-gyp -g

# 删除
find ~/.m2/repository/ -name "*.lastUpdated" -exec rm -rf {} ;

# 重新编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-logsearch-web

# 接着报错
[ERROR] PhantomJS not found on PATH
[ERROR] Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
[ERROR] Saving to /tmp/logsearch_npm_config_tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
[ERROR] Receiving...
[ERROR] 
[ERROR] Error making request.
[ERROR] Error: connect ETIMEDOUT 52.216.16.232:443
[ERROR]     at Object._errnoException (util.js:1019:11)
[ERROR]     at _exceptionWithHostPort (util.js:1041:20)
[ERROR]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1175:14)
[ERROR] 
[ERROR] Please report this full log at https://github.com/Medium/phantomjs

# cp -a /root/.node-gyp-bk/* /root/.node-gyp/
# 手动下载 phantomjs-2.1.1，放在/tmp/logsearch_npm_config_tmp/phantomjs/目录下
# 继续重新编译
mvn -B install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-logsearch-web

1.4 编译成功

[INFO] Ambari Logsearch Web ............................... SUCCESS [05:16 min]
[INFO] Ambari Logsearch Server ............................ SUCCESS [ 32.776 s]
[INFO] Ambari Logsearch Assembly .......................... SUCCESS [  0.450 s]
[INFO] Ambari Logsearch Integration Test .................. SUCCESS [ 23.482 s]
[INFO] ambari-infra ....................................... SUCCESS [  0.352 s]
[INFO] Ambari Infra Solr Client ........................... SUCCESS [  7.718 s]
[INFO] Ambari Infra Solr Plugin ........................... SUCCESS [ 10.133 s]
[INFO] Ambari Infra Manager ............................... SUCCESS [ 23.206 s]
[INFO] Ambari Infra Assembly .............................. SUCCESS [  0.327 s]
[INFO] Ambari Infra Manager Integration Tests ............. SUCCESS [ 12.458 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:07 min
[INFO] Finished at: 2020-03-09T11:42:28+08:00
[INFO] ------------------------------------------------------------------------

生成的rpm包如下：

# 服务端
/opt/ambari/apache-ambari-2.7.5-src/ambari-server/target/rpm/ambari-server/RPMS/x86_64/
ambari-server-2.7.5.0-0.x86_64.rpm

# 客户端
/opt/ambari/apache-ambari-2.7.5-src/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/
mbari-agent-2.7.5.0-0.x86_64.rpm

1.5 安装 Ambari Server

Install the rpm package from ambari-server/target/rpm/ambari-server/RPMS/noarch/

# 安装
cd /opt/ambari/apache-ambari-2.7.5-src/ambari-server/target/rpm/ambari-server/RPMS/x86_64/
yum install ambari-server*.rpm

# 配置
ambari-server setup

#
Database admin user (postgres):
Enter choice (1): 1
Database admin user (postgres): 
Database name (ambari): 
Postgres schema (ambari): 
Username (ambari): 
Enter Database Password (bigdata):

# 启动
ambari-server start

# 访问
ip:8080
admin/admin

1.6 安装

登入后，点击 launch install wizard 启动安装向导

查看版本的时候为空。

从Ambari 2.7.5版本开始，访问Ambari存储库需要进行身份验证。 要访问二进制文件，您必须首先具有必需的身份验证凭据（用户名和密码）。Cloudera向已注册的支持联系人发送的电子邮件中提供了新客户和合作伙伴的身份验证凭据。 现有用户可以在支持门户（https://my.cloudera.com）上提交非技术案例以获取凭证。以前，Ambari存储库位于AWS S3上。 自HDP 3.1.5 / Ambari 2.7.5起，存储库已移至https://archive.cloudera.com

#1. 访问下面网址进行注册
https://sso.cloudera.com/register.html

参考：https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ch_Installing_Ambari.html