Apache Atlas 2.1.0编译打包安装记录
Posted 终回首
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Apache Atlas 2.1.0编译打包安装记录相关的知识，希望对你有一定的参考价值。
目录
一、Apache Atlas简介
Atlas 是一个可伸缩和可扩展的核心基础数据治理服务集合 ，使企业能够有效地和高效地满足 Hadoop 中的合规性要求，并允许与整个企业数据生态系统的集成。
项目地址：
https://github.com/apache/atlas
详细介绍:
https://blog.csdn.net/qq_38247150/article/details/108756790
二、编译打包
版本
Apache Atlas 2.1.0
Apache Maven 3.6.3
JDK 1.8
Python 2.7.18
Ubuntu 20.04.2 LTS
注意：
不建议使用win10打包！！！
语言方面jdk必须是1.8！！！
python必须是2.7！！！
笔者使用的是win10的WSL2打包的
Apache Atlas不提供安装包，需要自己下载源码打包。
1 打开官网下载源码
官网地址：
https://atlas.apache.org/2.1.0/index#/Downloads
笔者这里下载2.1.0的source
wget https://www.apache.org/dyn/closer.cgi/atlas/2.1.0/apache-atlas-2.1.0-sources.tar.gz
2 解压
tar -zxvf apache-atlas-2.1.0-sources.tar.gz
3 打包
3.1 打包前的准备
# 进入打包后的目录
cd apache-atlas-sources-2.1.0
# 设置maven打包进程的jvm参数。（-Xms表示初始化JAVA堆的大小及该进程刚创建出来的时候，他的专属JAVA堆的大小，一旦对象容量超过了JAVA堆的初始容量，JAVA堆将会自动扩容到-Xmx大小）
export MAVEN_OPTS="-Xms3g -Xmx3g"
3.2 执行打包命令
如果环境中已经有装好Apache Hbase和Apache Solr，执行这条
mvn clean -DskipTests package -Pdist
如果环境中没有Hbase和Solr，希望使用Atlas内置的Hbase和Solr，执行这条；
笔者这里没有hadoop环境，只好使用这条命令了。
mvn clean -DskipTests package -Pdist,embedded-hbase-solr
如果希望使用Atlas内置的Apache Cassandra 和 Apache Solr，执行这条命令
mvn clean package -Pdist,embedded-cassandra-solr
打包过程要下很多jar包和node安装包，速度快慢看网络速度，网络慢或者有波动的话下载会中断，需要要多重试几次。
打包过程中会下载node安装包，如果有代理可以手动提前下载好放到指定目录，这样会快不少。
[INFO] Installing node version v12.16.0
[INFO] Downloading https://nodejs.org/dist/v12.16.0/node-v12.16.0-linux-x64.tar.gz to /root/.m2/repository/com/github/eirslett/node/12.16.0/node-12.16.0-linux-x64.tar.gz
[INFO] No proxies configured
[INFO] No proxy was configured, downloading directly
3.3 打包报错
Too many files with unapproved license
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (rat-check) on project apache-atlas: Too many files with unapproved license: 1274 See RAT report in: /opt/soft/apache-atlas-sources-2.1.0/target/rat.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :apache-atlas
检查license报错，添加参数-Drat.skip=true跳过license检查。再次执行命令
# 修改前命令
mvn clean -DskipTests package -Pdist,embedded-hbase-solr
# 修改后命令
mvn clean -DskipTests package -Pdist,embedded-hbase-solr -Drat.skip=true
最后，终于打包成功了
[INFO] Apache Atlas Server Build Tools 1.0 ................ SUCCESS [ 0.511 s]
[INFO] apache-atlas 2.1.0 ................................. SUCCESS [ 1.734 s]
[INFO] Apache Atlas Test Utility Tools 2.1.0 .............. SUCCESS [ 2.081 s]
[INFO] Apache Atlas Integration 2.1.0 ..................... SUCCESS [ 3.208 s]
[INFO] Apache Atlas Common 2.1.0 .......................... SUCCESS [ 1.013 s]
[INFO] Apache Atlas Client 2.1.0 .......................... SUCCESS [ 0.079 s]
[INFO] atlas-client-common 2.1.0 .......................... SUCCESS [ 0.391 s]
[INFO] atlas-client-v1 2.1.0 .............................. SUCCESS [ 0.582 s]
[INFO] Apache Atlas Server API 2.1.0 ...................... SUCCESS [ 0.819 s]
[INFO] Apache Atlas Notification 2.1.0 .................... SUCCESS [ 1.437 s]
[INFO] atlas-client-v2 2.1.0 .............................. SUCCESS [ 0.312 s]
[INFO] Apache Atlas Graph Database Projects 2.1.0 ......... SUCCESS [ 0.036 s]
[INFO] Apache Atlas Graph Database API 2.1.0 .............. SUCCESS [ 0.481 s]
[INFO] Graph Database Common Code 2.1.0 ................... SUCCESS [ 0.443 s]
[INFO] Apache Atlas JanusGraph-HBase2 Module 2.1.0 ........ SUCCESS [ 0.574 s]
[INFO] Apache Atlas JanusGraph DB Impl 2.1.0 .............. SUCCESS [ 2.195 s]
[INFO] Apache Atlas Graph DB Dependencies 2.1.0 ........... SUCCESS [ 0.749 s]
[INFO] Apache Atlas Authorization 2.1.0 ................... SUCCESS [ 0.772 s]
[INFO] Apache Atlas Repository 2.1.0 ...................... SUCCESS [ 4.489 s]
[INFO] Apache Atlas UI 2.1.0 .............................. SUCCESS [ 35.856 s]
[INFO] Apache Atlas New UI 2.1.0 .......................... SUCCESS [ 22.718 s]
[INFO] Apache Atlas Web Application 2.1.0 ................. SUCCESS [ 36.700 s]
[INFO] Apache Atlas Documentation 2.1.0 ................... SUCCESS [ 0.569 s]
[INFO] Apache Atlas FileSystem Model 2.1.0 ................ SUCCESS [ 0.709 s]
[INFO] Apache Atlas Plugin Classloader 2.1.0 .............. SUCCESS [ 0.455 s]
[INFO] Apache Atlas Hive Bridge Shim 2.1.0 ................ SUCCESS [ 0.919 s]
[INFO] Apache Atlas Hive Bridge 2.1.0 ..................... SUCCESS [ 2.675 s]
[INFO] Apache Atlas Falcon Bridge Shim 2.1.0 .............. SUCCESS [ 0.396 s]
[INFO] Apache Atlas Falcon Bridge 2.1.0 ................... SUCCESS [ 0.789 s]
[INFO] Apache Atlas Sqoop Bridge Shim 2.1.0 ............... SUCCESS [ 0.054 s]
[INFO] Apache Atlas Sqoop Bridge 2.1.0 .................... SUCCESS [ 2.056 s]
[INFO] Apache Atlas Storm Bridge Shim 2.1.0 ............... SUCCESS [ 0.160 s]
[INFO] Apache Atlas Storm Bridge 2.1.0 .................... SUCCESS [ 1.137 s]
[INFO] Apache Atlas Hbase Bridge Shim 2.1.0 ............... SUCCESS [ 0.586 s]
[INFO] Apache Atlas Hbase Bridge 2.1.0 .................... SUCCESS [ 1.855 s]
[INFO] Apache HBase - Testing Util 2.1.0 .................. SUCCESS [ 1.284 s]
[INFO] Apache Atlas Kafka Bridge 2.1.0 .................... SUCCESS [ 0.795 s]
[INFO] Apache Atlas classification updater 2.1.0 .......... SUCCESS [ 0.330 s]
[INFO] Apache Atlas Impala Hook API 2.1.0 ................. SUCCESS [ 0.045 s]
[INFO] Apache Atlas Impala Bridge Shim 2.1.0 .............. SUCCESS [ 0.049 s]
[INFO] Apache Atlas Impala Bridge 2.1.0 ................... SUCCESS [ 1.668 s]
[INFO] Apache Atlas Distribution 2.1.0 .................... SUCCESS [01:42 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:56 min
[INFO] Finished at: 2021-07-23T10:49:28+08:00
[INFO] ------------------------------------------------------------------------
打包结果
打包成功后会生成如下文件
root@company:/opt/soft/apache-atlas-sources-2.1.0/distro/target# pwd
/opt/soft/apache-atlas-sources-2.1.0/distro/target
root@company:/opt/soft/apache-atlas-sources-2.1.0/distro/target# ll
total 2080588
drwxr-xr-x 24 root root 4096 Jul 23 10:49 ./
drwxr-xr-x 6 root staff 4096 Jul 23 10:47 ../
-rw-r--r-- 1 root root 30 Jul 23 10:47 .plxarc
drwxr-xr-x 2 root root 4096 Jul 23 10:47 META-INF/
drwxr-xr-x 2 root root 4096 Jul 23 10:47 antrun/
drwxr-xr-x 3 root root 4096 Jul 23 10:49 apache-atlas-2.1.0-bin/
-rw-r--r-- 1 root root 708347094 Jul 23 10:49 apache-atlas-2.1.0-bin.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:49 apache-atlas-2.1.0-classification-updater/
-rw-r--r-- 1 root root 28455 Jul 23 10:49 apache-atlas-2.1.0-classification-updater.zip
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-falcon-hook/
-rw-r--r-- 1 root root 9266781 Jul 23 10:48 apache-atlas-2.1.0-falcon-hook.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-hbase-hook/
-rw-r--r-- 1 root root 11177405 Jul 23 10:48 apache-atlas-2.1.0-hbase-hook.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-hive-hook/
-rw-r--r-- 1 root root 11265413 Jul 23 10:48 apache-atlas-2.1.0-hive-hook.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-impala-hook/
-rw-r--r-- 1 root root 11227674 Jul 23 10:48 apache-atlas-2.1.0-impala-hook.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-kafka-hook/
-rw-r--r-- 1 root root 9277583 Jul 23 10:48 apache-atlas-2.1.0-kafka-hook.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-server/
-rw-r--r-- 1 root root 608395302 Jul 23 10:48 apache-atlas-2.1.0-server.tar.gz
-rw-r--r-- 1 root root 14224122 Jul 23 10:49 apache-atlas-2.1.0-sources.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-sqoop-hook/
-rw-r--r-- 1 root root 9257171 Jul 23 10:48 apache-atlas-2.1.0-sqoop-hook.tar.gz
drwxr-xr-x 3 root root 4096 Jul 23 10:48 apache-atlas-2.1.0-storm-hook/
-rw-r--r-- 1 root root 59010373 Jul 23 10:48 apache-atlas-2.1.0-storm-hook.tar.gz
drwxr-xr-x 2 root root 4096 Jul 23 10:48 archive-tmp/
-rw-r--r-- 1 root root 678902100 Jul 23 10:48 atlas-distro-2.1.0.jar
drwxr-xr-x 2 root root 4096 Jul 23 10:47 bin/
drwxr-xr-x 5 root root 4096 Jul 23 10:47 conf/
drwxr-xr-x 7 root root 4096 Jul 23 10:47 hbase/
drwxr-xr-x 3 root root 4096 Jul 23 10:47 hbase.temp/
drwxr-xr-x 2 root root 4096 Jul 23 10:47 maven-archiver/
drwxr-xr-x 3 root root 4096 Jul 23 10:47 maven-shared-archive-resources/
drwxr-xr-x 9 root root 4096 Jul 23 10:47 solr/
drwxr-xr-x 3 root root 4096 Jul 23 10:47 solr.temp/
drwxr-xr-x 3 root root 4096 Jul 23 10:47 test-classes/
其中有用的是这2个
apache-atlas-2.1.0-bin
apache-atlas-2.1.0-bin.tar.gz
三、安装
如果要部署atlas的服务器不是当前服务器，可以将压缩包apache-atlas-2.1.0-bin.tar.gz复制到目标服务器；
如果要部署的服务器就是当前服务器，那么把apache-atlas-2.1.0-bin目录拷贝到要安装的目录即可。笔者这里就是当前服务器
1 移动到安装目录
mv apache-atlas-2.1.0-bin /opt/soft
2 启动服务
cd /opt/soft/apache-atlas-2.1.0-bin/apache-atlas-2.1.0/
bin/atlas_start.py
3 启动成功，但没有进程
查看进程和端口发现hbase和solr并没有启动
运行官网提供的验证是否启动成功的命令，提示failed
curl -u admin:admin http://localhost:21000/api/atlas/admin/version
尝试访问登录页面，发现无法访问，至此可以确认Atlas没有启动成功
停掉Atlas
bin/atlas_stop.py
4 单独启动
4.1 启动Hbase
# 进入hbase启动脚本所在目录
cd ./hbase/bin
sh start-hbase.sh
提示输入密码
输入密码，按Enter
此时页面已经可以打开了
master(http://localhost:61530/rs-status)
regionserver(http://localhost:61510/master-status)
4.2 启动Solr
# 回到项目跟目录
cd -
# 进入solr启动脚本所在目录
cd ./solr/bin
./solr start -c -z localhost:2181 -p 9838 -force
此时页面已经可以打开了
solr地址（http://localhost:9838/solr/#/）
4.3 启动Atlas
4.3.1 启动Atlas后台报错
cd -
bin/atlas_start.py
启动未报错，后台日志./logs/application.log中有报错
Can not find the specified config set: vertex_index
2021-07-23 12:23:05,163 ERROR - [main:] ~ GraphBackedSearchIndexer.initialize() failed (GraphBackedSearchIndexer:367)
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://192.168.50.16:9838/solr: Can not find the specified config set: vertex_index
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at org.janusgraph.diskstorage.solr.Solr6Index.createCollectionIfNotExists(Solr6Index.java:1182)
at org.janusgraph.diskstorage.solr.Solr6Index.register(Solr6Index.java:376)
at org.janusgraph.diskstorage.indexing.IndexTransaction.register(IndexTransaction.java:96)
at org.janusgraph.graphdb.database.IndexSerializer.register(IndexSerializer.java:108)
at org.janusgraph.graphdb.database.management.ManagementSystem.addIndexKey(ManagementSystem.java:657)
at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphManagement.addMixedIndex(AtlasJanusGraphManagement.java:204)
4.3.2 解决后台报错
尝试手动创建三个collection：vertex_index、edge_index、fulltext_index
solr管理页面
http://localhost:9838/solr/#/
下面是添加一个Collection的操作步骤，依次添加3个collection
回到根目录，再次启动Atlas
bin/atlas_start.py
再打开一个ssh窗口用来看log
tail -f 100 ./logs/application.log
发现后台没有再报错
等待了大概几分钟后打开登录页面 http://localhost:21000/login.jsp
终于看到了盼望已久的Atlas美观大方的登录页面
默认用户名密码是 admin/admin
输入用户名密码
再次验证是否成功运行的命令，发现返回值已经正常
curl -u admin:admin http://localhost:21000/api/atlas/admin/version
{"Description":"Metadata Management and Data Governance Platform over Hadoop","Revision":"release","Version":"2.1.0","Name":"apache-atlas"}root@company:/opt/soft/apache-atlas-2.1.0-bin/apache-atlas-2.1.0/logs
至此已安装成功，导入官方案例数据跑个helloworld试试看
四、官方案例
官方提供了一些样例数据
类型定义样例
实体样例图
1 运行导入脚本
bin/quick_start.py
提示输入Atlas用户名密码，依次输入
admin
admin
日志
Enter username for atlas :- admin
Enter password for atlas :-
Creating sample types:
Created type [DB]
Created type [Table]
Created type [StorageDesc]
Created type [Column]
Created type [LoadProcess]
Created type [LoadProcessExecution]
Created type [View]
Created type [JdbcAccess]
Created type [ETL]
Created type [Metric]
Created type [PII]
Created type [Fact]
Created type [Dimension]
Created type [Log Data]
Created type [Table_DB]
Created type [View_DB]
Created type [View_Tables]
Created type [Table_Columns]
Created type [Table_StorageDesc]
Creating sample entities:
Created entity of type [DB], guid: e30002b4-56fe-43ed-9a00-fea9a26b51ef
Created entity of type [DB], guid: 96d7ee43-6e92-44b5-aa0e-e908a77528ae
Created entity of type [DB], guid: ea352c2f-1e3b-4726-9946-c2c465ad7b43
Created entity of type [Table], guid: 9623e3ee-e719-4ab9-b28e-b8b1364c43b6
Created entity of type [Table], guid: 6964f753-dde2-4055-8a20-592a94944234
Created entity of type [Table], guid: 8a4b6a8f-1a76-4926-8c99-a8238cfd5968
Created entity of type [Table], guid: fd21256c-71e4-4566-b6dc-2cc142ac9268
Created entity of type [Table], guid: ca8dca65-761a-455f-934d-02a743580ea5
Created entity of type [Table], guid: 67c57368-d363-436b-bbde-5f4227454bad
Created entity of type [Table], guid: 960ad166-5b2a-4b35-b4d4-d6b5a04af733
Created entity of type [Table], guid: 96153ee0-9a40-440b-8ade-074e2d74a9b1
Created entity of type [View], guid: 8edccae2-0d28-4c28-9c58-d78da72bde25
Created entity of type [View], guid: 1c7a5891-4de3-4a4e-9abf-51c92f954951
Created entity of type [LoadProcess], guid: e59a7d24-fc29-4b60-9be2-666fc78bea35
Created entity of type [LoadProcessExecution], guid: 6040b64b-5485-43e0-9f81-eeb6158a1b1d
Created entity of type [LoadProcessExecution], guid: 34da9320-5e0d-48f0-a8e0-07b2a41eafaf
Created entity of type [LoadProcess], guid: 2461bd99-1149-4027-8580-3cf5e228d913
Created entity of type [LoadProcessExecution], guid: a0f39e81-14e9-47b1-a198-ffcbbb2ef156
Created entity of type [LoadProcessExecution], guid: 0b5904fa-463e-42cd-9ebb-cd638a4576e8
Created entity of type [LoadProcess], guid: 7f1cb984-3bbd-4c51-9d91-bb6ae9df6bad
Created entity of type [LoadProcessExecution], guid: a6e6a0c6-53cd-4c81-b391-48174f29645a
Created entity of type [LoadProcessExecution], guid: c38056ba-c824-45d2-98a7-190e8e865573
Sample DSL Queries:
query [from DB] returned [3] rows.
query [DB] returned [3] rows.
query [DB where name=%22Reporting%22] returned [1] rows.
query [DB where name=%22encode_db_name%22] returned [ 0 ] rows.
query [Table where name=%2522sales_fact%2522] returned [1] rows.
query [DB where name="Reporting"] returned [1] rows.
query [DB where DB.name="Reporting"] returned [1] rows.
query [DB name = "Reporting"] returned [1] rows.
query [DB DB.name = "Reporting"] returned [1] rows.
query [DB where name="Reporting" select name, owner] returned [1] rows.
query [DB where DB.name="Reporting" select name, owner] returned [1] rows.
query [DB has name] returned [3] rows.
query [DB where DB has name] returned [3] rows.
query [DB is JdbcAccess] returned [ 0 ] rows.
query [from Table] returned [8] rows.
query [Table] returned [8] rows.
query [Table is Dimension] returned [5] rows.
query [Column where Column isa PII] returned [3] rows.
query [View is Dimension] returned [2] rows.
query [Column select Column.name] returned [10] rows.
query [Column select name] returned [9] rows.
query [Column where Column.name="customer_id"] returned [1] rows.
query [from Table select Table.name] returned [8] rows.
query [DB where (name = "Reporting")] returned [1] rows.
query [DB where DB is JdbcAccess] returned [ 0 ] rows.
query [DB where DB has name] returned [3] rows.
query [DB as db1 Table where (db1.name = "Reporting")] returned [ 0 ] rows.
query [Dimension] returned [9] rows.
query [JdbcAccess] returned [2] rows.
query [ETL] returned [10] rows.
query [Metric] returned [4] rows.
query [PII] returned [3] rows.
query [`Log Data`] returned [4] rows.
query [Table where name="sales_fact", columns] returned [4] rows.
query [Table where name="sales_fact", columns as column select column.name, column.dataType, column.comment] returned [4] rows.
query [from DataSet] returned [10] rows.
query [from Process] returned [3] rows.
Sample Lineage Info:
time_dim(Table) -> loadSalesDaily(LoadProcess)
sales_fact_daily_mv(Table) -> loadSalesMonthly(LoadProcess)
loadSalesDaily(LoadProcess) -> sales_fact_daily_mv(Table)
sales_fact(Table) -> loadSalesDaily(LoadProcess)
loadSalesMonthly(LoadProcess) -> sales_fact_monthly_mv(Table)
Sample data added to Apache Atlas Server.
导入完成，可以看到导入了一些type（类型），entity（实体），DSL Queries（查询），Lineage Info（关系线）
2 查看样例数据
打开官网 http://192.168.50.16:21000/
点击CLASSIFICITION
3 UI功能介绍
打开页面就能看到一共三块SEARCH(搜索)、CLASSIFICITION（分类）、GLOSSARY术语表
3.1 SEARCH(搜索)
搜索分两种模式Basic（基础）、Advance（进阶）
基础只能根据类型、分类、术语、文本搜索元数据
进阶可以根据类型、DSL查询语言搜索元数据
3.2 CLASSIFICITION（分类）
这里感觉包括了UI至少60%的功能，包括属性、血缘、分类、标签的查看
GLOSSARY术语表
根据术语搜索
吐槽
- 安装过程摩擦力太大，太不平滑
- Apache Atlas官方文档更新有点不及时，官网的话就只能看看原理类的，操作类有一些已经过时了
- 用户手册几乎没有，全靠自己摸索
参考资料
Apache Atlas 官方文档 https://atlas.apache.org/#/
六子大数据 大佬的博客 https://blog.csdn.net/qq_38247150/article/details/108756790
Y_尘 大佬的博客 https://blog.csdn.net/Y_anger/article/details/105514126
PS:有误之处，请不吝赐教！
以上是关于Apache Atlas 2.1.0编译打包安装记录的主要内容，如果未能解决你的问题，请参考以下文章