hudi源码编译

Posted 2021-09-13 奇妙的门木君

环境&源码准备

git 源码

登陆https://github.com/apache/hudi

git clone https://github.com/apache/hudi.git

代码编译

可用的maven settings设置（还是会缺少一些包）

 <mirrors> 
    <!-- mirror | Specifies a repository mirror site to use instead of a given 
      repository. The repository that | this mirror serves has an ID that matches 
      the mirrorOf element of this mirror. IDs are used | for inheritance and direct 
      lookup purposes, and must be unique across the set of mirrors. | --> 
    <mirror>
    <id>aliyunmaven</id>
    <mirrorOf>*</mirrorOf>
    <name>spring-plugin</name>
    <url>https://maven.aliyun.com/repository/spring-plugin</url>
 </mirror>

  <mirror>
    <id>central</id>
    <name>Maven Repository Switchboard</name>
    <url>https://repo1.maven.org/maven2/</url>
    <mirrorOf>central</mirrorOf>
</mirror>


  </mirrors> 

准备结束以后可以开始编译

mvn clean package -DskipTests

下面说一说编译遇到的问题
主要还是缺jar包,个人都是手工maven仓库里下载

启动命令

/usr/local/spark/bin/spark-shell \\
  --jars `ls /Users/##/SourceCode/hudi/hudi/packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-*.*.*-SNAPSHOT.jar` \\