Mac搭建Hadoop源码阅读环境

Posted 2020-08-27 骁枫

1、本次Hadoop源码阅读环境使用的阅读工具是idea，Hadoop版本是2.7.3。需要安装的工具包括idea、jdk、maven、protobuf等

2、jdk，使用的版本是1.8版，在jdk官网下载jdk-8u111-macosx-x64.dmg，点击安装，一路next。

3、idea安装，略

4、maven，使用的版本是3.3.9,下载apache-maven-3.3.9-bin.tar,解压：

tar -zxvf  apache-maven-3.3.9-bin.tar

进入 Maven 安装根目录 cd apache-maven-3.3.9

进入 Maven配置文件目录 cd conf

编辑settings.xml文件 vi  settings.xml

设置Maven本地库的路径：这个路径要创建好。

 <localRepository>/opt/PRG/apache-maven-3.3.9/repository</localRepository>

同时在加入阿里云的Maven库，参考：maven使用阿里云仓库

<mirror>
          <id>nexus-aliyun</id>
          <mirrorOf>*</mirrorOf>
          <name>Nexus aliyun</name>
          <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

设置Maven环境变量：

vi ~/.bash_profile

export MAVEN_HOME=/opt/PRG/apache-maven-3.3.9
export PATH=${PATH}:${MAVEN_HOME}/bin:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
export PATH

更新环境变量 source ~/.bash_profile

mvn -version 能看到相应的版本，则Maven安装成功。

5、安装protobuf，下载protobuf-2.5.0.tar.gz，下载地址： http://pan.baidu.com/s/1pJlZubT （网友提供）

解压：tar -zxvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

依次执行，参考：http://blog.sina.com.cn/s/blog_7d1531ed0101fmc5.html

 1 ./configure
 2 make
 3 make check
 4 make install

如果报没有权限的错误，su root 切换到root执行。

最后输入  protoc --version命令，如显示libprotoc 2.5.0则安装成功

 6、编译Hadoop源码

在Hadoop官网下载hadoop-2.7.3-src.tar.gz源码

解压：tar -zxvf  hadoop-2.7.3-src.tar.gz

cd hadoop-2.7.3-src

执行：

mvn package -Pdist -DskipTests -Dtar

最后是漫长的编译过程。。。

下载依赖jar和相关文件。。。

最后输出：[INFO] Reactor Summary：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  5.429 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  2.148 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.061 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.297 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.200 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.627 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.751 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.644 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  6.750 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.126 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:10 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  5.191 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:12 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.037 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:38 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 34.932 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 13.482 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.823 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.039 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.041 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 28.981 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 22.892 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.028 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  7.887 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [  9.785 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.782 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.300 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.558 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.048 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  4.785 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  3.260 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.024 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.129 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.734 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.026 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  4.279 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.812 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.124 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 14.922 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.181 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.904 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.072 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  4.399 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  4.986 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.767 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.552 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  2.718 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  5.164 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  8.356 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  2.022 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.104 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.251 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.897 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.855 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.406 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.023 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.771 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.460 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.841 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  6.743 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.029 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.379 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.696 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.022 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 29.114 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:08 min
[INFO] Finished at: 2017-01-02T11:41:11+08:00
[INFO] Final Memory: 229M/959M
[INFO] ------------------------------------------------------------------------

没有报错，则说明编译成功。

7、把编译好的Hadoop源码导入Idea

源码路径

找到编译好的Hadoop，选择即可，next。

选择Maven工程 

 

之后一路next即可，之后就可以开始阅读源码了。