64位CentOS上编译 Hadoop 2.2.0

Posted 2020-06-11

 

1. 下载Hadoop 2.2.0 源码包，并解压

$ wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
$ tar zxf hadoop-2.2.0-src.tar.gz

2. 安装下面的软件

 $ sudo yum install cmake lzo-devel  zlib-devel  gcc autoconf automake libtool   ncurses-devel openssl-devel

3. 安装Maven

$ wget http://mirror.esocc.com/apache/maven/maven-3/3.1.4/binaries/apache-maven-3.1.4-bin.tar.gz
$ sudo tar zxf apache-maven-3.1.4-bin.tar.gz -C /opt
$ sudo vim /etc/profile
export MAVEN_HOME=/opt/apache-maven-3.1.4
export PATH=$PATH:$MAVEN_HOME/bin

使之生效  source /etc/profile

4. 安装Ant

$ wget http://apache.dataguru.cn//ant/binaries/apache-ant-1.9.4-bin.tar.gz
$ sudo tar zxf apache-ant-1.9.4-bin.tar.gz -C /opt
$ sudo vim /etc/profile
export ANT_HOME=/opt/apache-ant-1.9.4
export PATH=$PATH:$ANT_HOME/bin

5. 安装Findbugs

$ wget http://prdownloads.sourceforge.net/findbugs/findbugs-2.0.3.tar.gz?download
$ sudo tar zxf findbugs-2.0.3.tar.gz -C /opt
$ sudo vim /etc/profile
export FINDBUGS_HOME=/opt/findbugs-2.0.3
export PATH=$PATH:$FINDBUGS_HOME/bin

6. 安装protobuf

$ wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
$ tar zxf protobuf-2.5.0.tar.gz
$ cd protobuf-2.5.0
$ ./configure
$ make
$ sudo make install

7. 给Hadoop源码打一个patch

最新的Hadoop 2.2.0 的Source Code 压缩包解压出来的code有个bug 需要patch后才能编译。否则编译hadoop-auth 会提示下面错误：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:
[ERROR] /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13] cannot access org.mortbay.component.AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found

Patch: https://issues.apache.org/jira/browse/HADOOP-10110

wget https://issues.apache.org/jira/secure/attachment/12614482/HADOOP-10110.patch; patch -p0 < HADOOP-10110.patch

8. 编译 Hadoop

cd hadoop-2.2.0-src
mvn package -DskipTests -Pdist,native -Dtar

9. 替换掉32位的native库

用 hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib/native 替换掉 hadoop-2.2.0/lib/native。

rm -rf ~/local/opt/hadoop-2.2.0/lib/native
cp ./hadoop-dist/target/hadoop-2.2.0/lib/native ~/local/opt/hadoop-2.2.0/lib/

 

hadoop namenode -format

start-all.sh

jps