Hadoop2.6.0实践：000 环境搭建

Posted 2020-08-29

##################### Centos6.4VM_01_os.rar ################################################
准备工作
/opt
/opt/modules 软件安装目录
/opt/softwares 软件包（tar、bin、zip）
/opt/tools（eclipse等）
/opt/data（测试数据）
/home/hadoop（工具和数据）

##################### Centos6.4VM_01_os.rar ################################################
1、创建hadoop用户，分配权限

###########################################
# 创建新用户hadoop
[[email protected] /]# useradd -m hadoop -G root -s /bin/bash
[[email protected] /]# passwd hadoop
# 增加管理员权限
[[email protected] /]# visudo
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
hadoop ALL=(root) NOPASSWD:ALL

验证hadoop用户无密码sudo权限
[[email protected] ~]$sudo service iptables status

#####################################################################
2、配置网络
配置ip、hostname、hosts、本地dns、防火墙

###########################################
设置ip
192.168.126.130
255.255.255.0
192.168.126.2
192.168.0.1

[[email protected] network-scripts]# more /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=b36ccffb-5370-4ded-a68d-1ebbb70aa7f9
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR=192.168.126.130
PREFIX=24
GATEWAY=192.168.126.2
DNS1=172.20.10.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
HWADDR=00:0C:29:31:83:95
LAST_CONNECT=1483940638

[[email protected] ~]# ifconfig

###########################################
设置hostname

[[email protected] ~]# hostname
[[email protected] ~]# hostname hadoop-master
[[email protected] ~]# more /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop-master
GATEWAY=192.168.126.2

###########################################
设置本地dns（/etc/hosts）
[[email protected] etc]# more /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.126.130 hadoop-master hadoop-master.dragon.org

###########################################
关闭防火墙
[[email protected] ~]# service iptables stop
iptables：清除防火墙规则：[确定]
iptables：将链设置为政策 ACCEPT：filter [确定]
iptables：正在卸载模块：[确定]
[[email protected] ~]# vim /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled

###########################################
# 检查是否安装ssh-server和ssh-clients
[[email protected] ~]# rpm -qa | grep ssh
openssh-server-5.3p1-84.1.el6.x86_64
openssh-clients-5.3p1-84.1.el6.x86_64
openssh-askpass-5.3p1-84.1.el6.x86_64
openssh-5.3p1-84.1.el6.x86_64
libssh2-1.4.2-1.el6.x86_64

# 如果没有安装，执行如下命令安装
[[email protected] ~]$sudo yum install openssh-server
[[email protected] ~]$sudo yum install openssh-clients

# 配置SSH无密码登陆
[[email protected] ~]$ ssh localhost
The authenticity of host ‘localhost (::1)‘ can‘t be established.
RSA key fingerprint is 09:0a:d3:7f:b4:43:8f:fa:a8:07:e2:37:e7:b7:00:ab.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost‘ (RSA) to the list of known hosts.
[email protected]‘s password:
Last login: Sun Jan 8 19:36:46 2017 from 192.168.126.1
[[email protected] ~]$ exit

[[email protected] ~]$ cd ~/.ssh/

# 利用ssh-keygen生成密钥
[[email protected] .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
9d:72:cc:49:7a:7c:51:44:80:1c:4f:ec:3e:d8:46:82 [email protected]
The key‘s randomart image is:
+--[ RSA 2048]----+
| ..+o++ |
| oo.. |
| ...o |
| EB.oo. |
| S X*. |
| +..= |
| . . |
| |
| |
+-----------------+

# 将密钥加入到授权中
[[email protected] .ssh]$ cat id_rsa.pub >> authorized_keys
[[email protected] .ssh]$ chmod 600 ./authorized_keys
[[email protected] .ssh]$

##################### centos6.4VM_02_No_jdk&hadoop.rar ######################

3、安装jdk

#############################################
卸载OpenJdk
[[email protected] ~]$ java -version
[[email protected] ~]$ javac -version
[[email protected] ~]$ rpm -qa | grep java

tzdata-java-2012j-1.el6.noarch
java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64

[[email protected] ~]$ sudo yum remove java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
[[email protected] ~]$ sudo yum remove java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
[[email protected] ~]$ sudo yum remove tzdata-java-2012j-1.el6.noarch

检查一下
[[email protected] ~]$ rpm -qa | grep java
[[email protected] ~]$ java -version
[[email protected] ~]$ javac -version

#############################################
安装OracleJdk
$ cd/opt/softwares
$ sudo tar -zxvf jdk-7u80-linux-x64.tar.gz -C /opt/modules

$ ls -al /opt/modules
$ sudo vim /etc/profile

## add by lsq for jdk-env begin
export JAVA_HOME=/opt/modules/jdk/jdk1.7.0_80
export PATH=$JAVA_HOME/bin:$PATH
export CLASS_PATH=$CLASS_PATH:.:$JAVA_HOME/lib
## add by lsq for jdk-env end

[[email protected] jdk1.7.0_80]$ source /etc/profile
[[email protected] jdk1.7.0_80]$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

[[email protected] jdk1.7.0_80]$ javac -version
javac 1.7.0_80

##################### centos6.4VM_02_No_jdk&hadoop.rar ######################

4、安装hadoop

#*********************************************
下载hadoop
校验hadoop
cat hadoop-2.6.0.tar.gz.mds | grep ‘MD5‘
md5sum hadoop-2.6.0.tar.gz | tr "a-z" "A-Z"
比较两个结果中的MD5值是否一样
#*********************************************

[[email protected] ~]$ sudo cp *.gz /opt/softwares
[[email protected] ~]$ cd /opt/softwares
[[email protected] softwares]$ ls -al

$ cd/opt/softwares
$ sudo tar -zxvf hadoop-2.6.0.tar.gz -C /opt/modules
$ sudo chown -R hadoop:hadoop /opt/modules/hadoop-2.6.0

$ cd /opt/modules/hadoop-2.6.0/bin
[[email protected] bin]$ ./hadoop version
Hadoop 2.6.0
Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997
Compiled by sjlee on 2016-10-02T23:43Z
Compiled with protoc 2.5.0
From source with checksum f05c9fa095a395faa9db9f7ba5d754
This command was run using /opt/modules/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar

$ ls -al /opt/modules
$ sudo vim /etc/profile

##########################################################
# liangshengqi begin

export JAVA_HOME=/opt/modules/jdk1.7.0_80
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

export HADOOP_HOME=/opt/modules/hadoop-2.6.0
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/natvie
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

# liangshengqi end
##########################################################
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib:$HADOOP_HOME/lib/native"

[[email protected] bin]$ source /etc/profile
[[email protected] bin]$ tail /etc/profile

[[email protected] modules]$ echo $YARN_HOME
/opt/modules/hadoop-2.6.0

设置hadoop-env.sh中的java环境变量
cd /opt/modules/hadoop-2.6.0
vim ./etc/hadoop/hadoop-env.sh

# The java implementation to use.
export JAVA_HOME=/opt/modules/jdk1.7.0_80

设置hadoop-env.sh中的java环境变量
vim ./etc/hadoop/yarn-env.sh
export JAVA_HOME=/opt/modules/jdk1.7.0_80

设置mapred-env.sh中的java环境变量
vim ./etc/hadoop/mapred-env.sh
export JAVA_HOME=/opt/modules/jdk1.7.0_80

#########################################################################

5、验证单机模式
Hadoop 默认模式为非分布式模式，无需进行其他配置即可运行。非分布式即单 Java 进程，方便进行调试。

cd /opt/modules/hadoop-2.6.0
cp ./etc/hadoop/*.xml /opt/data/input
mkdir /opt/data/input
chmod -R 777 /opt/data

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep /opt/data/input /opt/data/output ‘dfs[a-z.]+‘

[[email protected] data]# ls -al /opt/data/output
总用量 20
drwxrwxr-x. 2 hadoop hadoop 4096 1月 10 19:28 .
drwxrwxrwx. 4 root root 4096 1月 10 19:28 ..
-rw-r--r--. 1 hadoop hadoop 11 1月 10 19:28 part-r-00000
-rw-r--r--. 1 hadoop hadoop 12 1月 10 19:28 .part-r-00000.crc
-rw-r--r--. 1 hadoop hadoop 0 1月 10 19:28 _SUCCESS
-rw-r--r--. 1 hadoop hadoop 8 1月 10 19:28 ._SUCCESS.crc

cat /opt/data/output/*
1 dfsadmin

#########################################################################

6、配置伪分布式

#修改配置文件-环境变量

#修改配置文件-hadoop的配置文件
#hadoop的配置文件主要有core-site.xml 、 hdfs-site.xml 、 yarn-site.xml 三个文件。
cd /usr/opt/hadoop/etc/hadoop

##############################################################
#core-site.xml

<!--一定要配置 系统默认的缓存文件夹在每次重启后会被删除-->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/data/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-master.dragon.org:9000</value>
</property>
</configuration>

##############################################################
#hdfs-site.xml

<！--这个属性节点是为了防止后面eclipse存在拒绝读写设置的 -->
<configuration>

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/data/hadoop/dfs/name</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/data/hadoop/dfs/data</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

</configuration>

##############################################################
#mapred-site.xml.template(如果想启动yarn，修改此文件名为mapred-site.xml)
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

##############################################################
#yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

##############################################################
接下来创建需要的文件夹

cd /opt/data/hadoop
mkdir tmp dfs dfs/name dfs/data
chmod 777 -R data

到目前为止所有的配置都已经完成。

##############################################################
# 启动文件系统

hdfs namenode -format
./sbin/start-dfs.sh
./sbin/start-yarn.sh
# 开启历史服务器，才能在web中查看任务运行情况
./sbin/mr-jobhistory-daemon.sh start historyserver
# 不启动yarn，是 mapred.LocalJobRunner 在跑任务
# 启动yarn后，是 mapred.YARNRunner 在跑任务
# 启动yarn后，有个好处是可以通过web界面查看任务运行情况，http://localhost:8088/cluster

hadoop dfs -ls /
hadoop dfs -mkdir -p /examples/grep/input
hadoop dfs -put /opt/data/input/*.xml /examples/grep/input
hadoop dfs -ls /examples/grep/input

# 运行自带例子
hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep /examples/grep/input /examples/grep/output ‘dfs[a-z.]+‘

ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

############################################################问题处理####################################################
#备份native目录
mv native native_bak
#重建native目录
mkdir native
#解压hadoop-native-64-2.6.0.tar到这个目录
#修改so权限
chmod 755 libhadoop.so.1.0.0
chmod 755 libhdfs.so.0.0.0
#重建失效的链接文件
rm -rf libhadoop.so
rm -rf libhdfs.so
[[email protected] native]$ ln -s libhadoop.so.1.0.0 libhadoop.so
[[email protected] native]$ ln -s libhdfs.so.0.0.0 libhdfs.so
#检查文件列表
[[email protected] native]$ ls -al
总用量 4364
drwxrwxr-x 2 hadoop hadoop 4096 1月 14 11:03 .
drwxr-xr-x 4 hadoop hadoop 4096 1月 14 10:52 ..
-rw-r--r-- 1 root root 1119486 12月 2 2014 libhadoop.a
-rw-r--r-- 1 root root 1486964 12月 2 2014 libhadooppipes.a
lrwxrwxrwx 1 hadoop hadoop 18 1月 14 11:02 libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x 1 root root 671189 12月 2 2014 libhadoop.so.1.0.0
-rw-r--r-- 1 root root 581944 12月 2 2014 libhadooputils.a
-rw-r--r-- 1 root root 359458 12月 2 2014 libhdfs.a
lrwxrwxrwx 1 hadoop hadoop 16 1月 14 11:03 libhdfs.so -> libhdfs.so.0.0.0
-rwxr-xr-x 1 root root 228435 12月 2 2014 libhdfs.so.0.0.0

##############################################################

hadoop “util.NativeCodeLoader: Unable to load native-hadoop library for your platform”

hadoop安装完以后，经常会提示一下警告：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable
搜了好多文章，都说是跟系统位数有关系，我使用的是Centos 6.5 64位操作系统。

前两天在做Docker镜像的时候发现了一个步骤可以解决这个问题，亲试了一下，果然不再提示了。

首先下载hadoop-native-64-2.4.0.tar：
http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.4.0.tar
如果你是hadoop2.6的可以下载下面这个：
http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.6.0.tar
下载完以后，解压到hadoop的native目录下，覆盖原有文件即可。操作如下：
tar -x hadoop-native-64-2.4.0.tar -C hadoop/lib/native/

###########################################################################################################