搭建Hadoop2.6.4伪分布式

Posted 2020-07-09 xdlysk

准备工作

操作系统

CentOS 7

软件环境

1. JDK 1.7.0_79 下载地址
2. SSH，正常来说是系统自带的，若没有请自行搜索安装方法

关闭防火墙

systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动

设置HostName

[[email protected] ~]# hostname localhost

安装环境

安装JDK

[[email protected] ~]# tar -xzvf jdk-7u79-linux-x64.tar.gz

配置java环境变量

[[email protected] ~]# vi /etc/profile
#添加如下配置
JAVA_HOME=/root/jdk1.7.0_79
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME
export PATH
export CLASSPATH

验证java

[[email protected] ~]# java -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

待输出以上内容时说明java已安装配置成功。

安装Hadoop

下载Hadoop 2.6.4

安装Hadoop 2.6.4

[[email protected] ~]# tar -xzvf hadoop-2.6.4.tar.gz

配置Hadoop环境变量

[[email protected] ~]# vim /etc/profile
#添加以下配置
export HADOOP_HOME=/root/hadoop-2.6.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin


[[email protected] ~]# vim /root/hadoop-2.6.4/etc/hadoop/hadoop-env.sh
#修改以下配置
# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
export JAVA_HOME=/root/jdk1.7.0_79

验证Hadoop

[[email protected] ~]# hadoop version
Hadoop 2.6.4
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 5082c73637530b0b7e115f9625ed7fac69f937e6
Compiled by jenkins on 2016-02-12T09:45Z
Compiled with protoc 2.5.0
From source with checksum 8dee2286ecdbbbc930a6c87b65cbc010
This command was run using /root/hadoop-2.6.4/share/hadoop/common/hadoop-common-2.6.4.jar

修改Hadoop配置文件

配置文件均存放在/root/hadoop-2.6.4/etc/hadoop

<!-- core-site.xml-->
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>


<!-- hdfs-site.xml -->
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>


<!-- mapred-site.xml -->
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>


<!-- yarn-site.xml -->
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

SSH免密码登陆

[[email protected] ~]# ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa
[[email protected] ~]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

输入以下命令，如果不要求输入密码则表示配置成功：

[[email protected] ~]# ssh localhost
Last login: Fri May  6 05:17:32 2016 from 192.168.154.1

执行Hadoop

格式化hdfs

[[email protected] ~]# hdfs namenode -format

启动NameNode，DataNode和YARN

[[email protected] ~]# start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /root/hadoop-2.6.4/logs/hadoop-root-namenode-localhost.out
localhost: starting datanode, logging to /root/hadoop-2.6.4/logs/hadoop-root-datanode-localhost.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /root/hadoop-2.6.4/logs/hadoop-root-secondarynamenode-localhost.out

[[email protected] ~]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /root/hadoop-2.6.4/logs/yarn-root-resourcemanager-localhost.out
localhost: starting nodemanager, logging to /root/hadoop-2.6.4/logs/yarn-root-nodemanager-localhost.out

向hdfs上传测试文件

首先在/root/test中建立test1.txt和test2.txt，分别输入“hello world”和“hello hadoop”并保存。

使用如下命令将文件上传至hdfs的input目录中：

[[email protected] ~]# hadoop fs -put /root/test/ input
[[email protected] ~]# hadoop fs -ls input
Found 2 items
-rw-r--r--   1 root supergroup         12 2016-05-06 06:35 input/test1.txt
-rw-r--r--   1 root supergroup         13 2016-05-06 06:35 input/test2.txt

执行wordcount demo

输入以下命令并等待执行结果：

[[email protected] ~]# hadoop jar /root/hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount input output
16/05/06 06:44:15 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/05/06 06:44:16 INFO input.FileInputFormat: Total input paths to process : 2
16/05/06 06:44:17 INFO mapreduce.JobSubmitter: number of splits:2
16/05/06 06:44:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462530786445_0001
16/05/06 06:44:18 INFO impl.YarnClientImpl: Submitted application application_1462530786445_0001
16/05/06 06:44:18 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1462530786445_0001/
16/05/06 06:44:18 INFO mapreduce.Job: Running job: job_1462530786445_0001
16/05/06 06:44:33 INFO mapreduce.Job: Job job_1462530786445_0001 running in uber mode : false
16/05/06 06:44:33 INFO mapreduce.Job:  map 0% reduce 0%
16/05/06 06:44:52 INFO mapreduce.Job:  map 50% reduce 0%
16/05/06 06:44:53 INFO mapreduce.Job:  map 100% reduce 0%
16/05/06 06:45:03 INFO mapreduce.Job:  map 100% reduce 100%
16/05/06 06:45:03 INFO mapreduce.Job: Job job_1462530786445_0001 completed successfully
16/05/06 06:45:04 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=55
                FILE: Number of bytes written=320242
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=249
                HDFS: Number of bytes written=25
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=34487
                Total time spent by all reduces in occupied slots (ms)=7744
                Total time spent by all map tasks (ms)=34487
                Total time spent by all reduce tasks (ms)=7744
                Total vcore-milliseconds taken by all map tasks=34487
                Total vcore-milliseconds taken by all reduce tasks=7744
                Total megabyte-milliseconds taken by all map tasks=35314688
                Total megabyte-milliseconds taken by all reduce tasks=7929856
        Map-Reduce Framework
                Map input records=2
                Map output records=4
                Map output bytes=41
                Map output materialized bytes=61
                Input split bytes=224
                Combine input records=4
                Combine output records=4
                Reduce input groups=3
                Reduce shuffle bytes=61
                Reduce input records=4
                Reduce output records=3
                Spilled Records=8
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=364
                CPU time spent (ms)=3990
                Physical memory (bytes) snapshot=515538944
                Virtual memory (bytes) snapshot=2588155904
                Total committed heap usage (bytes)=296755200
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=25
        File Output Format Counters 
                Bytes Written=25

查看执行结果

[[email protected] ~]# hadoop fs -ls output
Found 2 items
-rw-r--r--   1 root supergroup          0 2016-05-06 06:45 output/_SUCCESS
-rw-r--r--   1 root supergroup         25 2016-05-06 06:45 output/part-r-00000
[[email protected] ~]# hadoop fs -cat output/part-r-00000
hadoop  1
hello   2
world   1

至此，Pseudo-Distributed就已经完成了。

完全分布式可参考这里