MacBook Pro 安装Hadoop 2.7.1 伪分布式

Posted 2021-03-12 Books&Coffee

 配置ssh

• 允许本机远程连接：

系统偏好设置->共享->点击选中 远程登录 【否则报错：ssh: connect to host localhost port 22: Connection refused】

• 允许免密ssh

ssh-keygen -t rsa【一直回车到底】

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod og-wx ~/.ssh/authorized_keys

• 再次验证 

[@wooluwakerdeMBP:~]$ ssh localhost

Last login: Sat Feb  1 16:29:20 2020 from ::1

 

安装java

[@wooluwakerdeMBP:~]$ java -version

java version "1.8.0_231"

Java(TM) SE Runtime Environment (build 1.8.0_231-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

 

安装Hadoop

• 下载hadoop

　Hadoop 版本：hadoop-2.7.1.tar.gz

• 解压Hadoop

　　　　 tar -zxvf hadoop-2.7.1.tar.gz

• 修改配置文件 hadoop-env.sh

# 添加 JAVA_HOME
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home

• 修改配置文件  core-site.xml

# 配置namenode的主机名和端口号，hadoop临时目录

<configuration>
    
    <!-- 指定HDFS老大（namenode）的通信地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://0.0.0.0:9000</value>
    </property>
    <!-- 指定hadoop运行时产生文件的存储路径 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/walker/software/hadoop/hadoop-2.7.1/temp</value>
    </property>

</configuration>

 

• 修改配置文件  hdfs-site.xml
  • 默认副本数3，修改为1
  • dfs.namenode.name.dir指明fsimage存放目录，多个目录用逗号隔开。
  • dfs.datanode.data.dir指定块文件存放目录，多个目录逗号隔开

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
　　 <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/Users/walker/software/hadoop/hadoop-2.7.1/tmp/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/Users/walker/software/hadoop/hadoop-2.7.1/tmp/hdfs/data</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:50070</value>
    </property>
</configuration>

 

• 修改配置文件  mapred-site.xml

<configuration>
    
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

</configuration>

 

• 修改配置文件  yarn-site.xml

 

<configuration>
　　<property>
    <!-- Site specific YARN configuration properties --> 
        <name>yarn.nodemanager.aux-services</name> 
        <value>mapreduce_shuffle</value> 
    </property>
</configuration>

 

配置Hadoop环境变量

export HADOOP_HOME=/Users/walker/software/hadoop/hadoop-2.7.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 

格式化NameNode

$ cd $HADOOP_HOME

$ bin/hdfs namenode -format

## 格式化成功后的结果

20/02/01 18:13:53 INFO common.Storage: Storage directory /Users/walker/software/hadoop/hadoop-2.7.1/tmp/hdfs/name has been successfully formatted.
20/02/01 18:13:53 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/02/01 18:13:53 INFO util.ExitUtil: Exiting with status 0
20/02/01 18:13:53 INFO namenode.NameNode: SHUTDOWN_MSG: 

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at wooluwakerdeMBP.lan/192.168.2.194

 

启动Hadoop

$ start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
20/02/01 18:17:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
Starting namenodes on [0.0.0.0]
The authenticity of host \'0.0.0.0 (0.0.0.0)\' can\'t be established.
ECDSA key fingerprint is SHA256:RYiYsOpM6+P18wPsX0uqkxG/SrG2QtHZocO3PJrJOD8.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added \'0.0.0.0\' (ECDSA) to the list of known hosts.
0.0.0.0: starting namenode, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/hadoop-walker-namenode-wooluwakerdeMBP.lan.out
localhost: starting datanode, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/hadoop-walker-datanode-wooluwakerdeMBP.lan.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/hadoop-walker-secondarynamenode-wooluwakerdeMBP.lan.out
20/02/01 18:17:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
starting yarn daemons
starting resourcemanager, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/yarn-walker-resourcemanager-wooluwakerdeMBP.lan.out
localhost: starting nodemanager, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/yarn-walker-nodemanager-wooluwakerdeMBP.lan.out
$ jps
32960 Jps
32640 DataNode
32737 SecondaryNameNode
32567 NameNode
32920 NodeManager
32842 ResourceManager

 

验证 HDFS 和 YARN 的WEBUI

• http://localhost:50070 打开能进入hdfs管理页面，表示启动成功。

 

 

• http://localhost:8088 打开能进入hadoop进程管理页面，表示启动成功