markdown Hadoop2.7.5单节点学习

Posted 2021-05-06

# [官方文档](http://hadoop.apache.org/docs/r2.7.5/hadoop-project-dist/hadoop-common/SingleCluster.html)

# Windows下使用Hadoop
报错：JAVA_HOME is incorrectly set with hadoop
这是因为Java默认安装在C:\Program Files文件夹下，而路径中有空格会导致Hadoop报错，可以在JAVA_HOME的环境变量中将Program Files改为Progra~1。

# 伪分布式配置
## 修改配置文件
- core-site.xml

```xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
```

- hdfs-site.xml

```xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>
```

## 使用ssh无密码登录
```zsh
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
$ ssh-add -D
$ ssh-add

# 测试是否可以无密码登录
$ ssh localhost
```
## 启动NameNode和DataNode进程
```zsh
$ bin/hdfs namenode -format
$ sbin/start-dfs.sh
```
## 访问NameNode的WebUI
**http://localhost:500070**

## 创建MapReduce任务所需的HDFS目录
```zsh
$ bin/hdfs dfs -mkdir /user
$ bin/hdfs dfs -mkdir /user/leo
```

## 停止NameNode和DataNode进程
```zsh
$ sbin/stop-dfs.sh
```

# 使用YARN
## 修改配置文件
- etc/hadoop/mapred-site.xml

```xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
```

- etc/hadoop/yarn-site.xml

```xml
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>
```

## 启动ResourceManager和NodeManager后台进程
```zsh
$ sbin/start-yarn.sh
```

## 访问ResourceManager的WebUI
**http://localhost:8088**

## 停止ResourceManager和NodeManager后台进程
```zsh
$ sbin/stop-yarn.sh
```