《Hadoop权威指南 第4版》 - 第四章 关于YARN - hadoop的集群资源管理系统

Posted 2020-11-24 好好学习，天天向上

简介

• YARN 提供请求和使用hadoop集群资源的API
• 向上隐藏细节
• 提供更高层的API

4.1 YARN应用运行机制

- 资源请求
- 应用生命周期
- 构建yarn应用

4.2 YARN与MapReduce 1相比 (MapReduce特指hadoop1 的版本, 2/3依次对应)

- 

4.3 YARN中的调度

• 调度选项
  • FIFO调度器
  • 容量调度器 (多个请求队列调用一个hadoop集群, 每个队列请求量上限不可逾越)
  • 公平调度器 (动态平衡资源调度, 大作业多分配)

启动YARN并运行MapReduce程序

先启动namenode和datanode

[root@localhost hadoop-2.7.2]# sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-namenode-localhost.localdomain.out
[root@localhost hadoop-2.7.2]# sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-localhost.localdomain.out

（1）配置集群在YARN上运行MR,启动 resourcemanager 和 nodemanager

[root@localhost hadoop-2.7.2]# sbin/yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-root-resourcemanager-localhost.localdomain.out
[root@localhost hadoop-2.7.2]# sbin/yarn-daemon.sh start nodemanager    
starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-root-nodemanager-localhost.localdomain.out
[root@localhost hadoop-2.7.2]# jps
33232 ResourceManager
31922 NameNode
32018 DataNode
33475 NodeManager
33603 Jps

web端查看HDFS文件系统
- 虚拟机里面可以正常访问

- 外面的windows访问异常 http://192.168.147.132:8088/cluster         TODO

（2）启动、测试集群增、删、查

• 先清除之前的log

[root@localhost hadoop-2.7.2]# hdfs dfs -rm -R /output
20/01/02 21:06:23 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /output
[root@localhost hadoop-2.7.2]# hdfs dfs -ls .
Found 2 items
drwxr-xr-x   - root supergroup          0 2020-01-02 15:55 books
drwxr-xr-x   - root supergroup          0 2020-01-02 19:39 input
[root@localhost hadoop-2.7.2]# 

（3）在YARN上执行WordCount案例

• 使用hadoop的命令

[root@localhost hadoop-2.7.2]# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /input /output
20/01/02 21:10:41 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
20/01/02 21:10:45 INFO input.FileInputFormat: Total input paths to process : 3
20/01/02 21:10:45 INFO mapreduce.JobSubmitter: number of splits:3
20/01/02 21:10:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1577969816125_0001
20/01/02 21:10:47 INFO impl.YarnClientImpl: Submitted application application_1577969816125_0001

• 截图
  

• 结果: hello world共有三个文件, 每个中各一句

[root@localhost hadoop-2.7.2]# bin/hdfs dfs -cat /output/*             
Hello   3
World   3
[root@localhost hadoop-2.7.2]#