学习笔记Hadoop—— Hadoop基础操作—— YARN资源管理与调度策略

Posted 2021-10-22 别呀

六、YARN资源管理与调度策略

6.1、YARN资源管理

MapReduce任务/资源流程：

---

由5.1章节我们可知，当把yarn.scheduler.capacity.maximum-am-resource-percent参数调整为0.2 时，我们可以同时运行2个任务，此时Application Master最大资源为5120M（每个任务启动的AppMaster资源为2048M）

当参数调整为0.3时，可同时运行4个任务

6.2、YARN调度策略

Yarn的资源调度器是可配置的，Yarn定义了一套接口规范(接口ResourceScheduler)，用户可以按照自己的需求实现这个接口中的方法。

Yarn自身自带了三种调度器，分别是：FIFO Scheduler、Capacity Scheduler和Fair Scheduler。

Yarn的资源调度器默认使用Capacity Scheduler，但是默认只有一个root.default队列。

调整Yarn资源队列：

1. 添加dev,prod 队列，删除default队列；
2. 修改default队列配置为dev队列，并修改其capacity为40 ， maxium-capacity 为60 ；
3. 复制dev队列属性，并修改为prod队列，修改其capacity为60 ， maxium-capacity 为80 ；
4. 重启YARN集群；

cd /opt/hadoop-3.1.4/etc/hadoop
vi capacity-scheduler.xml

修改如下：

修改完后，配置prod，在dev末尾添加：

<!-- config prod queue -->
 <property>
    <name>yarn.scheduler.capacity.root.prod.capacity</name>
    <value>60</value>
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.prod.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.prod.maximum-capacity</name>
    <value>80</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.prod.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.prod.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.prod.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.prod.acl_application_max_priority</name>
    <value>*</value>
    <description>
      The ACL of who can submit applications with configured priority.
      For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
    </description>
  </property>

   <property>
     <name>yarn.scheduler.capacity.root.prod.maximum-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Maximum lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        This will be a hard time limit for all applications in this
        queue. If positive value is configured then any application submitted
        to this queue will be killed after exceeds the configured lifetime.
        User can also specify lifetime per application basis in
        application submission context. But user lifetime will be
        overridden if it exceeds queue maximum lifetime. It is point-in-time
        configuration.
        Note : Configuring too low value will result in killing application
        sooner. This feature is applicable only for leaf queue.
     </description>
   </property>

   <property>
     <name>yarn.scheduler.capacity.root.prod.default-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Default lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        If the user has not submitted application with lifetime value then this
        value will be taken. It is point-in-time configuration.
        Note : Default lifetime can't exceed maximum lifetime. This feature is
        applicable only for leaf queue.
     </description>
   </property>

配置prod完后，保存、退出，拷贝到node1、node2、node3下：

scp capacity-scheduler.xml node1:/opt/hadoop-3.1.4/etc/hadoop/
scp capacity-scheduler.xml node2:/opt/hadoop-3.1.4/etc/hadoop/
scp capacity-scheduler.xml node3:/opt/hadoop-3.1.4/etc/hadoop/

最后，重启下集群(进入sbin目录)

./stop-yarn.sh
./start-yarn.sh

启动成功

最后，我们查看下网页的情况，浏览器网址输入：

master:8088

OK，我们配置成功了！

测试：
接下来，我们来测试一下：

① 我们提交到dev队列

在网页上我们可以看到任务已经提交了

在网页上，我们可以看占用资源查看：

② 提交到prod队列

通过提交到dev和prod ，我们可以得出：不管是dev还是prod queue都不会占用Max_Capacity