Spark集群上运行jar程序，状态一直Accepted且不停止不报错

Posted 2020-08-24

 

如果运行Spark集群时状态一直为Accepted且不停止不报错，比如像下面这样的情况：

15/06/14 11:33:33 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:34 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:35 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:36 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:37 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:38 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:39 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:40 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
15/06/14 11:33:41 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)

 

一般是由于有多个用户同时向集群提交任务或一个用户向集群同时提交了多个任务导致Yarn资源的分配错误。解决这个问题，只需要更改Hadoop的配置文件：/etc/hadoop/conf/capacity-scheduler.xml，把选项：yarn.scheduler.capacity.maximum-am-resource-percent从0.1改成0.5。顾名思义，这个选项是增加Yarn可调度的资源量，当然也可以视具体情况增加更多。也可见，默认情况下，Yarn没有将很多资源分配给任务的能力。