Spark运行内存溢出---＞spark.yarn.executor.memoryOverhead

Posted 2021-06-26 Mr.zhou_Zxy

Spark运行内存溢出（spark.yarn.executor.memoryOverhead）：

hadoop内有个内存监控
可以设置当task,map,reduce or job 的累计使用的内存量限制的时候就会自动杀死进程
修改hadoop下的etc下的hadoop下的mapred-site.xml 配置文件，修改成功后重启hadoop和yarn

<property>
	<name>mapreduce.cluster.map.memory.mb</name>
	<value>-1</value>
</property>
<property>
	<name>mapreduce.cluster.reduce.memory.mb</name>
	<value>-1</value>
</property>

这里配置文件的目的是关闭监控机制，默认值为-1，只有值大于0的时候才会启动监控机制

• 报错详情

21/06/08 19:08:46 INFO yarn.Client:
         client token: N/A
         diagnostics: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, mo                               st recent failure: Lost task 0.3 in stage 0.0 (TID 3, hadoop, executor 78): ExecutorLostFailure (executor 78 exited caused by one of the running tasks) Rea                               son: Container killed by YARN for exceeding memory limits. 2.4 GB of 2.1 GB virtual memory used. Consider boosting spark.yarn.executor.memoryOverhead.
Driver stacktrace:
         ApplicationMaster host: 192.168.130.129
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1623149878341
         final status: FAILED
         tracking URL: http://hadoop:8088/proxy/application_1623122452198_0002/
         user: root
Exception in thread "main" org.apache.spark.SparkException: Application application_1623122452198_0002 finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1104)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1150)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)