spark运行任务报错：Container [...] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB
Posted gxiaobai
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了spark运行任务报错：Container [...] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB相关的知识，希望对你有一定的参考价值。
spark版本：1.6.0
scala版本：2.10
报错日志：
Application application_1562341921664_2123 failed 2 times due to AM Container for appattempt_1562341921664_2123_000002 exited with exitCode: -104
For more detailed output, check the application tracking page: http://winner-offline-namenode:8088/cluster/app/application_1562341921664_2123 Then click on links to logs of each attempt.
Diagnostics: Container [pid=8891,containerID=container_e12_1562341921664_2123_02_000001] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB physical memory used; 5.0 GB of 6.3 GB virtual memory used. Killing container.
Dump of the process-tree for container_e12_1562341921664_2123_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 8891 8889 8891 8891 (bash) 0 0 112881664 367 /bin/bash -c LD_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64: /usr/java/jdk1.8.0_162/bin/java -server -Xmx2048m -Djava.io.tmpdir=/disk1/hadoop/yarn/local/usercache/hadoop/appcache/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001/tmp ‘-XX:+PrintGCDetails‘ ‘-XX:+PrintGCTimeStamps‘ -Dhdp.version=2.6.4.0-91 -Dspark.yarn.app.container.log.dir=/disk1/hadoop/yarn/log/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class ‘com.winner.wifi.WinWifiAPIncre‘ --jar file:/hadoop/datadir/guozy/jar/winnercloud-assembly.jar --arg ‘2019-07-02‘ --arg ‘bg‘ --arg ‘St_P00094#St_P00087#Xsj_00001P00016#Cqhm_P00091#Langold_00004P00001#Xcjt_00004P00010#Djzx_P00001#Xsj_00001P00035#Xsj_00001P00013#Whc_P00081#Sygc_00001P00001#Szc_P00017#Lyyysc_P00001#Longfor_00001P00013#Lgjt_P00040#Wfjjt_P00028#Qzkyss_00001P00001#Hfjt_P00002#Xxhjgjgc_P00001#Xsj_00001P00024#Wdl_P00103#Hualian_00008P00020#Xl_P00067#Bfc_P00114#Snjt_00001P00001#Lfjt_P00002#Xuhui_P00050#Bailian_00002P00051#Xinyuan_P00077#Wdl_P00105#Hualian_P00004#Wfjjt_P00095#Fxhyjt_00001P00001#Xsj_00001P00023#Xsj_00001P00020#Xhbh_P00089#Szc_P00001#Yxgw_00001P00001#Wfjjt_P00035#Mybh_P00001#Yhjt_00002P00001#Wdl_P00106#Nydtjt_00001P00001#Xsj_00001P00002#Xuhui_P00038#Xsj_00001P00017#Xsj_00001P00012#Xsj_00001P00027#Lfjt_P00001#Xsj_00001P00034#Xhbh_P00108#Wjjt_P00022#Hjgc_P00001#Zxtfgc_P00001#Xhzb_P00001#Txjt_P00001#Mybh_P00003#Hdjt_00002P00018#Bailian_00002P00026#Hrzd_00002P00001#Bailian_00002P00052#Hxmkl_P00007#Ayjt_P00002#Wdl_P00107#Bailian_00002P00004#Ayjt_00004P00013#Hxmkl_P00009#Xsj_00001P00006#Wfjjt_P00027‘ --arg ‘1‘ --arg ‘150‘ --executor-memory 2048m --executor-cores 1 --properties-file /disk1/hadoop/yarn/local/usercache/hadoop/appcache/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001/__spark_conf__/__spark_conf__.properties 1> /disk1/hadoop/yarn/log/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001/stdout 2> /disk1/hadoop/yarn/log/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001/stderr
|- 8906 8891 8891 8891 (java) 51299 2993 5230977024 787116 /usr/java/jdk1.8.0_162/bin/java -server -Xmx2048m -Djava.io.tmpdir=/disk1/hadoop/yarn/local/usercache/hadoop/appcache/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001/tmp -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dhdp.version=2.6.4.0-91 -Dspark.yarn.app.container.log.dir=/disk1/hadoop/yarn/log/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class com.winner.wifi.WinWifiAPIncre --jar file:/hadoop/datadir/guozy/jar/winnercloud-assembly.jar --arg 2019-07-02 --arg bg --arg St_P00094#St_P00087#Xsj_00001P00016#Cqhm_P00091#Langold_00004P00001#Xcjt_00004P00010#Djzx_P00001#Xsj_00001P00035#Xsj_00001P00013#Whc_P00081#Sygc_00001P00001#Szc_P00017#Lyyysc_P00001#Longfor_00001P00013#Lgjt_P00040#Wfjjt_P00028#Qzkyss_00001P00001#Hfjt_P00002#Xxhjgjgc_P00001#Xsj_00001P00024#Wdl_P00103#Hualian_00008P00020#Xl_P00067#Bfc_P00114#Snjt_00001P00001#Lfjt_P00002#Xuhui_P00050#Bailian_00002P00051#Xinyuan_P00077#Wdl_P00105#Hualian_P00004#Wfjjt_P00095#Fxhyjt_00001P00001#Xsj_00001P00023#Xsj_00001P00020#Xhbh_P00089#Szc_P00001#Yxgw_00001P00001#Wfjjt_P00035#Mybh_P00001#Yhjt_00002P00001#Wdl_P00106#Nydtjt_00001P00001#Xsj_00001P00002#Xuhui_P00038#Xsj_00001P00017#Xsj_00001P00012#Xsj_00001P00027#Lfjt_P00001#Xsj_00001P00034#Xhbh_P00108#Wjjt_P00022#Hjgc_P00001#Zxtfgc_P00001#Xhzb_P00001#Txjt_P00001#Mybh_P00003#Hdjt_00002P00018#Bailian_00002P00026#Hrzd_00002P00001#Bailian_00002P00052#Hxmkl_P00007#Ayjt_P00002#Wdl_P00107#Bailian_00002P00004#Ayjt_00004P00013#Hxmkl_P00009#Xsj_00001P00006#Wfjjt_P00027 --arg 1 --arg 150 --executor-memory 2048m --executor-cores 1 --properties-file /disk1/hadoop/yarn/local/usercache/hadoop/appcache/application_1562341921664_2123/container_e12_1562341921664_2123_02_000001/__spark_conf__/__spark_conf__.properties
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143.
Failing this attempt. Failing the application.
报错截图：
主要资源配置：
--driver-memory 2g
--executore-memory 2g
临时解决方案：
增大driver端的内存
--driver-memory 3g
--executore-memory 2g
问题解决。
这里有一点还没有搞太明白，就是为什么只要driver端内存比executore内存大就可以呢，具体深层次的原因还没找到，如果网友知道，还望赐教，不胜感激
以上是关于spark运行任务报错：Container [...] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB的主要内容，如果未能解决你的问题，请参考以下文章
CDH6.3.2 Hive on spark报错is running beyond physical memory limits