Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十三）kafka+spark streaming打包好的程序提交时提示虚拟内存不足（Container is running

Posted 2020-12-03 yy3b2007com

异常问题：Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

spark-submit提交脚本：

[[email protected] work]$ more submit.sh 
#! /bin/bash
jars=""

for file in `ls /home/spark/work/jars/*.jar`
do
        jars=$file,$jars
        #echo $jars
done


echo "------------------------------------"
echo $jars
echo "------------------------------------"

/opt/spark-2.2.1-bin-hadoop2.7/bin/spark-submit --jars $jars --master yarn --verbose --driver-java-options "-XX:+TraceClassPaths" --num-executors 2 --executor-memory 1G --executor-cores 1 --driver-memory 1G --class com.dx.streaming.producer.TestProducer /home/spark/work/kafka-streaming-test.jar

在master节点上提交后，抛出异常：

18/07/09 05:00:16 INFO yarn.Client: Application report for application_1531069155353_0001 (state: ACCEPTED)
18/07/09 05:00:17 INFO yarn.Client: Application report for application_1531069155353_0001 (state: FAILED)
18/07/09 05:00:17 INFO yarn.Client: 
         client token: N/A
         diagnostics: Application application_1531069155353_0001 failed 2 times due to AM Container for appattempt_1531069155353_0001_000002 exited with  exitCode: -103
Failing this attempt.Diagnostics: [2018-07-09 05:00:16.132]Container [pid=5674,containerID=container_e22_1531069155353_0001_02_000001] is running beyond virtual memory limits. Current usage: 195.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_e22_1531069155353_0001_02_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 5674 5673 5674 5674 (bash) 0 0 115896320 350 /bin/bash -c /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg ‘192.168.0.120:40486‘ --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties 1> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stdout 2> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stderr 
        |- 5687 5674 5674 5674 (java) 208 35 2323283968 49770 /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.0.120:40486 --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties 
[2018-07-09 05:00:16.133]
[2018-07-09 05:00:16.152]Container killed on request. Exit code is 143[2018-07-09 05:00:16.152]
[2018-07-09 05:00:16.156]Container exited with a non-zero exit code 143. [2018-07-09 05:00:16.156]
For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1531069155353_0001 Then click on links to logs of each attempt.
. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1531083596749
         final status: FAILED
         tracking URL: http://master:8088/cluster/app/application_1531069155353_0001
         user: spark
18/07/09 05:00:17 INFO yarn.Client: Deleted staging directory hdfs://HA/user/spark/.sparkStaging/application_1531069155353_0001
18/07/09 05:00:17 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
▽       at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/07/09 05:00:17 INFO server.AbstractConnector: Stopped [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/07/09 05:00:17 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.0.120:4040
18/07/09 05:00:17 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
18/07/09 05:00:17 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
18/07/09 05:00:17 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Stopped
18/07/09 05:00:17 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/07/09 05:00:17 INFO memory.MemoryStore: MemoryStore cleared
18/07/09 05:00:17 INFO storage.BlockManager: BlockManager stopped
18/07/09 05:00:17 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
18/07/09 05:00:17 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running
18/07/09 05:00:17 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/07/09 05:00:17 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/07/09 05:00:17 INFO util.ShutdownHookManager: Shutdown hook called
18/07/09 05:00:17 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.2.1-bin-hadoop2.7/spark-255c65cf-19d8-481a-94bc-843e058eb669

错误原因：

Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。

解决方案：

上网一查，发现有两种方法能解决这个问题:

    1.将yarn.nodemanager.vmem-check-enabled的值改为false，即不检查VM的值；（修改yarn-site.xml）参考《https://discuss.elastic.co/t/unable-to-start-elasticsearch-yarn-container-is-running-beyond-virtual-memory-limits/26102/6》

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

     2.将yarn.scheduler.minimum-allocation-mb的值调高一些，默认是1024mb,或者修改yarn.nodemanager.vmem-pmem-ratio的值，默认为2.1，将该值改得更大。

于是，这里我采用了第一种方法，关闭vn检查，需要重新格式化namenode，参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA》，再次提交，果然好了。终于解决了这个困扰一天的问题啦！