spark-submit提交任务到yarn错误
Posted mergy
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了spark-submit提交任务到yarn错误相关的知识，希望对你有一定的参考价值。
1.Error initializing SparkContext.
20/06/29 05:52:43 INFO yarn.Client: Deleted staging directory hdfs://master:9000/user/hadoop/.sparkStaging/application_1593402611188_0003
20/06/29 05:52:43 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
遇到这问题先不要纠结这个点，先看其他错误。我的另一个错误是
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1593402611188_0003Then, click on links to logs of each attempt. Diagnostics: Wrong FS: file://home/hadoop/data/hadoop/tmp/nm-local-dir, expected: file:/// Failing this attempt. Failing the application.
也就是找不到yarn的nm-local-dir所以应该在yarn-site.xmlite.xml里配置yarn.nodemanager.local-dirs
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/hadoop/data/hadoop/tmp/nm-local-dir</value>
</property>
以上是关于spark-submit提交任务到yarn错误的主要内容，如果未能解决你的问题，请参考以下文章