Spark job 部署模式
Posted share23
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Spark job 部署模式相关的知识，希望对你有一定的参考价值。
Spark job 的部署有两种模式，Client && Cluster
spark-submit .. --deploy-mode client | cluster
【上传 Jar 包】
[[email protected] ~]$ hdfs dfs -put myspark.jar data
【Client】
默认值，Driver 运行在 Client 端主机上。
spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 myspark.jar /user/centos/temptags.txt
【cluster】
Driver 运行在某个 Worker 节点上。客户端值负责提交 job。
spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 --deploy-mode cluster hdfs://mycluster/user/centos/data/myspark.jar /user/centos/temptags.txt
[[email protected] ~]$ xcall.sh jps ==================== s101 jps =================== 2981 Master 2568 NameNode 2889 DFSZKFailoverController 3915 Jps ==================== s102 jps =================== 2961 CoarseGrainedExecutorBackend 2450 Worker 2325 JournalNode 2246 DataNode 2187 QuorumPeerMain 3005 Jps ==================== s103 jps =================== 2457 Worker 2331 JournalNode 2188 QuorumPeerMain 3292 CoarseGrainedExecutorBackend 2253 DataNode 3310 Jps ==================== s104 jps =================== 2193 QuorumPeerMain 2981 DriverWrapper 3094 Jps 2455 Worker 2328 JournalNode 2252 DataNode 3038 CoarseGrainedExecutorBackend
[[email protected] /soft/spark-2.1.0-bin-hadoop2.7/bin]$ ./spark-submit --class com.share.scala.mr.TaggenCluster --master spark://s101:7077 --deploy-mode cluster hdfs://s101/user/centos/data/myspark.jar /user/centos/temptags.txt
以上是关于Spark job 部署模式的主要内容，如果未能解决你的问题，请参考以下文章