Spark运行模式：cluster与client

Posted 2020-10-04 诸葛萧晁

When run SparkSubmit --class [mainClass], SparkSubmit will call a childMainClass which is

1. client mode, childMainClass = mainClass

2. standalone cluster mde, childMainClass = org.apache.spark.deploy.Client

3. yarn cluster mode, childMainClass = org.apache.spark.deploy.yarn.Client

The childMainClass is a wrapper of mainClass. The childMainClass will be called in SparkSubmit, and if cluster mode, the childMainClass will talk to the the cluster and launch a process on one woker to run the mainClass.

 

ps. use "spark-submit -v" to print debug infos.

 

Yarn client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn JavaWordCount.jar

childMainclass: org.apache.spark.examples.JavaWordCount

Yarn cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn-cluster JavaWordCount.jar

childMainclass: org.apache.spark.deploy.yarn.Client

 

Standalone client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 JavaWordCount.jar

childMainclass: org.apache.spark.examples.JavaWordCount

Stanalone cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 --deploy-mode cluster JavaWordCount.jar

childMainclass: org.apache.spark.deploy.rest.RestSubmissionClient (if rest, else org.apache.spark.deploy.Client)

 

Taking standalone spark as example, here is the client mode workflow. The mainclass run in the driver application which could be reside out of the cluster.