Spark 3.1.3 HADOOP_CONF_DIR or YARN_CONF_DIR must be set

Posted 2022-08-06 @SmartSi

1. 现象

使用如下命令以 client 模式提交作业到 Yarn 上时：

spark-submit \\
  --class com.spark.example.core.base.WordCount \\
  --master yarn \\
  --deploy-mode client \\
  --executor-memory 2g \\
  --num-executors 4 \\
  --executor-cores 1 \\
spark-example-3.1-1.0.jar \\
/data/word-count/word-count-input /data/word-count/word-count-output

抛出如下异常：

Exception in thread "main" org.apache.spark.SparkException: When running with master \'yarn\' either