SparkConf

Posted 2022-12-06 大冰的小屋

SparkConf的注释：

/**
 * Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
 *
 * Most of the time, you would create a SparkConf object with `new SparkConf()`, which will load
 * values from any `spark.*` Java system properties set in your application as well. In this case,
 * parameters you set directly on the `SparkConf` object take priority over system properties.
 *
 * For unit tests, you can also call `new SparkConf(false)` to skip loading external settings and
 * get the same configuration no matter what the system properties are.
 *
 * All setter methods in this class support chaining. For example, you can write
 * `new SparkConf().setMaster("local").setAppName("My app")`.
 *
 * Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified
 * by the user. Spark does not support modifying the configuration at runtime.
 *
 * @param loadDefaults whether to also load values from Java system properties
 */

从注释中可以看出，SparkConf是用于设置Spark的配置属性。默认情况下，会加载任何以spark.*开头的Java系统属性。而且其设置是支持链式操作，因为它的set方法都会返回this引用。
部分源码阅读

class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging 

  import SparkConf._

  /** Create a SparkConf that loads defaults from system properties and the classpath */
  def this() = this(true)    // 默认设为true，使用系统属性

  private val settings = new ConcurrentHashMap[String, String]()

  if (loadDefaults) 
    // 加载任何以spark.*开头的系统属性
    for ((key, value) <- Utils.getSystemProperties if key.startsWith("spark.")) 
      set(key, value)
    
  

  /** Set a configuration variable. */
  def set(key: String, value: String): SparkConf = 
    if (key == null) 
      throw new NullPointerException("null key")
    
    if (value == null) 
      throw new NullPointerException("null value for " + key)
    
    logDeprecationWarning(key)
    settings.put(key, value)
    this     // 为了链式使用，返回该对象的引用
  


  /**
   * The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to
   * run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
   */
  def setMaster(master: String): SparkConf = 
    set("spark.master", master)
  

...