SparkConf
Posted 大冰的小屋
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了SparkConf相关的知识,希望对你有一定的参考价值。
SparkConf的注释:
/**
* Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
*
* Most of the time, you would create a SparkConf object with `new SparkConf()`, which will load
* values from any `spark.*` Java system properties set in your application as well. In this case,
* parameters you set directly on the `SparkConf` object take priority over system properties.
*
* For unit tests, you can also call `new SparkConf(false)` to skip loading external settings and
* get the same configuration no matter what the system properties are.
*
* All setter methods in this class support chaining. For example, you can write
* `new SparkConf().setMaster("local").setAppName("My app")`.
*
* Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified
* by the user. Spark does not support modifying the configuration at runtime.
*
* @param loadDefaults whether to also load values from Java system properties
*/
从注释中可以看出,SparkConf是用于设置Spark的配置属性。默认情况下,会加载任何以spark.*开头的Java系统属性。而且其设置是支持链式操作,因为它的set方法都会返回this引用。
部分源码阅读
class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging
import SparkConf._
/** Create a SparkConf that loads defaults from system properties and the classpath */
def this() = this(true) // 默认设为true,使用系统属性
private val settings = new ConcurrentHashMap[String, String]()
if (loadDefaults)
// 加载任何以spark.*开头的系统属性
for ((key, value) <- Utils.getSystemProperties if key.startsWith("spark."))
set(key, value)
/** Set a configuration variable. */
def set(key: String, value: String): SparkConf =
if (key == null)
throw new NullPointerException("null key")
if (value == null)
throw new NullPointerException("null value for " + key)
logDeprecationWarning(key)
settings.put(key, value)
this // 为了链式使用,返回该对象的引用
/**
* The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to
* run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
*/
def setMaster(master: String): SparkConf =
set("spark.master", master)
...
以上是关于SparkConf的主要内容,如果未能解决你的问题,请参考以下文章
Spark 运行时错误 - ClassDefNotFound: SparkConf