Spark- 共享变量

Posted 2021-01-17 rzcong

Shared Variables

Normally, when a function passed to a Spark operation (such as map or reduce) is executed on a remote cluster node, it works on separate copies of all the variables used in the function. These variables are copied to each machine, and no updates to the variables on the remote machine are propagated back to the driver program. Supporting general, read-write shared variables across tasks would be inefficient. However, Spark does provide two limited types of shared variables for two common usage patterns: broadcast variables and accumulators.

Broadcast Variables

Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used, for example, to give every node a copy of a large input dataset in an efficient manner. Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost.

Spark actions are executed through a set of stages, separated by distributed “shuffle” operations. Spark automatically broadcasts the common data needed by tasks within each stage. The data broadcasted this way is cached in serialized form and deserialized before running each task. This means that explicitly creating broadcast variables is only useful when tasks across multiple stages need the same data or when caching the data in deserialized form is important.

Broadcast variables are created from a variable v by calling SparkContext.broadcast(v). The broadcast variable is a wrapper around v, and its value can be accessed by calling the value method. 

 

共享变量

通常，当在远程集群节点上执行传递给Spark操作（例如mapor reduce）的函数时，它将在函数中使用的所有变量的单独副本上工作。这些变量将复制到每台计算机，并且远程计算机上的变量更新不会传播回驱动程序。支持跨任务的通用，读写共享变量效率低下。但是，Spark确实为两种常见的使用模式提供了两种有限类型的共享变量：广播变量和累加器。

广播变量

广播变量允许程序员在每台机器上保留一个只读变量，而不是随副本一起发送它的副本。例如，它们可用于以有效的方式为每个节点提供大输入数据集的副本。Spark还尝试使用有效的广播算法来分发广播变量，以降低通信成本。

Spark动作通过一组阶段执行，由分布式“shuffle”操作分隔。Spark自动广播每个阶段中任务所需的公共数据。以这种方式广播的数据以序列化形式缓存并在运行每个任务之前反序列化。这意味着显式创建广播变量仅在跨多个阶段的任务需要相同数据或以反序列化形式缓存数据很重要时才有用。

广播变量是v通过调用从变量创建的SparkContext.broadcast(v)。广播变量是一个包装器v，可以通过调用该value 方法来访问它的值。

java 版本：

package cn.rzlee.spark;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.VoidFunction;
import org.apache.spark.broadcast.Broadcast;

import java.util.Arrays;
import java.util.List;


/**
 * @Author ^_^
 * @Create 2018/11/3
 */
public class BroadcastVariable {

    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("Persist").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);


        // 在Java中，创建共享变量，就是调用SparkContext的broadcast()方法
        // 获取的返回结果是Broadcast<T>类型
        final int factor =3;
        final Broadcast<Integer> factorBroadcast = sc.broadcast(factor);

        List<Integer> numberList =  Arrays.asList(1,2,3,4,5,6,7,8,9);
        JavaRDD<Integer> numbers = sc.parallelize(numberList);

        // 让集合中的每个数字都乘以外部定义的那个 factor
        JavaRDD<Integer> multipleNumbers = numbers.map(new Function<Integer, Integer>() {
            @Override
            public Integer call(Integer v1) throws Exception {

                // 使用共享变量时，调用其value()方法，即可获取其内部封装的值
                Integer factor = factorBroadcast.value();
                return v1 * factor;
            }
        });

        multipleNumbers.foreach(new VoidFunction<Integer>() {
            @Override
            public void call(Integer integer) throws Exception {
                System.out.println(integer);
            }
        });



        sc.close();
    }



}

scala 版本：

package cn.rzlee.spark.scala

import org.apache.spark.broadcast.Broadcast
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object BroadcastVariable {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local").setAppName(this.getClass.getSimpleName)
    val sc = new SparkContext(conf)

    val factor = 3
    val factorBroadcast: Broadcast[Int] = sc.broadcast(factor)
    
    val numbersArray = Array(1,2,3,4,5,6,7,8,9)
    val numbers: RDD[Int] = sc.parallelize(numbersArray, 1)
    val mutipleNumbers: RDD[Int] = numbers.map(num =>num * factorBroadcast.value)

    mutipleNumbers.foreach(num=>println(num))
    sc.stop()
  }
}

 

 

 

累加器

累加器是仅通过关联操作“添加”的变量，因此可以并行有效地支持。它们可用于实现计数器（如MapReduce）或总和。Spark本身支持数值类型的累加器，程序员可以添加对新类型的支持。如果使用名称创建累加器，它们将显示在Spark的UI中。这对于理解运行阶段的进度非常有用（注意：Python中尚不支持此功能）。

v通过调用从初始值创建累加器SparkContext.accumulator(v)。然后，在集群上运行的任务可以使用add方法或+=运算符（在Scala和Python中）添加到它。但是，他们无法读懂它的价值。只有驱动程序可以使用其value方法读取累加器的值。

下面的代码显示了一个累加器用于添加数组的元素：

 

java 版本：

package cn.rzlee.spark.core;

import org.apache.spark.Accumulator;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.VoidFunction;

import java.util.Arrays;
import java.util.List;

/**
 * @Author ^_^
 * @Create 2018/11/3
 */
public class AccumulatorVariable {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("AccumulatorVariable").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);

        // 创建Accumulator变量
        // 需要调用SparkContext的accumulator()方法
        Accumulator<Integer> sum = sc.accumulator(0);

        List<Integer> numbersList = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 9, 8);
        JavaRDD<Integer> numbers = sc.parallelize(numbersList);

        numbers.foreach(new VoidFunction<Integer>() {
            @Override
            public void call(Integer integer) throws Exception {
                // 然后在函数内部，就可以对Accumulator变量，调用add()方法，累加值
                sum.add(integer);
            }
        });
        // 在driver程序中，可以调用accumulator的value()方法，获取其值
        System.out.println(sum.value());

    }
}

scala 版本：

package cn.rzlee.spark.scala

import org.apache.spark.{Accumulable, Accumulator, SparkConf, SparkContext}

object AccumulatorValiable {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local")
    val sc = new SparkContext(conf)

    val sum: Accumulator[Int]  = sc.accumulator(0)
    val numbersArray = Array(1,2,3,4,5,6,7,8,9)

    val numbers = sc.parallelize(numbersArray,1)
    numbers.foreach(number=>sum+=number)
    println(sum.value)
  }
}