如何用java创建一个用于构建广播变量spark的类标记？

Posted 2021-04-01

我需要在我的JAVA函数中创建一个borodcast，一个字符串列表的广播

Broadcast<List<String>>broadcastSp = sc.broadcast(Tvalue,classTag<T>evidence);

T值是“my_list”（List<String>)，但问题是如何创建第二部分classTag<T>evidence。

答案

应该没有必要。在Java中工作时，不应该使用org.apache.spark.SparkContext，它是为Scala设计的。而是使用org.apache.spark.api.java.JavaSparkContext

引用官方文档：

import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.SparkConf;

SparkConf conf = new SparkConf().setAppName("broadcast").setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);

它的广播方法不需要ClassTags（example from the official docs）：

Broadcast<int[]> broadcastVar = sc.broadcast(new int[] {1, 2, 3});

另一答案

这是我的功能：

public static JavaSparkContext sc;
public static Broadcast < List < String >> broadcastP;
public static void main(String args[]) throws Exception {
    sc = new JavaSparkContext("local", "test");
    .
    .
    .
    private static List < String > fFtItemSets(JavaRDD < String > base_initiale) throws Exception {
      List < String > kMinusOneSets;
      List < String > k_sets;

      int i = 1;
      k_sets = remplir_ksets_intiale(base_initiale);
      System.out.println(k_sets);
      List < String > k_sets1 = k_sets;
      int NB = k_sets.size();
      while (NB > 1) {
        kMinusOneSets = k_sets;
        k_sets = Jointure(kMinusOneSets);

        k_sets = Elagage(k_sets, k_sets1, i);
        System.out.println(k_sets);


        Broadcast < List < String >> broadcastSp = sc.broadcast(k - sets, List < String > );
        //main.sc.broadcast(k_sets);

        System.out.println(broadcastSp.value());


        k_sets = tr_freq(k_sets, broadcastnSp, base_initiale, i);
        k_sets1 = k_sets;
        NB++;
        i++;

      }
      return k_sets;
    }