Spark中RDD的反转

Posted 2023-04-15

【中文标题】Spark中RDD的反转

【英文标题】：Inversing of an RDD in spark

【发布时间】：2017-05-06 15:24:01

【问题描述】：

我有一个 RDD[(timestamp,(a,b,c))] 如下：

Timestamp   a       b       c
5:00 PM      523    384      40
6:00 PM      384    60     nan

我需要将上面的RDD转换为下面的RDD

key  values
a    [523,384]
b    [384,60]
c    [40,nan]

在 spark 中实现上述功能的最有效方法是什么？

【参考方案1】：

     val raw = spark.sparkContext.parallelize(Seq(
      ("5:00 PM","523" ,"384" ,"40"),
      ("6:00 PM","384","60","nan")))
      .toDF("Timestamp", "a", "b","c")

    // drop timestamp column 
    val data = raw.drop("Timestamp")

    // iterate through columns and return value as tuple 
    val newData = data.columns.map(colName =>
       (colName, data.select(colName).map(r=>r.getAs[String](0)).collect())
    )
    // create a new Datafrane
    val finalData = spark.sparkContext.parallelize(newData).toDF("key", "value")

    finalData.show()