将 scala 数据框列组合成单个案例类
Posted
技术标签:
【中文标题】将 scala 数据框列组合成单个案例类【英文标题】：combine scala dataframe columns into single case class 【发布时间】：2019-02-28 20:21:15 【问题描述】：我有一个如下所示的数据框：
+--------+-----+--------------------+
| uid| iid| color|
+--------+-----+--------------------+
|41344966| 1305| red|
|41344966| 1305| green|
我想尽可能高效地做到这一点：
+--------+--------------------+
| uid| recommendations|
+--------+--------------------+
|41344966| [[2174, red...|
|41345063| [[2174, green...|
|41346177| [[2996, orange...|
|41349171| [[2174, purple...|
res98: org.apache.spark.sql.Dataset[userRecs] = [uid: int, recommendations: array<struct<iid:int,color:string>>]
所以我想按 uid 将记录分组到一个对象数组中。每个对象都是一个带有参数 iid 和颜色的类。
case class itemData (iid: Int, color: String)
case class userRecs (uid: Int, recommendations: Array[itemData])
【问题讨论】：
【参考方案1】：这是你想要的吗？
scala> case class itemData (iid: Int, color: String)
defined class itemData
scala> case class userRecs (uid: Int, recommendations: Array[itemData])
defined class userRecs
scala> val df = spark.createDataset(Seq(
(41344966,1305,"red"),
(41344966,1305,"green"),
(41344966,2174,"red"),
(41345063,2174,"green"),
(41346177,2996,"orange"),
(41349171,2174,"purple")
)).toDF("uid", "iid", "color")
df: org.apache.spark.sql.DataFrame = [uid: int, iid: int ... 1 more field]
scala> (df.select($"uid", struct($"iid",$"color").as("itemData"))
.groupBy("uid")
.agg(collect_list("itemData").as("recommendations"))
.as[userRecs]
.show())
+--------+--------------------+
| uid| recommendations|
+--------+--------------------+
|41344966|[[1305, red], [13...|
|41345063| [[2174, green]]|
|41346177| [[2996, orange]]|
|41349171| [[2174, purple]]|
+--------+--------------------+
【讨论】：
以上是关于将 scala 数据框列组合成单个案例类的主要内容，如果未能解决你的问题，请参考以下文章