Spark Scala数据框具有单个Group By的多个聚合[重复]

Posted 2023-04-13

【中文标题】Spark Scala数据框具有单个Group By的多个聚合[重复]

【英文标题】：Spark Scala Data Frame to have multiple aggregation of single Group By [duplicate]

【发布时间】：2019-06-17 17:34:10

【问题描述】：

Spark Scala 数据框具有单个分组的多个聚合。 例如

val groupped = df.groupBy("firstName", "lastName").sum("Amount").toDF()

但是如果我需要 Count、Sum、Max 等呢

/* Below Does Not Work  , but this is what the intention is  
val groupped = df.groupBy("firstName", "lastName").sum("Amount").count().toDF()
*/

输出 groupped.show()

--------------------------------------------------
| firstName | lastName| Amount|count | Max | Min  |
--------------------------------------------------

【问题讨论】：

// 计算最大年龄和平均工资，按部门和性别分组。 ds.groupBy($"department", $"gender").agg(Map( "salary" -> "avg", "age" -> "max" )) 请参阅文档中的 groupyBy 示例spark.apache.org/docs/2.3.0/api/scala/…跨度>

@user10958683 True.. 它是重复的，但 Zaks 的答案更具可读性

【参考方案1】：

case class soExample(firstName: String, lastName: String, Amount: Int)
val df =  Seq(soExample("me", "zack", 100)).toDF

import org.apache.spark.sql.functions._

val groupped = df.groupBy("firstName", "lastName").agg(
     sum("Amount"),
     mean("Amount"), 
     stddev("Amount"),
     count(lit(1)).alias("numOfRecords")
   ).toDF()

display(groupped)