Spark UDF 找不到参数 num 的隐式值：Numeric [Nothing]

Posted 2023-03-23

【中文标题】Spark UDF 找不到参数 num 的隐式值：Numeric [Nothing]

【英文标题】：Spark UDF could not find implicit value for parameter num: Numeric[Nothing]

【发布时间】：2019-04-23 14:43:01

【问题描述】：

我正在尝试编写可以接受任何参数类型并返回该类型结果的通用 add 方法

 def addExactUDF[T](x: T, y: T)(implicit num: Numeric[T]): T = 
    import num._
    x + y
  

  def addExact(value1: Column, value2: Column, dataType: String): Column =
    dataType match 
      case "Int" => expr(s"addExactUDF(cast($value1 AS INT), cast($value2 AS INT))")
      case "Double" => expr(s"addExactUDF(cast($value1 AS DOUBLE), cast($value2 AS DOUBLE))")
    

现在当我尝试注册 UDF 时

object FilterFunctionsUtil extends MathFunctionsNameSpace with StringFunctionsNameSpace 
  lazy val registerAsSparkUdf: UserDefinedFunction = 
    val sqlContext = SparkSessionFactory.getSparkSession(Map(), Nil).sqlContext

    sqlContext.udf.register("addExactUDF", addExactUDF _)
    sqlContext.udf.register("subtractExactUDF", subtractExactUDF _)
  

给我报错

   Error:(36, 44) could not find implicit value for parameter num: Numeric[Nothing]
        sqlContext.udf.register("addExactUDF", addExactUDF _)
Error:(36, 44) not enough arguments for method addExactUDF: (implicit num: Numeric[Nothing])Nothing.
Unspecified value parameter num.
    sqlContext.udf.register("addExactUDF", addExactUDF _)

如何添加隐式来完成这项工作

【参考方案1】：

在注册 UDF 时指定隐式类型。您可能需要在需要时为每种类型注册单独的 UDF。

  import sqlContext.implicits._
  val df = sc.parallelize(Seq((1,2),(3,4))).toDF("val1","val2")

  df.createOrReplaceTempView("tempTable")
  sqlContext.cacheTable("tempTable")

  sqlContext.udf.register("addExtractIntUDF",addExtractUDF[Int] _)

  sqlContext.sql("select addExtractIntUDF(val1,val2) from temptable").show(false)

结果

+---------------------------------------------+ |UDF:addExtractIntUDF(val1,val2) | +----------------------------------------------------------+ |3                                                   | |7                                                 | +--------------------------------------------+