Spark Project中动态更改表中要查询的列

Posted 2023-04-15

【中文标题】Spark Project中动态更改表中要查询的列

【英文标题】：Change Columns to be queried in the table dynamically in Spark Project

【发布时间】：2019-02-06 08:02:56

【问题描述】：

我在 Spark 2.3 中运行以下 SQL 查询，如下所示：

val dataJoin = s"""SELECT
                date,
                a.mth,
        weekday,
        if(a.x_days=b.x_days,b.y,c.z) as total,
        case when (a.x - day(date)) +1 <=0 then 1
                         when (a.x - day(date)) +1 > 366 then 999
                         else (a.x - day(date)) +1  end  as glt,
        GROSS
        FROM tableA a 
        left join tableB b
        on a.mth = b.mth and a.x_days=b.x_days"""

val data =sparkVal.sql(dataJoin)

我想让查询变为动态，即如果我想选择更多列或更少列，则不应触及模块，只需在一个地方更改即可。

如何让它实现呢？有没有办法像这样配置，XML 什么的？

【参考方案1】：

您似乎想修改查询的 select 子句中的列。

您可以在 s""" 语法中使用 $ 进行字符串插值。动态列可以单独存储在列表中，然后使用 mkString(",") 连接。 您可以执行以下操作

val sel_list = List("date","a.mth","weekday").mkString(",")

val dataJoin = s"""SELECT
           $sel_list ,
    if(a.x_days=b.x_days,b.y,c.z) as total,
    case when (a.x - day(date)) +1 <=0 then 1
                     when (a.x - day(date)) +1 > 366 then 999
                     else (a.x - day(date)) +1  end  as glt,
    GROSS
    FROM tableA a
    left join tableB b
    on a.mth = b.mth and a.x_days=b.x_days"""
   println(dataJoin)

结果

SELECT
           date,a.mth,weekday , 
    if(a.x_days=b.x_days,b.y,c.z) as total,
    case when (a.x - day(date)) +1 <=0 then 1
                     when (a.x - day(date)) +1 > 366 then 999
                     else (a.x - day(date)) +1  end  as glt,
    GROSS
    FROM tableA a
    left join tableB b
    on a.mth = b.mth and a.x_days=b.x_days