在 pyspark 中查找过去 n 天内出现的 id

Posted 2023-03-31

【中文标题】在 pyspark 中查找过去 n 天内出现的 id

【英文标题】：finding the occurrence of id's in past n days in pyspark

【发布时间】：2021-07-15 18:18:23

【问题描述】：

我有这个下面的数据集，并想计算过去 n 天 id 的出现次数。我正在尝试使用 pyspark 中的 windows 函数来做到这一点

id	date
123	7/14/2021
123	7/13/2021
123	7/11/2021
123	6/1/2021
234	7/14/2021
234	7/1/2021
234	1/13/2020
234	6/1/2021

预期输出：

id	occurance in last 5 days	occurance in last 10 days
123	2	3
234	1	2

下面是我试过的代码

select * ,
count(id) over(
  partition by id 
  order by cast(date as timestamp)
  range between interval 5 days preceding and current row
)as id_cnt
from
t1

【参考方案1】：

select 
   id
   , count(*) 'occurrence in last 10 days'
   , count(when date >= today - interval 5 dyas then 1 end) 'occurrence in last 5 days', 
from t1
where date between today and today - interval 10 days
group by id

【参考方案2】：

PySpark 方式 将字符串转换为日期格式 取当前日期并获取日差 按真实值分组

df = spark.createDataFrame([(123,"07-14-2021"),(123,"07-10-2021"),(123,"07-11-2021"),(123,"06-11-2021"),(234,"07-14-2021"),(234,"07-11-2021"),(234,"01-13-2020"),(234,"06-01-2021")],["id","date"])
df = df.withColumn("date", F.to_date(F.col("date"),"MM-dd-yyyy"))
df = df.withColumn("current_dt", F.current_date())
df = df.withColumn("day_diff", F.datediff("current_dt","date"))
df = df.withColumn("5-days-diff", F.when(F.col("day_diff") <= 5, 1).otherwise(0)).withColumn("10-days-diff", F.when(F.col("day_diff") <=10, 1).otherwise(0))
df_grp = df.groupBy("id").agg(F.sum("5-days-diff").alias("5-days-diff"), F.sum("10-days-diff").alias("10-days-diff"))
df.show()
df_grp.show()
+---+----------+----------+--------+-----------+------------+
| id|      date|current_dt|day_diff|5-days-diff|10-days-diff|
+---+----------+----------+--------+-----------+------------+
|123|2021-07-14|2021-07-16|       2|          1|           1|
|123|2021-07-10|2021-07-16|       6|          0|           1|
|123|2021-07-11|2021-07-16|       5|          1|           1|
|123|2021-06-11|2021-07-16|      35|          0|           0|
|234|2021-07-14|2021-07-16|       2|          1|           1|
|234|2021-07-11|2021-07-16|       5|          1|           1|
|234|2020-01-13|2021-07-16|     550|          0|           0|
|234|2021-06-01|2021-07-16|      45|          0|           0|
+---+----------+----------+--------+-----------+------------+

+---+-----------+------------+
| id|5-days-diff|10-days-diff|
+---+-----------+------------+
|123|          2|           3|
|234|          2|           2|
+---+-----------+------------+