使用三个单独的键连接两个数据帧

Posted 2023-04-15

【中文标题】使用三个单独的键连接两个数据帧

【英文标题】：Join two dataframes using three seperate keys

【发布时间】：2021-11-17 13:28:22

【问题描述】：

import pyspark.sql.functions as F

data1 = [("10/05/21", 1, "White", 3000), ("10/06/21", 2, "Blue", 4100), ("10/07/21", 3, "Green", 6200)]
df1 = spark.createDataFrame(data1, ["START", "KEY1", "Color", "OTHER"])

data2 = [(1, 2, 3, 3000), (2, 3, 2, 4100), (3, 1, 2, 6200)]
df2 = spark.createDataFrame(data2, ["KEY2", "KEY3", "KEY4", "NUMBER"])

df_result = df1.withColumnRenamed("START", "DATE1").join(
    df2,
    F.col("KEY1") == F.col("KEY2")
).select("DATE1", "KEY3", "KEY4").join(
    df1.withColumnRenamed("START", "DATE2"),
    F.col("KEY1") == F.col("KEY3")
).select("DATE1", "DATE2", "KEY4").join(
    df1.withColumnRenamed("START", "DATE3"),
    F.col("KEY1") == F.col("KEY4")
).select("DATE1", "DATE2", "DATE3")

df_result.show()

我正在尝试使用找到 here 的方法，但似乎无法加入它，因为说无法解决 key4。

【参考方案1】：

您在第一次加入后没有选择KEY4，导致加入后select 不可用，条件为KEY1 = KEY3。

df_result = df1.withColumnRenamed("START", "DATE1").join(
    df2,
    F.col("KEY1") == F.col("KEY2")
).select("DATE1", "KEY3", "KEY4").join(
    df1.withColumnRenamed("START", "DATE2"),
    F.col("KEY1") == F.col("KEY3")
).select("DATE1", "DATE2", "KEY4").join(
    df1.withColumnRenamed("START", "DATE3"),
    F.col("KEY1") == F.col("KEY4")
).select("DATE1", "DATE2", "DATE3")

输出

+--------+--------+--------+
|   DATE1|   DATE2|   DATE3|
+--------+--------+--------+
|10/05/21|10/06/21|10/07/21|
|10/07/21|10/05/21|10/06/21|
|10/06/21|10/07/21|10/06/21|
+--------+--------+--------+