pyspark - 将收集的列表转换为元组

Posted 2021-05-01

我的数据框如下：

+----------------------------------+
| invoice_id | newcolor            |
+------------+---------------------+
|         1  | [red, white, green] | 
+------------+---------------------+
|         2  | [red, green]        |       
+------------+---------------------+

我需要一个新栏目如下：

[('red', 'color'), ('white', 'color), ('green','color)]
[('red', 'color'), ('green','color)]

答案

您可以将udf函数定义为

from pyspark.sql import functions as F
from pyspark.sql import types as T
def addColor(x):
    return [[color, 'color'] for color in x]

udfAddColor = F.udf(addColor, T.ArrayType(T.StringType()))

然后将它与.withColumn一起使用

df.withColumn('newcolor', udfAddColor(df.newcolor)).show(truncate=False)

你应该得到你想要的输出

+----------+----------------------------------------------+
|invoice_id|newcolor                                      |
+----------+----------------------------------------------+
|1         |[[red, color], [white, color], [green, color]]|
|2         |[[red, color], [green, color]]                |
+----------+----------------------------------------------+