如何将字符串 dict 转换为 pyspark 数据框？

Posted 2023-02-16

【中文标题】如何将字符串 dict 转换为 pyspark 数据框？

【英文标题】：How to convert string dict to pyspark dataframe?

【发布时间】：2021-05-14 23:24:26

【问题描述】：

    "input":[("James", "Sales", 3000),
        ("Michael", "Sales", 4600),
        ("Robert", "Sales", 4100),
        ("Maria", "Finance", 3000),
        ("James", "Sales", 3000),
        ("Scott", "Finance", 3300),
        ("Jen", "Finance", 3900),
        ("Jeff", "Marketing", 3000),
        ("Kumar", "Marketing", 2000),
        ("Saif", "Sales", 4100)],
    
    "deptColumns" : ["employee_name", "department", "salary"]

【参考方案1】：

假设数据是一个字符串，你可以eval它并使用spark.createDataFrame将它加载到一个spark数据帧中：

data = """
    "input":[("James", "Sales", 3000),
        ("Michael", "Sales", 4600),
        ("Robert", "Sales", 4100),
        ("Maria", "Finance", 3000),
        ("James", "Sales", 3000),
        ("Scott", "Finance", 3300),
        ("Jen", "Finance", 3900),
        ("Jeff", "Marketing", 3000),
        ("Kumar", "Marketing", 2000),
        ("Saif", "Sales", 4100)],
    
    "deptColumns" : ["employee_name", "department", "salary"]
"""

import ast
data = ast.literal_eval(data)

df = spark.createDataFrame(data['input'], data['deptColumns'])

df.show()
+-------------+----------+------+
|employee_name|department|salary|
+-------------+----------+------+
|        James|     Sales|  3000|
|      Michael|     Sales|  4600|
|       Robert|     Sales|  4100|
|        Maria|   Finance|  3000|
|        James|     Sales|  3000|
|        Scott|   Finance|  3300|
|          Jen|   Finance|  3900|
|         Jeff| Marketing|  3000|
|        Kumar| Marketing|  2000|
|         Saif|     Sales|  4100|
+-------------+----------+------+