如何将spark数据帧列名和行数据转换为json数据

Posted 2021-03-28

我有一个pyspark数据帧

 spark = SparkSession
        .builder
        .appName("NPS_TF")
        .getOrCreate()
 df2 = spark.createDataFrame([
        ("unknown", 1,2,3)
    ], ["Assign", "xs[0]","xs[1]","xs[2]"])

 df2.limit(1).show()

如何将部分数据帧列名称和第一行数据转换为此json格式：

{"fields": ["xs[0]", "xs[1]", "xs[2]"], "values": [[1,2,3]]}

答案

检查此解决方案

df2 = spark.createDataFrame([("unknown", 1,2,3)], ["Assign", "xs[0]","xs[1]","xs[2]"])
fields = ['xs[0]', 'xs[1]', 'xs[2]']
values = df2.rdd.map(lambda p: [p[field] for field in fields]).collect()

json_obj = {
    'fields': fields,
    'values': values
}