规范化复杂的嵌套 JSON 文件
Posted
技术标签:
【中文标题】规范化复杂的嵌套 JSON 文件【英文标题】：Normalize a complex nested JSON file 【发布时间】：2020-04-02 05:32:37 【问题描述】：我正在尝试将下面的 json 文件标准化为 4 个表 - “内容”、“模块”、“图像”和“另一个表中的其他所有内容”
"id": "0000050a",
"revision": 1580225050941,
"slot": "product-description",
"type": "E",
"create_date": 1580225050941,
"modified_date": 1580225050941,
"creator": "Auto",
"modifier": "Auto",
"audit_info":
"date": 1580225050941,
"source": "AutoService",
"username": "Auto"
,
"total_ID": 1,
"name": "Auto_A1AM78C64UM0Y8_B07JCJR5HW",
"content": [
"ID": ["B01"],
"content_revision": 1580225050941,
"template":
"module": [
"id": "module-11",
"text": null,
"header": [
"id": "title",
"value": null,
"decorators": []
],
"paragraph": [
"id": "description",
"value": [],
"decorators": []
],
"image": [
"id": "image",
"assetId": "/images/2cdabb786d10.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,970,300_PT0_SX970_V1__",
"src": "image.jpg",
"originalSrc": null
],
"integer": null,
"chart": null,
"list": null,
"video": null,
"gallery": null,
"composite": null,
"collection": null,
"product": null
,
"id": "module-6",
"text": null,
"header": [
"id": "title1",
"value": "Dest ",
"decorators": []
,
"id": "title2",
"value": "cc",
"decorators": []
,
"id": "title3",
"value": "Col",
"decorators": []
,
"id": "title4",
"value": "C",
"decorators": []
,
"id": "caption1",
"value": null,
"decorators": []
,
"id": "caption2",
"value": null,
"decorators": []
,
"id": "caption3",
"value": null,
"decorators": []
,
"id": "caption4",
"value": null,
"decorators": []
],
"paragraph": [
"id": "description1",
"value": [" Sport"],
"decorators": [
[]
]
,
"id": "description2",
"value": ["elements "],
"decorators": [
[]
]
,
"id": "description3",
"value": ["Film "],
"decorators": [
[]
]
,
"id": "description4",
"value": ["Our signature "],
"decorators": [
[]
]
],
"image": [
"id": "image1",
"assetId": "/images/dbbfc9873e31.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image2_.jpg",
"originalSrc": null
,
"id": "image2",
"assetId": "/images/f577ae005.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "test.jpg",
"originalSrc": null
,
"id": "image3",
"assetId": "/images/-0df21c5216d0.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image.jpg",
"originalSrc": null
,
"id": "image4",
"assetId": "/images/78d26b9c-408c-4299-8ea8-e9257f170320.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image.jpg",
"originalSrc": null
,
"id": "thumb1",
"assetId": "/images/-bbbfc9873e31.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image.jpg",
"originalSrc": null
,
"id": "thumb2",
"assetId": "/images/e56f577ae005.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image_.jpg",
"originalSrc": null
,
"id": "thumb3",
"assetId": "/images/0df21c5216d0.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image_.jpg",
"originalSrc": null
,
"id": "thumb4",
"assetId": "/images/-e9257f170320.jpg",
"alt": " ",
"viewLarger": null,
"styleCodes": "__CR0,0,300,300_PT0_SX300_V1__",
"src": "image.jpg",
"originalSrc": null
],
"integer": null,
"chart": null,
"list": null,
"video": null,
"gallery": null,
"composite": null,
"collection": null,
"product": null
],
"renderType": "VERTICAL"
,
"locale_data":
"locale": "en_US",
"identified_by": "MACHINE_DETECT"
],
"badges": []
我可以成功地将 JSON 扁平化为一个大数据帧，并将标头作为 JSON 路径。但我想将 JSON 规范化为单独的表。例如，模块表应该有 ID、Text、Header_ID、Header_Value 等列。 Image 表应该有Image_ID、Assest_ID、Src 等列。谁能帮我将此 JSON 规范化为 4 个表。
【问题讨论】：
查看jmespath 或glom 以获取有关如何使用嵌套数据的指导。 【参考方案1】：你可以使用https://towardsdatascience.com/flattening-json-objects-in-python-f5343c794b10定义的函数，如下，然后使用json_normalize：
import pandas as pd
import json
with open('test.json') as json_file:
data = json.load(json_file)
def flatten_json(y):
out =
def flatten(x, name=''):
if type(x) is dict:
for a in x:
flatten(x[a], name + a + '_')
elif type(x) is list:
i = 0
for a in x:
flatten(a, name + str(i) + '_')
i += 1
else:
out[name[:-1]] = x
flatten(y)
return out
module = flatten_json(data["content"][0])
module = pd.json_normalize(module)
然后，您要做的就是根据您描述的四个类别选择列。 输出是：
ID_0 content_revision ... locale_data_locale locale_data_identified_by
0 B01 1580225050941 ... en_US MACHINE_DETECT
然后您选择如下，例如为您的模块和图像 DataFrames：
module = df.loc[:,df.columns.str.contains("module")]
image = df.loc[:,df.columns.str.contains("image")]
例如你得到的模块结果是：
template_module_0_id ... template_module_1_product
0 module-11 ... None
然后，我给出模块DataFrame转换的例子，你只有两个模块，所以你可以在重命名列后做一个concat：
module1 = module.loc[:,module.columns.str.contains("module_0")]
module1.columns = module1.columns.str.replace("_0","")
module2 = module.loc[:,module.columns.str.contains("module_1")]
module2.columns = module2.columns.str.replace("_1","")
modules = pd.concat([module1,
module2])
你会得到：
template_module_id ... template_module_image_7_originalSrc
0 module-11 ... NaN
0 module-6 ... None
如果您有更多元素，另一个选择是直接在您想要的嵌套元素上使用 flatten_json 和 json_normalize 函数。
【讨论】：
感谢拉斐尔·阿杰拉德。我确实使用了 flatten 功能。我有很多 JSON 文件要处理，模块和图像的数量可能会有所不同。所以我试图以递归方式执行此操作，并针对任何级别的嵌套 JSON。以上是关于规范化复杂的嵌套 JSON 文件的主要内容，如果未能解决你的问题，请参考以下文章
如何使用 json_normalize 规范化嵌套的 json