将嵌套的 json 导入为具有多个嵌套的表

Posted 2023-03-25

【中文标题】将嵌套的 json 导入为具有多个嵌套的表

【英文标题】：Importing a nested json as a table with multiple nesting

【发布时间】：2016-05-09 13:42:31

【问题描述】：

在我们的数据中，我们有 json 字段，其中包括重复的部分，以及无限的嵌套可能性（我目前的示例非常简单）。在看到 BQ 重复字段和记录后，我决定尝试将数据重组为重复记录字段，因为我们的用例与分析相关，然后想测试数据的不同用例，看看哪种方法更有效（时间/成本/难度）用于我们打算对其进行的分析。我创建了一个我想上传到 BQ 的示例 json 记录，它使用了我认为我们需要的所有功能（我已经验证使用 http://jsonlint.com/）：

    "aid": "6dQcrgMVS0",
    "hour": "2016042723",
    "unixTimestamp": "1461814784",
    "browserId": "BdHOHp2aL9REz9dXVeKDaxdvefE3Bgn6NHZcDQKeuC67vuQ7PBIXXJda3SOu",
    "experienceId": "EXJYULQOXQ05",
    "experienceVersion": "1.0",
    "pageRule": "V1XJW61TPI99UWR",
    "userSegmentRule": "67S3YVMB7EMQ6LP",
    "branch": [
        "branchId": "1",
        "branchType": "userSegments",
        "itemId": "userSegment67S3YVMB7EMQ6LP",
        "headerId": "null",
        "itemMethod": "null"
    , 
        "branchId": "1",
        "branchType": "userSegments",
        "itemId": "userSegment67S3YVMB7EMQ6LP",
        "headerId": "null",
        "itemMethod": "null"
    ],
    "event": [
        "eventId": "546",
        "eventName": "testEvent",
        "eventDetails": [
            "key": "a",
            "value": "1"
        , 
            "key": "b",
            "value": "2"
        , 
            "key": "c",
            "value": "3"
        ]
    , 
        "eventId": "547",
        "eventName": "testEvent2",
        "eventDetails": [
            "key": "d",
            "value": "4"
        , 
            "key": "e",
            "value": "5"
        , 
            "key": "f",
            "value": "6"
        ]
    ]

我正在使用BQ接口，将这个json上传到具有以下结构的表中：

[
    
        "name": "aid",
        "type": "STRING",
        "mode": "NULLABLE"
    ,
    
        "name": "hour",
        "type": "INTEGER",
        "mode": "NULLABLE"
    ,
    
        "name": "unixTimestamp",
        "type": "INTEGER",
        "mode": "NULLABLE"
    ,
    
        "name": "browserId",
        "type": "STRING",
        "mode": "NULLABLE"
    ,
    
        "name": "experienceId",
        "type": "STRING",
        "mode": "NULLABLE"
    ,
    
        "name": "experienceVersion",
        "type": "FLOAT",
        "mode": "NULLABLE"
    ,
    
        "name": "pageRule",
        "type": "STRING",
        "mode": "NULLABLE"
    ,
    
        "name": "userSegmentRule",
        "type": "STRING",
        "mode": "NULLABLE"
    ,
    
        "name": "branch",
        "type": "RECORD",
        "mode": "REPEATED",
        "fields": [
            
                "name": "branchId",
                "type": "STRING",
                "mode": "NULLABLE"
            ,
            
                "name": "branchType",
                "type": "STRING",
                "mode": "NULLABLE"
            ,
            
                "name": "itemId",
                "type": "STRING",
                "mode": "NULLABLE"
            ,
            
                "name": "headerId",
                "type": "STRING",
                "mode": "NULLABLE"
            ,
            
                "name": "itemMethod",
                "type": "STRING",
                "mode": "NULLABLE"
            
        ]
    ,
    
        "name": "event",
        "type": "RECORD",
        "mode": "REPEATED",
        "fields": [
            
                "name": "evenId",
                "type": "STRING",
                "mode": "NULLABLE"
            ,
            
                "name": "eventName",
                "type": "STRING",
                "mode": "NULLABLE"
            ,
            
                "name": "eventDetails",
                "type": "RECORD",
                "mode": "REPEATED",
                "fields": [
                    
                        "name": "key",
                        "type": "STRING",
                        "mode": "NULLABLE"
                    ,
                    
                        "name": "value",
                        "type": "STRING",
                        "mode": "NULLABLE"
                    
                ]
            
        ]
    
]

我的工作失败了

JSON parsing error in row starting at position 0 in <file_id>. Expected key (error code: invalid)

我可能无法在一个表中进行多个嵌套，但该错误似乎更像是解析 JSON 本身存在问题。我能够生成并成功导入带有简单重复记录的 json（参见下面的示例）：

"eventId": "546",
"eventName": "testEvent",
"eventDetails": [
    "key": "a",
    "value": "1"
, 
    "key": "b",
    "value": "2"
, 
    "key": "c",
    "value": "3"
]

感谢任何建议。

【参考方案1】：

您的架构似乎没有任何问题，因此 BigQuery 应该能够使用您的架构加载您的数据。

首先，确保您上传的是newline-delimited JSON to BigQuery。您的示例行在 JSON 行的中间有许多换行符，解析器试图将每一行解释为单独的 JSON 行。

其次，看起来您的架构在“event”记录中具有键“evenId”，但您的示例行具有键“eventId”。

【讨论】：

感谢回答，我使用的是 JSON 换行符分隔格式。数据存储在一行中的 txt 文件中（为了更清晰，我只发布了格式化的文件）。修复字段名称拼写错误仍然会产生相同的错误。

谢谢，我明白你的意思了。根据您所说的，我能够成功上传示例行。