取消嵌套存储在列中的 JSON 字符串 [BigQuery]
Posted
技术标签:
【中文标题】取消嵌套存储在列中的 JSON 字符串 [BigQuery]【英文标题】：Unnesting JSON String stored in a column [BigQuery] 【发布时间】：2021-04-16 13:12:11 【问题描述】：我有一个表，其中一列包含原始 JSON 字符串，如下所示：
存储在 order_lines 中的 JSON 示例：
"STR_BLK_002":
"amount":167,
"type":"part spare",
"total_discount":0,
"color":"Black",
"is_out_of_stock":false,
"variable_fields":
"Size":"XL",
"trueColor":"Black"
,
"category_id":"44356721",
"status_list":[
"id":1,
"time":"2021-04-01T15:01:54.746Z",
"status":"ORDER PLACED"
,
"id":2,
"time":"2021-04-02T10:31:00.397Z",
"status":"PACKED"
,
"id":3,
"time":"2021-04-04T10:31:01.719Z",
"status":"SHIPPED"
,
"id":3,
"time":"2021-04-04T18:12:06.896Z",
"status":"SHIPPED"
],
"product_id":270,
"price_per_quantity":167,
"quantity":1,
"maximum_quantity":10,
"variant_name":"Helmet strap",
"current_status":30,
"estimated_delivery":"09 Apr 2021",
"total_before_discount":167,
"delivery_statuses":[
"time":"2021-04-01T15:10:13.594Z",
"status":"FULFILLABLE"
,
"time":"2021-04-02T10:31:00.397Z",
"status":"PACKED"
,
"time":"2021-04-03T10:31:01.197Z",
"status":"READY_TO_SHIP"
,
"time":"2021-04-04T10:31:01.719Z",
"status":"DISPATCHED"
,
"time":"2021-04-04T18:12:06.896Z",
"status":"SHIPPED"
],
"sku_code":"STR_BLK_002"
我想取消嵌套这个字符串，以便可以单独访问键值对。此外，sku_code（上面共享的示例中的“STR_BLK_002”）在任何其他列中均不可用，并且该字符串可以包含多个单个 sku，因此如果有 2 个对应于订单的 sku，则 JSON 字符串将为：
"STR_BLK_002":
"amount":167,
"type":"part spare",
"total_discount":0,
"color":"Black",
"is_out_of_stock":false,
"variable_fields":
"Size":"XL",
"trueColor":"Black"
,
"category_id":"44356721",
"status_list":[
"id":1,
"time":"2021-04-01T15:01:54.746Z",
"status":"ORDER PLACED"
,
"id":2,
"time":"2021-04-02T10:31:00.397Z",
"status":"PACKED"
,
"id":3,
"time":"2021-04-04T10:31:01.719Z",
"status":"SHIPPED"
,
"id":3,
"time":"2021-04-04T18:12:06.896Z",
"status":"SHIPPED"
],
"product_id":270,
"price_per_quantity":167,
"quantity":1,
"maximum_quantity":10,
"variant_name":"Helmet strap",
"current_status":3,
"estimated_delivery":"09 Apr 2021",
"total_before_discount":167,
"delivery_statuses":[
"time":"2021-04-01T15:10:13.594Z",
"status":"FULFILLABLE"
,
"time":"2021-04-02T10:31:00.397Z",
"status":"PACKED"
,
"time":"2021-04-03T10:31:01.197Z",
"status":"READY_TO_SHIP"
,
"time":"2021-04-04T10:31:01.719Z",
"status":"DISPATCHED"
,
"time":"2021-04-04T18:12:06.896Z",
"status":"SHIPPED"
],
"sku_code":"STR_BLK_002"
,
"STR_BLK_008":
"amount":590,
"type":"accessory",
"total_discount":0,
"color":"blue",
"is_out_of_stock":false,
"variable_fields":
"Size":"XL",
"trueColor":"prussian blue"
,
"category_id":"65577970",
"status_list":[
"id":1,
"time":"2021-04-06T15:01:54.746Z",
"status":"ORDER PLACED"
,
"id":2,
"time":"2021-04-07T10:31:00.397Z",
"status":"PACKED"
,
"id":3,
"time":"2021-04-07T10:31:01.719Z",
"status":"SHIPPED"
,
"id":3,
"time":"2021-04-08T18:12:06.896Z",
"status":"SHIPPED"
],
"product_id":276,
"price_per_quantity":590,
"quantity":1,
"maximum_quantity":5,
"variant_name":"Car Perfume",
"current_status":3,
"estimated_delivery":"09 Apr 2021",
"total_before_discount":590,
"delivery_statuses":[
"time":"2021-04-06T15:10:13.594Z",
"status":"FULFILLABLE"
,
"time":"2021-04-07T10:31:00.397Z",
"status":"PACKED"
,
"time":"2021-04-07T10:31:01.197Z",
"status":"READY_TO_SHIP"
,
"time":"2021-04-08T10:31:01.719Z",
"status":"DISPATCHED"
,
"time":"2021-04-10T18:12:06.896Z",
"status":"SHIPPED"
],
"sku_code":"STR_BLK_008"
我想将此信息分成单独的列，以便获取每个 SKU 的对应值。
【问题讨论】：
此链接抛出 404 错误。 @SergeyGeron cloud.google.com/bigquery/docs/reference/standard-sql/… 【参考方案1】：所以基本上我认为你想要做的是首先将你的列转换为一个结构数组，而不是这样：
"STR_BLK_002": ...,
"STR_BLK_003": ...
你有这样的东西：
[
"amount":167,
"type":"part spare",
"total_discount":0,
...
,
"amount":590,
"type":"accessory",
"total_discount":0,
...
]
使用该格式的数据，您可以利用UNNEST 将每个条目放入自己的行中，然后使用JSON functions 将字段提取到自己的列中，例如JSON_EXTRACT_SCALAR
为了做到这一点，我构建了一个javascript UDF 来查找对象中的键，然后遍历每个键以创建一个结构数组。
CREATE TEMP FUNCTION format_json(str STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS r"""
var obj = JSON.parse(str);
var keys = Object.keys(obj);
var arr = [];
for (i = 0; i < keys.length; i++)
arr.push(JSON.stringify(obj[keys[i]]));
return arr;
""";
SELECT
JSON_EXTRACT_SCALAR(formatted_json,'$.amount') as amount
,JSON_EXTRACT_SCALAR(formatted_json,'$.type') as type
,JSON_EXTRACT_SCALAR(formatted_json,'$.total_discount') as total_discount
,JSON_EXTRACT_SCALAR(formatted_json,'$.color') as color
,JSON_EXTRACT_SCALAR(formatted_json,'$.is_out_of_stock') as is_out_of_stock
,JSON_EXTRACT_SCALAR(formatted_json,'$.sku_code') as sku_code
from
testing.json_test
left join unnest(format_json(order_lines)) as formatted_json
结果如下：
【讨论】：
【参考方案2】：下面应该会给你一个好的开始
select
json_extract_scalar(line, '$.sku_code') as sku_code,
json_extract_scalar(line, '$.amount') as amount,
json_extract_scalar(line, '$.type') as type,
json_extract_scalar(line, '$.total_discount') as total_discount,
json_extract_scalar(line, '$.color') as color,
json_extract_scalar(line, '$.variable_fields.Size') as Size,
json_extract_scalar(line, '$.variable_fields.trueColor') as trueColor,
from `project.dataset.table`,
unnest(split(regexp_replace(regexp_replace(order_lines, r'\s', ''), r'"STR_BLK_\d+":', '"STR_BLK":'),'"STR_BLK":')) order_line with offset,
unnest([struct('' || trim(order_line, ',') || '' as line)])
where offset > 0
如果应用于您问题中的第一个示例 - 输出是
如果应用于您问题中的第二个示例 - 输出是
希望，您可以将此示例扩展到您心中的任何最终目标
【讨论】：
非常感谢，但仍有一个疑问。 sku_code 可以是任何东西，不仅限于“STR_BLK”。在这种情况下我该如何处理？ @米哈伊尔以上是关于取消嵌套存储在列中的 JSON 字符串 [BigQuery]的主要内容，如果未能解决你的问题，请参考以下文章
将列从 varchar 转换为 nvarchar 是不是会更改存储在列中的字符串的编码？