如何将字符串解析为 HIVE 中的映射数组

Posted 2023-03-23

【中文标题】如何将字符串解析为 HIVE 中的映射数组

【英文标题】：How to parse an string to an array of maps in HIVE

【发布时间】：2021-09-09 18:26:19

【问题描述】：

我有一个从系统日志中提取的配置单元表。数据以一种奇怪的格式（映射数组）编码，其中数组的每个元素都包含field_name，它是value。列类型为 STRING。就像下面的例子一样：

select 1 as user_id, '["field":"name", "value":"Bob", "field":"gender", "value":"M"]' as user_info
union all
select 2 as user_id, '["field":"gender", "value":"F", "field":"age", "value":22, "field":"name", "value":"Ana"]' as user_info;

它创建了这样的东西：

user_id	user_info
1	["field":"name", "value":"Bob", "field":"gender", "value":"M"]
2	["field":"gender", "value":"F", "field":"age", "value":22, "field":"name", "value":"Ana"]

请注意，数组大小并不总是相同的。我正在尝试将地图数组转换为简单的地图。然后，这就是我所期望的结果：

user_id	user_info
1	"name":"Bob", "gender":"M"
2	"name":"Ana", "gender":"F", "age":22

我计划通过 3 个步骤实现：(1) 解析字符串列以创建地图数组，(2) 分解数组（使用横向视图），(3) 收集字段列表并将它们分组user_id

我正在努力完成第一步：解析字符串列以创建地图数组。任何帮助将不胜感激：D

【参考方案1】：

查看代码中的 cmets。要转换为映射的字符串数组由此split(user_info, '(?<=\\) *, *(?=\\)') 生成。然后将其分解，并将每个元素转换为地图。

with mydata as
(select 1 as user_id, '["field":"name", "value":"Bob", "field":"gender", "value":"M"]' as user_info
union all
select 2 as user_id, '["field":"gender", "value":"F", "field":"age", "value":22, "field":"name", "value":"Ana"]' as user_info
)

select user_id,
       --build new map
       str_to_map(concat('name:', name, nvl(concat(',','gender:', gender),''),  nvl(concat(',','age:', age),'') )) as user_info
from 
(
select user_id, 
      --get name, gender, age, aggregate by user_id
      max(case when user_info['field'] = 'name' then user_info['value'] end) name,
      max(case when user_info['field'] = 'gender' then user_info['value'] end) gender,
      max(case when user_info['field'] = 'age' then user_info['value'] end) age
      
from      
(
select s.user_id, 
       --remove  and ", convert to map
       str_to_map(regexp_replace(e.element,'^\\| *"|\\$','')) as user_info 
from
(
select user_id, regexp_replace(user_info, '^\\[|\\]$','') as user_info -- remove []
 from mydata
)s lateral view outer explode(split(user_info, '(?<=\\) *, *(?=\\)'))e as element --split by comma between  with optional spaces in between
) s
group by user_id
)s

结果：

user_id   user_info 
1        "name":"Bob","gender":"M"
2        "name":"Ana","gender":"F","age":"22"