pandas json_normalize 所有列都有嵌套字典展平
Posted
技术标签:
【中文标题】pandas json_normalize 所有列都有嵌套字典展平【英文标题】：pandas json_normalize all columns have nested dictionaries flattening 【发布时间】：2019-06-10 02:53:39 【问题描述】：我有一个从非官方谷歌字典 API 返回的嵌套字典 (json)。
看起来像这样：
'word': 'slack',
'phonetic': '/slak/',
'meaning': 'adjective': ['definition': 'Not taut or held tightly in position; loose.',
'example': 'a slack rope',
'synonyms': ['loose',
'limp',
'not taut',
'not tight',
'hanging',
'flapping'],
'definition': '(of business) characterized by a lack of work or activity; quiet.',
'example': 'business was rather slack',
'definition': 'Having or showing laziness or negligence.',
'example': 'slack accounting procedures',
'synonyms': ['lax',
'negligent',
'neglectful',
'remiss',
'careless',
'slapdash',
'slipshod',
'lackadaisical',
'lazy',
'inefficient',
'incompetent',
'inattentive',
'offhand',
'casual',
'disorderly',
'disorganized'],
'definition': '(of a tide) neither ebbing nor flowing.',
'example': 'soon the water will become slack, and the tide will turn'],
'noun': ['definition': 'The part of a rope or line which is not held taut; the loose or unused part.',
'example': 'I picked up the rod and wound in the slack',
'synonyms': ['looseness', 'play', 'give'],
'definition': 'Casual trousers.',
'definition': 'A spell of inactivity or laziness.',
'example': 'he slept deeply, refreshed by a little slack in the daily routine',
'synonyms': ['lull',
'pause',
'respite',
'spell of inactivity',
'interval',
'break',
'hiatus',
'breathing space']],
'verb': ['definition': 'Loosen (something, especially a rope).',
'definition': 'Decrease or reduce in intensity, quantity, or speed.',
'example': 'the flow of blood slacked off',
'synonyms': ['reduce',
'lessen',
'slacken',
'slow',
'ease off',
'ease up'],
'definition': 'Work slowly or lazily.',
'example': 'she reprimanded her girls if they were slacking',
'synonyms': ['idle',
'shirk',
'be inactive',
'be lazy',
'be indolent',
'sit back and do nothing',
'waste time',
'lounge about'],
'definition': 'Slake (lime).'],
'adverb': ['definition': 'Loosely.',
'example': 'their heads were hanging slack in attitudes of despair']
这就是slack这个词的意思。 为了得到这个含义，我们可以谷歌这个含义或简单地使用以下代码：
import numpy as np
import pandas as pd
import json
from pandas.io.json import json_normalize
from io import StringIO
import requests
word = 'slack'
url = 'https://googledictionaryapi.eu-gb.mybluemix.net/?define=' + word
response = requests.get(url)
content = response.content.decode('utf-8') # list of ugly strings
j = json.loads(content) # json list having nested dictionary
j = j[0]
j
现在，字典 j 有三个键。
j.keys() # dict_keys(['word', 'phonetic', 'meaning'])
我主要感兴趣的是意思：
j['meaning'].keys() # dict_keys(['adjective', 'noun', 'verb', 'adverb'])
为了获取 pandas 数据框，我使用了以下代码：
json_normalize(data=j['meaning'])
这给出了一个只有 4 列的数据框。
这里，每个词性（形容词、名词等）都必须有“定义”键，“示例”和“同义词”是可选的。
j['meaning']['adjective'][0].keys() # dict_keys(['definition', 'example', 'synonyms'])
如何获取具有 4 * 3 = 12 列的数据框，列名称如 adjective_definition、adjective_example、...、verb_synonyms？
我试图从以下链接中获得一些想法：
http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.io.json.json_normalize.htmlhttps://www.kaggle.com/jboysen/quick-tutorial-flatten-nested-json-in-pandas/notebookpandas.io.json.json_normalize with very nested json
但是，无法解决问题。我们将不胜感激。
【问题讨论】：
【参考方案1】：我认为使用json_normalize 的record_path 参数可以解决您的问题。由于 record_path 旨在成为 json 对象或记录列表的单一路径，因此我不得不多次调用 json_normalize 然后将结果连接起来以获取包含所需数据的数据框。您还可以尝试使用 record_prefix 参数来设置列命名约定。希望这会有所帮助！
from pandas.io.json import json_normalize
from io import StringIO
import requests
word = 'slack'
url = 'https://googledictionaryapi.eu-gb.mybluemix.net/?define=' + word
response = requests.get(url)
content = response.content.decode('utf-8') # list of ugly strings
j = json.loads(content) # json list having nested dictionary
j = j[0]
df_adj = json_normalize(data=j['meaning'], record_path=["adjective"], record_prefix="adjective.")
df_verb = json_normalize(data=j['meaning'], record_path=["verb"], record_prefix="verb.")
df_adv = json_normalize(data=j['meaning'], record_path=["adverb"], record_prefix="adverb.")
df_noun = json_normalize(data=j['meaning'], record_path=["noun"], record_prefix="noun.")
df = pd.concat([df_adj, df_verb, df_adv, df_noun], axis=1)
print(df.head(3))
【讨论】：
【参考方案2】：有点混乱的解决方案，但我认为它有效。以j 开头作为示例字典：
res = pd.concat([json_normalize(v, meta=['definition', 'example', 'synonyms']).add_prefix(k + '_')
for k, v in j['meaning'].items()],
axis=1)
# The output is super wide and hard to read in console output,
# but hopefully this confirms the output is (close to) what you need
res
adjective_definition \
0 Not taut or held tightly in position; loose.
1 (of business) characterized by a lack of work or activity; quiet.
2 Having or showing laziness or negligence.
3 (of a tide) neither ebbing nor flowing.
adjective_example \
0 a slack rope
1 business was rather slack
2 slack accounting procedures
3 soon the water will become slack, and the tide will turn
adjective_synonyms \
0 [loose, limp, not taut, not tight, hanging, flapping]
1 NaN
2 [lax, negligent, neglectful, remiss, careless, slapdash, slipshod, lackadais...
3 NaN
noun_definition \
0 The part of a rope or line which is not held taut; the loose or unused part.
1 Casual trousers.
2 A spell of inactivity or laziness.
3 NaN
noun_example \
0 I picked up the rod and wound in the slack
1 NaN
2 he slept deeply, refreshed by a little slack in the daily routine
3 NaN
noun_synonyms \
0 [looseness, play, give]
1 NaN
2 [lull, pause, respite, spell of inactivity, interval, break, hiatus, breathi...
3 NaN
verb_definition \
0 Loosen (something, especially a rope).
1 Decrease or reduce in intensity, quantity, or speed.
2 Work slowly or lazily.
3 Slake (lime).
verb_example \
0 NaN
1 the flow of blood slacked off
2 she reprimanded her girls if they were slacking
3 NaN
verb_synonyms \
0 NaN
1 [reduce, lessen, slacken, slow, ease off, ease up]
2 [idle, shirk, be inactive, be lazy, be indolent, sit back and do nothing, wa...
3 NaN
adverb_definition adverb_example
0 Loosely. their heads were hanging slack in attitudes of despair
1 NaN NaN
2 NaN NaN
3 NaN NaN
【讨论】：
感谢您的单行。【参考方案3】：实际上12列的想法被认为不是最好的，然后在玩了一会儿代码之后，我想出了更好看的结果。
import numpy as np
import pandas as pd
import json
from pandas.io.json import json_normalize
import requests
word = 'slack'
url = 'https://googledictionaryapi.eu-gb.mybluemix.net/?define=' + word
response = requests.get(url)
content = response.content.decode('utf-8') # list of ugly strings
data = json.loads(content) # json list having nested dictionary
data = data[0]
df = pd.DataFrame()
for i in data['meaning'].keys():
x = json_normalize(data=data['meaning'][i])
x['part_of_speech'] = i
df = df.append(x,sort=False)
df = df[['part_of_speech', 'definition', 'example', 'synonyms']]
df
给出结果：
part_of_speech definition example synonyms
0 adjective Not taut or held tightly in position; loose. a slack rope [loose, limp, not taut, not tight, hanging, fl...
1 adjective (of business) characterized by a lack of work ... business was rather slack NaN
2 adjective Having or showing laziness or negligence. slack accounting procedures [lax, negligent, neglectful, remiss, careless,...
3 adjective (of a tide) neither ebbing nor flowing. soon the water will become slack, and the tide... NaN
0 noun The part of a rope or line which is not held t... I picked up the rod and wound in the slack [looseness, play, give]
1 noun Casual trousers. NaN NaN
2 noun A spell of inactivity or laziness. he slept deeply, refreshed by a little slack i... [lull, pause, respite, spell of inactivity, in...
0 verb Loosen (something, especially a rope). NaN NaN
1 verb Decrease or reduce in intensity, quantity, or ... the flow of blood slacked off [reduce, lessen, slacken, slow, ease off, ease...
2 verb Work slowly or lazily. she reprimanded her girls if they were slacking [idle, shirk, be inactive, be lazy, be indolen...
3 verb Slake (lime). NaN NaN
0 adverb Loosely. their heads were hanging slack in attitudes of... NaN
【讨论】：
以上是关于pandas json_normalize 所有列都有嵌套字典展平的主要内容，如果未能解决你的问题，请参考以下文章
Pandas json_normalize 不会展平所有嵌套字段