具有可迭代对象的字典字典到具有多索引的可迭代对象索引的熊猫数据框

Posted 2023-03-12

【中文标题】具有可迭代对象的字典字典到具有多索引的可迭代对象索引的熊猫数据框

【英文标题】：Dictionary of dictionaries with iterables to pandas Dataframe with multiindex by index of iterables

【发布时间】：2019-11-09 10:51:17

【问题描述】：

我想要一个来自字典字典的多索引 pandas DataFrame。内部字典包含相同长度的列表/numpy 数组。

x = 'a': 'x': [0, 1, 2], 'y': [1 ,2 ,3],
     'b': 'x': [4, 6, 8], 'y': [9, 8, 7]
some_function(x)
=>
          x      y  <- first index
      0 1 2  0 1 2  <- second index
    a 0 1 2  1 2 3
    b 4 6 8  9 8 7

这是我已经尝试过的，但是有没有更有效的方法？只和熊猫一样？或者有没有可以解决这个问题的 pandas 函数？

def dict_of_dicts_of_collections_to_multiindex_df(dict_of_dicts_of_collections):

    x = dict_of_dicts_of_collections

    result = 

    for outer_key, intermediate_dict in x.items():
        result[outer_key] = 

        for intermediate_key, collection in intermediate_dict.items():
            try:
                for i, e in enumerate(collection):
                    result[outer_key][(intermediate_key, i)] = e
            except TypeError:
                pass

    return pd.DataFrame(result).T

【参考方案1】：

我为这个问题创建了两种替代方法，并对结果进行了计时。还包括其他答案以及原始功能。

from copy import deepcopy
import pandas as pd
from collections import defaultdict
import numpy as np


x = 'a': 'x': [0, 1, 2], 'y': [1 ,2 ,3],
     'b': 'x': [4, 6, 8], 'y': [9, 8, 7]


test = deepcopy(x)
for i in range(1000):
    test.update(f'a_i':test['a'])

test2 = k:key: val*300 for key, val in v.items() for k, v in x.items()
print(len(test2['a']['x']))
for i in range(1000):
    test2.update(f'a_i':test2['a'])

def dict_of_dicts_of_collections_to_multiindex_df(dict_of_dicts_of_collections):
    x = dict_of_dicts_of_collections
    result = 
    for outer_key, intermediate_dict in x.items():
        result[outer_key] = 
        for intermediate_key, collection in intermediate_dict.items():
            try:
                for i, e in enumerate(collection):
                    result[outer_key][(intermediate_key, i)] = e
            except TypeError:
                pass
    return pd.DataFrame(result).T


def out_from_other_answer(data):
    df = pd.DataFrame(data).T    
    cols = df.columns
    df = pd.concat([df[col].apply(pd.Series) for col in cols], axis=1)
    #tweaked to  avoid hardcoding [0, 1, 2]
    df.columns = pd.MultiIndex.from_product([cols, range(len(df.columns)//len(cols))])
    return df

def out2(dict_of_dicts):
    df = pd.DataFrame(list(dict_of_dicts.values()))
    out_df = pd.concat([pd.DataFrame(df[col].values.tolist())
                        for col in df.columns
                        ],
                        axis=1,
                        keys=df.columns,
                        )
    out_df.index = dict_of_dicts.keys()
    return out_df

def out3(data):
    temp = defaultdict(list)
    for d in list(data.values()):
        for k, v in d.items():
            temp[k].append(v)
    out = pd.concat([pd.DataFrame(v) for v in temp.values()], axis=1, keys=temp.keys())
    out.index = data.keys()
    return out

让我们看看结果。 用小数据x。

%timeit dict_of_dicts_of_collections_to_multiindex_df(x)
%timeit out_from_other_answer(x)
%timeit out2(x)
%timeit out3(x)
1.63 ms ± 102 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
4.49 ms ± 492 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.44 ms ± 102 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
1.49 ms ± 98.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

使用更多行但相同的列进行测试。

%timeit dict_of_dicts_of_collections_to_multiindex_df(test)
%timeit out_from_other_answer(test)
%timeit out2(test)
%timeit out3(test)
70.3 ms ± 10.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
494 ms ± 40.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
4.81 ms ± 185 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.37 ms ± 115 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

更多的行和更多的列，test2

%timeit dict_of_dicts_of_collections_to_multiindex_df(test2)
%timeit out_from_other_answer(test2)
%timeit out2(test2)
%timeit out3(test2)
1.24 s ± 19.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
1.1 s ± 63.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
590 ms ± 39.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
598 ms ± 44 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

似乎每个解决方案在数据维度增加方面受到的影响不同。总的来说，out3 似乎是最好的选择。本质上，最好在处理输入数据之前更改其布局。

【参考方案2】：

您是否考虑过一些逆向工程？我的意思是构建一个多索引数据框并检查它在您print(df.to_dict()) 时的样子使用此answer 的第一部分，我们可以得到您想要的输出，

import pandas as pd
data = [[0, 1, 2, 1 ,2 ,3],
        [4, 6, 8, 9, 8, 7]]

df = pd.DataFrame(data)
df.colums =  pd.MultiIndex.from_product([['x','y'], [0,1,2]])
df.index = ['a', 'b']

print(df.to_dict())

('x', 0): 'a': 0, 'b': 4,
 ('x', 1): 'a': 1, 'b': 6,
 ('x', 2): 'a': 2, 'b': 8,
 ('y', 0): 'a': 1, 'b': 9,
 ('y', 1): 'a': 2, 'b': 8,
 ('y', 2): 'a': 3, 'b': 7

因此，如果您可以将数据分为两个列表，则可以使用 pd.MultiIndex.from_product 技巧。

否则

import pandas as pd
data = 'a': 'x': [0, 1, 2], 'y': [1 ,2 ,3],
        'b': 'x': [4, 6, 8], 'y': [9, 8, 7]

df = pd.DataFrame(data).T

# you obeserve that expand list to columns
df["x"].apply(pd.Series)

# then using this for every column
# and again pd.MultiIndex.from_product
# gives you the desired output
cols = df.columns
df = pd.concat([df[col].apply(pd.Series) for col in cols], axis=1)
df.columns = pd.MultiIndex.from_product([cols, [0,1,2]])