Python中多个词典和列表字典的高效快速数据存储和处理，以及列表的两个词典的交集

Posted 2021-04-05

我有一个表格例子字典： -

 all_ways = {key1:[list1], key2:[list2], ... keyN[listN]}

我想只找到ith列表中的那些元素，使得它是至少另一个列表的一部分，例如jth list (i!=j)，然后对于所有键只存储满足上述条件的那些元素。

除了使用多个循环之外，还有什么好办法吗？

例：-

key1: [1,2,3,4,5,100,111,123] 
key2: [1,2,3,13,15,17] 
key3:[100,110,111,1000] 

回答：

key1:[1,2,3,100,111] 
key2:[1,2,3]
key3:[100,111]

答案

您可以在线性时间（无嵌套循环）中实现此目的，以达到列表元素的总数：

from collections import Counter
from itertools import chain

d = {'key1': [1,2,3,4,5,100,111,123],
     'key2': [1,2,3,13,15,17],
     'key3': [100,110,111,1000]}

# histogramm over the chained lists
c = Counter(chain(*d.values()))  
# c = Counter(chain(*map(set, d.values())))  # if there are duplicates in the lists

# for every key k in d, pick only those elements x from their value lists v
# that occur more than once overall (thus in multiple lists)
result = {k: [x for x in v if c[x] > 1] for k, v in d.items()}
# {'key1': [1, 2, 3, 100, 111], 
#  'key2': [1, 2, 3], 
#  'key3': [100, 111]}

collections.Counter包含每个唯一列表元素中元素出现的列表数量，同时忽略单个列表中的重复项（通过set）。然后，嵌套的理解只选择计数大于1的元素，因此保证它们出现在多个列表中。

另一答案

以下是一些其他方法，对现实数据集进行基准测试。该答案假设列表中没有重复项，输出顺序（每个键的重复项）无关紧要。

import numpy as np
from itertools import groupby, chain
from collections import Counter

d = {'key'+str(i): list(np.random.randint(0, 1000000, 50)) for i in range(1, 50000)}

def return_dups_jp1(d):
    def get_dups(a):
        seen = set()
        dup = set()
        for x in a:
            if x not in seen:
                seen.add(x)
            else:
                dup.add(x)
        return dup

    def apply_dups(d, dups):
        return {k: list(set(v) & dups) for k, v in d.items()}

    return apply_dups(d, get_dups(chain.from_iterable(d.values())))

def return_dups_jp2(d):
    b = sorted(chain.from_iterable(d.values()))
    dups = {group[0] for group in groupby(b) if len(list(group[1])) > 1}
    return {k: list(set(v) & dups) for k, v in d.items()}

def return_dups_sch(d):
    c = Counter(chain(*d.values()))
    return {k: [x for x in v if c[x] > 1] for k, v in d.items()}

标杆：

x = return_dups_jp1(d)
y = return_dups_jp2(d)
z = return_dups_sch(d)

assert all(set(x['key'+str(i)]) == set(y['key'+str(i)]) for i in range(1, 50000))
assert all(set(y['key'+str(i)]) == set(z['key'+str(i)]) for i in range(1, 50000))

%timeit return_dups_jp1(d)
# 1 loop, best of 3: 2.56 s per loop

%timeit return_dups_jp2(d)
# 1 loop, best of 3: 5.12 s per loop

%timeit return_dups_sch(d)
# 1 loop, best of 3: 2.66 s per loop