快速搜索许多字符串以查找许多字典键
Posted
技术标签:
【中文标题】快速搜索许多字符串以查找许多字典键【英文标题】：Searching many strings for many dictionary keys, quickly 【发布时间】：2017-02-05 20:16:36 【问题描述】：我有一个独特的问题，我主要希望找出一些方法来加快这段代码的速度。我有一组存储在数据框中的字符串，每个字符串中都有多个名称，并且我知道在此步骤之前的名称数量，如下所示：
print df
description num_people people
'Harry ran with sally' 2 []
'Joe was swinging with sally' 2 []
'Lola Dances alone' 1 []
我正在使用带有我希望在描述中找到的键的字典，如下所示：
my_dict='Harry':'1283','Joe':'1828','Sally':'1298', 'Cupid':'1982'
然后使用 iterrows 在每个字符串中搜索匹配项，如下所示：
for index, row in df.iterrows():
row.people=[key for key in my_dict if re.findall(key,row.desciption)]
当运行时它以
结束print df
description num_people people
'Harry ran with sally' 2 ['Harry','Sally']
'Joe was swinging with sally' 2 ['Joe','Sally']
'Lola Dances alone' 1 ['Lola']
我看到的问题是，这段代码完成工作仍然相当慢，而且我有大量的描述和超过1000 键。有没有更快的方法来执行这个操作，比如使用找到的人数？
【问题讨论】：
【参考方案1】：更快的解决方案：
#strip ' in start and end of text, create lists from words
splited = df.description.str.strip("'").str.split()
#filtering
df['people'] = splited.apply(lambda x: [i for i in x if i in my_dict.keys()])
print (df)
description num_people people
0 'Harry ran with Sally' 2 [Harry, Sally]
1 'Joe was swinging with Sally' 2 [Joe, Sally]
2 'Lola Dances alone' 1 [Lola]
时间安排：
#[30000 rows x 3 columns]
In [198]: %timeit (orig(my_dict, df))
1 loop, best of 3: 3.63 s per loop
In [199]: %timeit (new(my_dict, df1))
10 loops, best of 3: 78.2 ms per loop
df['people'] = [[],[],[]]
df = pd.concat([df]*10000).reset_index(drop=True)
df1 = df.copy()
my_dict='Harry':'1283','Joe':'1828','Sally':'1298', 'Lola':'1982'
def orig(my_dict, df):
for index, row in df.iterrows():
df.at[index, 'people']=[key for key in my_dict if re.findall(key,row.description)]
return (df)
def new(my_dict, df):
df.description = df.description.str.strip("'")
splited = df.description.str.split()
df.people = splited.apply(lambda x: [i for i in x if i in my_dict.keys()])
return (df)
print (orig(my_dict, df))
print (new(my_dict, df1))
【讨论】：
以上是关于快速搜索许多字符串以查找许多字典键的主要内容，如果未能解决你的问题，请参考以下文章