更快地在新列 pandas 中添加未来间隔

Posted 2023-02-16

【中文标题】更快地在新列 pandas 中添加未来间隔

【英文标题】：adding future intervals in new column pandas faster

【发布时间】：2019-05-31 16:20:58

【问题描述】：

我想要实现的是将未来值附加到当前行的更快方法。我的数据框只有几 GB，因此在我的计算机上处​​理需要几个小时。我目前正在使用下面的代码来实现这一目标。但是我找不到用熊猫处理它的合适方法。我意识到在熊猫中遍历数据框效率低下。有没有大神可以帮忙？

days = pd.Series(data.day.unique())

for d in days:
    data_temp = data.loc[data['day'] == d]
    for i in range(0, 1439): #1439 number of min in a day

        t1 = data_temp.loc[data_temp['minutes'] == i]
        t2 = data_temp.loc[data_temp['minutes'] == i+5]
        t3 = data_temp.loc[data_temp['minutes'] == i+10]

        #Check if ID values exist in all three time intervals 
        ans = set(t1.ID) & set(t2.ID) & set(t3.ID)      
        ans_List = list(ans) 

        if (len(ans) >= 10):  #isolate only occurenses bigger than 10
            for j in range(10): 
                data_t1 = data_t1.append(t1.loc[t1.ID == ans_List[j]])
                data_t2 = data_t2.append(t2.loc[t2.ID == ans_List[j]])
                data_t3 = data_t3.append(t3.loc[t3.ID == ans_List[j]])

data_t1 = data_t1.reset_index(drop=True)
data_t2 = data_t2.reset_index(drop=True)
data_t3 = data_t3.reset_index(drop=True)

data_t1['a_t5'] = data_t2['a']
data_t1['b_t5'] = data_t2['b']
data_t1['c_t5'] = data_t2['c']

data_t1['a_t10'] = data_t3['a']
data_t1['b_t10'] = data_t3['b']
data_t1['c_t10'] = data_t3['c']

【参考方案1】：

附加可能有点致命。您可以享受一些加速。

第一印象是这几行：

for j in range(10): 
    data_t1 = data_t1.append(t1.loc[t1.ID == ans_List[j]])
    data_t2 = data_t2.append(t2.loc[t2.ID == ans_List[j]])
    data_t3 = data_t3.append(t3.loc[t3.ID == ans_List[j]])

可以通过改变这个来加快触摸速度：

import intertools # Put at top of doc


data_t1 = itertools.chain(data_t1,[t1.loc[t1.ID == ans_List[j]] for j in range(10)]) 
data_t2 = itertools.chain(data_t2,[t2.loc[t1.ID == ans_List[j]] for j in range(10)]) 
data_t3 = itertools.chain(data_t3,[t3.loc[t3.ID == ans_List[j]] for j in range(10)]) 

列表理解允许您避免不断追加，并且 intertools 可以快速有效地连接结果，这对于大型数据集尤其重要。