如何避免大型数据集的Pandas DataFrame中的for循环

Posted 2021-03-27

你能告诉我一种优化这段代码的方法吗？由于大型数据集，需要几十分钟才能完成...

df['sinistre'] = 0
for index_sin, row_sin in sinistre1.iterrows():
    date_surv = row_sin['DATESURV']
    quit_sin = df.loc[df['id_police'] == row_sin['id_police']]
    for index, row in quit_sin.iterrows():
        if row['DATEEFFE'] < date_surv < row['DATE_FIN']:
            df['sinistre'][index] = 1

以下是DataFrames sinistre1和df的示例数据集：

>>> sinistre1
  id_police id_sinistre    DATESURV
0      p123        s123  30/05/2017
1      p123        s124  30/11/2017
2      p123        s125  29/02/2018
3      b123        s126  28/02/2018
4      b123        s127  30/05/2018
>>> df
  id_police    DATEEFFE    DATE_FIN  prime  prime2
0      p123  24/01/2017  24/02/2017      0       0
1      p123  24/11/2017  24/12/2017      0      30
2      p123  25/02/2018  25/03/2018     10      10
3      b123  24/02/2018  24/03/2018     20      20
4      b123  24/03/2018  24/04/2018     30       0

这是预期的输出（这个想法是当sinistre1的DATESURV在DATEEFFE和DATE_FIN区间内，然后我标记sinistre）：

  id_police    DATEEFFE    DATE_FIN  prime  prime2  sinistre
0      p123  24/01/2017  24/02/2017      0       0         0
1      p123  24/11/2017  24/12/2017      0      30         1
2      p123  25/02/2018  25/03/2018     10      10         1
3      b123  24/02/2018  24/03/2018     20      20         1
4      b123  24/03/2018  24/04/2018     30       0         0

如果我无法避免循环，请显示更好的循环方式...提前谢谢！

答案

使用：

1. 第一个函数to_datetime用于转换所有日期时间列
2. 创建计数器列sinistre1以获得最后检查成员资格
3. 使用DataFrame.merge和left连接数据
4. 由Series.between和inclusive=True过滤并获得sinistre1系列中的s列
5. 最后用sinistre1用Series.isin检查最终s，转换为整数到True/False到1/0映射：

---

df['DATEEFFE'] = pd.to_datetime(df['DATEEFFE'])
df['DATE_FIN'] = pd.to_datetime(df['DATE_FIN'])
sinistre1['DATESURV'] = pd.to_datetime(sinistre1['DATESURV'])

df['sinistre1'] = np.arange(len(df))
df1 = df.merge(sinistre1, on='id_police', how='left')
mask = df1['DATESURV'].between(df1['DATEEFFE'], df1['DATE_FIN'], inclusive=False)
s = df1.loc[mask, 'sinistre1']
print (s)
4    1
8    2
9    3
Name: sinistre1, dtype: int32

df['sinistre1'] = df['sinistre1'].isin(s).astype(int)
#alternative
#df['sinistre1'] = np.where(df['sinistre1'].isin(s), 1, 0)
print (df)
  id_police   DATEEFFE   DATE_FIN  prime  prime2  sinistre1
0      p123 2017-01-24 2017-02-24      0       0          0
1      p123 2017-11-24 2017-12-24      0      30          1
2      p123 2018-02-25 2018-03-25     10      10          1
3      b123 2018-02-24 2018-03-24     20      20          1
4      b123 2018-03-24 2018-04-24     30       0          0

编辑：

df['DATEEFFE'] = pd.to_datetime(df['DATEEFFE'])
df['DATE_FIN'] = pd.to_datetime(df['DATE_FIN'])
sinistre1['DATESURV'] = pd.to_datetime(sinistre1['DATESURV'])

df1 = df.merge(sinistre1, on='id_police', how='left')
#filter all rows
df2 = df1[df1['DATESURV'].between(df1['DATEEFFE'], df1['DATE_FIN'], inclusive=False)]
print (df2)
  id_police   DATEEFFE   DATE_FIN  prime  prime2 id_sinistre   DATESURV
4      p123 2017-11-24 2017-12-24      0      30        s124 2017-11-30
8      p123 2018-02-25 2018-03-25     10      10        s125 2018-02-28
9      b123 2018-02-24 2018-03-24     20      20        s126 2018-02-28

#left join merge with inditator True
df = df.merge(df2.drop('DATESURV', axis=1), how='left', indicator=True)
#sinistre1 column - first rename and then compare both for 1, esle 0
df = df.rename(columns={'_merge':'sinistre1'})
df['sinistre1'] = df['sinistre1'].eq('both').astype(int)
print (df)
  id_police   DATEEFFE   DATE_FIN  prime  prime2 id_sinistre  sinistre1
0      p123 2017-01-24 2017-02-24      0       0         NaN          0
1      p123 2017-11-24 2017-12-24      0      30        s124          1
2      p123 2018-02-25 2018-03-25     10      10        s125          1
3      b123 2018-02-24 2018-03-24     20      20        s126          1
4      b123 2018-03-24 2018-04-24     30       0         NaN          0

另一答案

正如我在评论中所说。接受的答案和合并现在没有意义，因为我认为OP想要比较两个数据帧中的每一行，因此需要数据帧id_sinistre中的关键df。或者想要像以下一样使用combine_first：

df_merge = df.merge(sinistre1, on='id_police', how='left')
df_merge['DATESURV'] = pd.to_datetime(df_merge['DATESURV'])

df_merge['sinistre'] = np.where(df_merge['DATESURV'].between(df_merge['DATEEFFE'], df_merge['DATE_FIN']), 1, 0)

df_merge = df_merge.drop(['DATESURV', 'id_sinistre'], axis=1)

print(df_merge)
    DATEEFFE   DATE_FIN id_police  prime  prime2  sinistre
0 2017-01-24 2017-02-24      p123      0       0         0
1 2017-11-24 2017-12-24      p123      0      30         1
2 2018-02-25 2018-03-25      p123     10      10         1
3 2018-02-24 2018-03-24      b123     20      20         1
4 2018-03-24 2018-04-24      b123     30       0         0

另一答案

1. 在'id_police'列上左合并两个数据集
2. 使用业务逻辑编写lambda函数并将其应用于合并的数据集

即（未经测试）：

t_table = pd.merge(sinistre1, df, how='left', on='id_police')
t_table['sinistre'] = [1 if row['DATEEFFE'] < ds< row['DATE_FIN'] else
0 for row,_ in t_table.iterrows()]