如何避免大型数据集的Pandas DataFrame中的for循环

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了如何避免大型数据集的Pandas DataFrame中的for循环相关的知识,希望对你有一定的参考价值。

你能告诉我一种优化这段代码的方法吗?由于大型数据集,需要几十分钟才能完成...

df['sinistre'] = 0
for index_sin, row_sin in sinistre1.iterrows():
    date_surv = row_sin['DATESURV']
    quit_sin = df.loc[df['id_police'] == row_sin['id_police']]
    for index, row in quit_sin.iterrows():
        if row['DATEEFFE'] < date_surv < row['DATE_FIN']:
            df['sinistre'][index] = 1

以下是DataFrames sinistre1df的示例数据集:

>>> sinistre1
  id_police id_sinistre    DATESURV
0      p123        s123  30/05/2017
1      p123        s124  30/11/2017
2      p123        s125  29/02/2018
3      b123        s126  28/02/2018
4      b123        s127  30/05/2018
>>> df
  id_police    DATEEFFE    DATE_FIN  prime  prime2
0      p123  24/01/2017  24/02/2017      0       0
1      p123  24/11/2017  24/12/2017      0      30
2      p123  25/02/2018  25/03/2018     10      10
3      b123  24/02/2018  24/03/2018     20      20
4      b123  24/03/2018  24/04/2018     30       0

这是预期的输出(这个想法是当sinistre1的DATESURV在DATEEFFEDATE_FIN区间内,然后我标记sinistre):

  id_police    DATEEFFE    DATE_FIN  prime  prime2  sinistre
0      p123  24/01/2017  24/02/2017      0       0         0
1      p123  24/11/2017  24/12/2017      0      30         1
2      p123  25/02/2018  25/03/2018     10      10         1
3      b123  24/02/2018  24/03/2018     20      20         1
4      b123  24/03/2018  24/04/2018     30       0         0

如果我无法避免循环,请显示更好的循环方式...提前谢谢!

答案

使用:

  1. 第一个函数to_datetime用于转换所有日期时间列
  2. 创建计数器列sinistre1以获得最后检查成员资格
  3. 使用DataFrame.mergeleft连接数据
  4. Series.betweeninclusive=True过滤并获得sinistre1系列中的s
  5. 最后用sinistre1Series.isin检查最终s,转换为整数到True/False1/0映射:

df['DATEEFFE'] = pd.to_datetime(df['DATEEFFE'])
df['DATE_FIN'] = pd.to_datetime(df['DATE_FIN'])
sinistre1['DATESURV'] = pd.to_datetime(sinistre1['DATESURV'])

df['sinistre1'] = np.arange(len(df))
df1 = df.merge(sinistre1, on='id_police', how='left')
mask = df1['DATESURV'].between(df1['DATEEFFE'], df1['DATE_FIN'], inclusive=False)
s = df1.loc[mask, 'sinistre1']
print (s)
4    1
8    2
9    3
Name: sinistre1, dtype: int32

df['sinistre1'] = df['sinistre1'].isin(s).astype(int)
#alternative
#df['sinistre1'] = np.where(df['sinistre1'].isin(s), 1, 0)
print (df)
  id_police   DATEEFFE   DATE_FIN  prime  prime2  sinistre1
0      p123 2017-01-24 2017-02-24      0       0          0
1      p123 2017-11-24 2017-12-24      0      30          1
2      p123 2018-02-25 2018-03-25     10      10          1
3      b123 2018-02-24 2018-03-24     20      20          1
4      b123 2018-03-24 2018-04-24     30       0          0

编辑:

df['DATEEFFE'] = pd.to_datetime(df['DATEEFFE'])
df['DATE_FIN'] = pd.to_datetime(df['DATE_FIN'])
sinistre1['DATESURV'] = pd.to_datetime(sinistre1['DATESURV'])

df1 = df.merge(sinistre1, on='id_police', how='left')
#filter all rows
df2 = df1[df1['DATESURV'].between(df1['DATEEFFE'], df1['DATE_FIN'], inclusive=False)]
print (df2)
  id_police   DATEEFFE   DATE_FIN  prime  prime2 id_sinistre   DATESURV
4      p123 2017-11-24 2017-12-24      0      30        s124 2017-11-30
8      p123 2018-02-25 2018-03-25     10      10        s125 2018-02-28
9      b123 2018-02-24 2018-03-24     20      20        s126 2018-02-28

#left join merge with inditator True
df = df.merge(df2.drop('DATESURV', axis=1), how='left', indicator=True)
#sinistre1 column - first rename and then compare both for 1, esle 0
df = df.rename(columns={'_merge':'sinistre1'})
df['sinistre1'] = df['sinistre1'].eq('both').astype(int)
print (df)
  id_police   DATEEFFE   DATE_FIN  prime  prime2 id_sinistre  sinistre1
0      p123 2017-01-24 2017-02-24      0       0         NaN          0
1      p123 2017-11-24 2017-12-24      0      30        s124          1
2      p123 2018-02-25 2018-03-25     10      10        s125          1
3      b123 2018-02-24 2018-03-24     20      20        s126          1
4      b123 2018-03-24 2018-04-24     30       0         NaN          0
另一答案

正如我在评论中所说。接受的答案和合并现在没有意义,因为我认为OP想要比较两个数据帧中的每一行,因此需要数据帧id_sinistre中的关键df。或者想要像以下一样使用combine_first

df_merge = df.merge(sinistre1, on='id_police', how='left')
df_merge['DATESURV'] = pd.to_datetime(df_merge['DATESURV'])

df_merge['sinistre'] = np.where(df_merge['DATESURV'].between(df_merge['DATEEFFE'], df_merge['DATE_FIN']), 1, 0)

df_merge = df_merge.drop(['DATESURV', 'id_sinistre'], axis=1)

print(df_merge)
    DATEEFFE   DATE_FIN id_police  prime  prime2  sinistre
0 2017-01-24 2017-02-24      p123      0       0         0
1 2017-11-24 2017-12-24      p123      0      30         1
2 2018-02-25 2018-03-25      p123     10      10         1
3 2018-02-24 2018-03-24      b123     20      20         1
4 2018-03-24 2018-04-24      b123     30       0         0
另一答案
  1. 在'id_police'列上左合并两个数据集
  2. 使用业务逻辑编写lambda函数并将其应用于合并的数据集

即(未经测试):

t_table = pd.merge(sinistre1, df, how='left', on='id_police')
t_table['sinistre'] = [1 if row['DATEEFFE'] < ds< row['DATE_FIN'] else
0 for row,_ in t_table.iterrows()]

以上是关于如何避免大型数据集的Pandas DataFrame中的for循环的主要内容,如果未能解决你的问题,请参考以下文章

pandas.concat 和 numpy.append 的大型数据集的内存错误

大型数据集的 PyCharm Python 输出崩溃

在 Pandas 中加入两个大型数据集的最佳方法

Python Pandas 改进了目前需要约 400 分钟运行的大型数据集的计算时间

Datafram 实现作为正文发送邮件

python中大型数据集的文本分类