在多个条件下合并来自多个数据帧的数据

Posted 2021-05-04

我想合并多个数据帧，但仅当密钥匹配且日期范围在df1中的“InitialAdmit”日期范围的90天内时才会合并。我想保留df1中的所有行，并且只合并df2，df3等的其他行，只要它们与键匹配并且在日期范围内。

注意：首先合并dfs然后考虑日期范围条件对我不起作用。我第一次使用这种方法，但有很多条件，合并成功，但后来因为日期范围超出限制，我让脚本删除行。我需要以某种方式保留df1中的所有行。

Python Pandas: Merging data frames on multiple conditions - 这个问题很相似，但似乎合并然后应用条件。我认为更好的方法是应用条件，然后在条件满足时合并。虽然，我愿意接受建议。

DataFrames：

a = {'Key': [100000204, 100000255, 100000271,100000286,100000628], 
 'InitialAdmit': ['2012-06-04', '2012-05-03', '2012-01-16', '2012-10-26', '2012-02-21'],
 '90DayRange': ['2012-09-02', '2012-08-01', '2012-04-15', '2013-01-24', '2012-05-21']
}
df1 = pandas.DataFrame(data=a)
df1


b = {'Key': [100000208, 100000255, 100000723,100000286,100000866], 
 'InitialAdmit': ['2012-01-22', '2012-06-03', '2012-10-26', '2012-11-26', '2012-05-11'],
}
df2 = pandas.DataFrame(data=b)
df2 

c = {'Key': [100000255, 100000255, 100000702,100000221,100000628], 
 'InitialAdmit': ['2012-06-22', '2012-10-03', '2012-10-26', '2012-11-26', '2012-04-11'],
}
df3 = pandas.DataFrame(data=c)
df3

脚本：

df_NotIncludedRows = pandas.DataFrame()
df_final = pandas.DataFrame()
dfs = [df2] #I plan to add more dataframes so I'm iterating through this list of dfs 

for df in dfs: #iterate through each df in dfs
    for key in df1["Key"]: #iterate through each key found in column 'Id'
        if key in df["Key"]: # find any matching key from df1 in df2 (part of my issue exists here)
            if (df["Admit"] >= df1["InitialAdmit"]) | (df["Admit"] <= df1["90DayRange"]):
                df_final = pandas.merge(df1,df.loc[:],on='Key',how='left') # my df.loc[:] is a little off i think
            else:
                df_NotIncludedRows = df_NotIncludedRows.append(df.loc[:]) # same df.loc[:] issue i believe   
df_NotIncludedRows

答案

我仍然建议合并然后过滤，这里我们使用布尔索引和combine_first

df=df1.merge(df2,on='Key')
m=(df.InitialAdmit_y>=df.InitialAdmit_x)&(df.InitialAdmit_y<=df.InitialAdmit_x)
df1.set_index('Key').combine_first(df[m].set_index('Key'))


Out[215]: 
          90DayRange InitialAdmit InitialAdmit_x InitialAdmit_y
Key                                                            
100000204 2012-09-02   2012-06-04            NaT            NaT
100000255 2012-08-01   2012-05-03     2012-05-03     2012-06-03
100000271 2012-04-15   2012-01-16            NaT            NaT
100000286 2013-01-24   2012-10-26     2012-10-26     2012-11-26
100000628 2012-05-21   2012-02-21            NaT            NaT

另一答案

考虑reduce使用左连接进行链合并。下面用3份df2进行演示。此外，下面假设InitialAdmit是数据帧的最后一列。根据需要重新排序。

import pandas 
import numpy
from functools import reduce    
...

# LIST OF DATAFRAMES WITH SUFFIXING OF INITIALADMIT TO AVOID NAME COLLISION
dfList = [d.rename(columns={'InitialAdmit':'InitialAdmit_' + str(i)}) 
          for i,d  in enumerate([df1, df2, df2, df2])]

# USER-DEFINED METHOD CONDITIONING ON LAST COLUMN
def mergefilter(x, y):
    tmp = pandas.merge(x, y, on='Key', how='left')
    tmp.loc[~(tmp.iloc[:, -1].between(tmp['InitialAdmit_0'], tmp['90DayRange'])), 
            tmp.columns[-1]] = numpy.nan

    return tmp

finaldf = reduce(mergefilter, dfList)

print(finaldf)
#    90DayRange InitialAdmit_0        Key InitialAdmit_1 InitialAdmit_2 InitialAdmit_3
# 0  2012-09-02     2012-06-04  100000204            NaN            NaN            NaN
# 1  2012-08-01     2012-05-03  100000255     2012-06-03     2012-06-03     2012-06-03
# 2  2012-04-15     2012-01-16  100000271            NaN            NaN            NaN
# 3  2013-01-24     2012-10-26  100000286     2012-11-26     2012-11-26     2012-11-26
# 4  2012-05-21     2012-02-21  100000628            NaN            NaN            NaN