Pandas：根据条件将值从一个数据帧合并到另一个数据帧

Posted 2023-02-15

【中文标题】Pandas：根据条件将值从一个数据帧合并到另一个数据帧

【英文标题】：Pandas: Merge values from one dataframe to another based on condition

【发布时间】：2022-01-20 05:59:45

【问题描述】：

使用模糊逻辑和fuzzywuzzy 模块，我能够将名称（来自一个数据帧）与短名称（来自另一个数据帧）匹配。这两个 Dataframe 还包含一个表 ISIN。

这是我应用逻辑后得到的数据框。

ISIN                                      Name Currency         Value  % Weight  Asset Type Comments/ Assumptions          matches
236   NaN            Partnerre Ltd 4.875% Perp Sr:J      USD  1.684069e+05    0.0004         NaN                   NaN
237   NaN  Berkley (Wr) Corporation 5.700% 03/30/58      USD  6.955837e+04    0.0002         NaN                   NaN
238   NaN             Tc Energy Corp Flt Perp Sr:11      USD  6.380262e+04    0.0001         NaN                   NaN   TC ENERGY CORP
239   NaN                      Cash and Equivalents      USD  2.166579e+07    0.0499         NaN                   NaN
240   NaN                                       AUM      NaN  4.338766e+08    0.9999         NaN                   NaN  AUM IND BARC US

创建了一个新列“匹配”，这基本上意味着第二个数据帧的短名称与第一个数据帧的名称匹配。

来自 dataframe1 的 ISIN 为空，来自 dataframe2 的 ISIN 存在。在随后的匹配中（第一个数据帧的名称和第二个数据帧的短名称），我想将第二个数据帧的相关 ISIN 添加到第一个数据帧。

如何从第二个数据帧获取 ISIN 到第一个数据帧，以便我的最终输出看起来像这样？

ISIN                                      Name Currency         Value  % Weight  Asset Type Comments/ Assumptions          matches
236   NaN            Partnerre Ltd 4.875% Perp Sr:J      USD  1.684069e+05    0.0004         NaN                   NaN
237   NaN  Berkley (Wr) Corporation 5.700% 03/30/58      USD  6.955837e+04    0.0002         NaN                   NaN
238   78s9             Tc Energy Corp Flt Perp Sr:11      USD  6.380262e+04    0.0001         NaN                   NaN   TC ENERGY CORP
239   NaN                      Cash and Equivalents      USD  2.166579e+07    0.0499         NaN                   NaN
240   123e                                       AUM      NaN  4.338766e+08    0.9999         NaN                   NaN  AUM IND BARC US

编辑：数据框及其原始形式 df1

ISIN                                 Name Currency       Value  % Weight  Asset Type                              Comments/ Assumptions
0   NaN     Transcanada Trust 5.875 08/15/76      USD  7616765.00    0.0176         NaN  https://assets.cohenandsteers.com/assets/conte...
1   NaN      Bp Capital Markets Plc Flt Perp      USD  7348570.50    0.0169         NaN  Holding value for each constituent is derived ...
2   NaN       Transcanada Trust Flt 09/15/79      USD  7341250.00    0.0169         NaN                                                NaN
3   NaN      Bp Capital Markets Plc Flt Perp      USD  6734022.32    0.0155         NaN                                                NaN
4   NaN  Prudential Financial 5.375% 5/15/45      USD  6508290.68    0.0150         NaN                                                NaN
(241, 7)

df2

Short Name          ISIN
0  ABU DHABI COMMER  AEA000201011
1  ABU DHABI NATION  AEA002401015
2  ABU DHABI NATION  AEA006101017
3  ADNOC DRILLING C  AEA007301012
4  ALPHA DHABI HOLD  AEA007601015
(66987, 2)

EDIT 2：从数据帧中获取匹配的模糊逻辑

df1 = pd.read_excel('file.xlsx', sheet_name=1, usecols=[1, 2, 3, 4, 5, 6, 8], header=1)
df2 = pd.read_excel("Excel files/file2.xlsx", sheet_name=0, usecols=[1, 2], header=1)

# empty lists for storing the matches
# later
mat1 = []
mat2 = []
p = []

# converting dataframe column
# to list of elements
# to do fuzzy matching
list1 = df1['Name'].tolist()
list2 = df2['Short Name'].tolist()

# taking the threshold as 80
threshold = 93

# iterating through list1 to extract
# it's closest match from list2
for i in list1:
    mat1.append(process.extractOne(i, list2, scorer=fuzz.token_set_ratio))
df1['matches'] = mat1

# iterating through the closest matches
# to filter out the maximum closest match
for j in df1['matches']:
    if j[1] >= threshold:
        p.append(j[0])
    mat2.append(",".join(p))
    p = []

# storing the resultant matches back
# to df1
df1['matches'] = mat2
print("\nDataFrame after Fuzzy matching using token_set_ratio():")
#print(df1.to_csv('todays-result1.csv'))
print(df1.head(20))

【参考方案1】：

假设您的第一个数据框的 ISIN 填写为空，那么一个简单的 merge 将满足您的需求。如果您需要保留第一个数据帧中的非空 ISIN，则需要使用布尔掩码：-

df1 = pd.DataFrame(
  [[None, "Apple", "appl"], 
  [None, "Google", "ggl"], 
  [None, "Amazon", 'amzn']], 
  columns=["ISIN", "Name", "matches"]
)

df2 = pd.DataFrame(
  [["ISIN1", "appl"], 
  ["ISIN2", "ggl"]], 
  columns= ["ISIN", "Short Name"]
)

missing_isin = df1['ISIN'].isnull()

df1.loc[missing_isin, 'ISIN'] = df1.loc[missing_isin][['matches']].merge(
    df2[['ISIN', 'Short Name']], 
    how='left', 
    left_on='matches', 
    right_on='Short Name'
)['ISIN']

left_on / right_on :- 与数据框匹配的列名

how='left' :- （简单来说）保留最左侧数据帧的顺序/索引，查看docs 了解更多信息

【讨论】：

我正在检查这个解决方案，非常感谢分享，会告诉你结果。

所有代码都不起作用，上面的代码返回 ISIN 但错误。第二个代码抛出错误 'AttributeError: 'Series' object has no attribute 'merge'

您能否分享两个数据框的可重现样本，其中仅包含原始问题中的相关列？我可以尝试重现结果并调整所需的任何内容。我发布的解决方案是对数据框进行假设

请检查我更新的问题

我已经修改了我的答案。顺便说一句，当我说可重现的示例时，我的意思是一段代码，我可以很容易地自己执行。我已经为你添加了这样一个例子。您可以看到更新后的 sn-p 进行了所需的匹配。这不起作用的唯一原因是matches 列与Short Names 列不直接匹配