使用python pandas加入多个CSV文件

Posted 2021-03-28

我试图通过使用python pandas从多个csv文件创建一个CSV文件。

accreditation.csv -

"pid","accreditation_body","score"
"25799","TAAC","4.5"
"25796","TAAC","5.6"
"25798","DAAC","5.7"

ref_university -

"id","pid","survery_year","end_year"
"1","25799","2018","2018"
"2","25797","2016","2018"

我想通过阅读table_structure.csv的指令来创建一个新表。我想加入两个表并重写accreditation.csv。 REFERENCES ref_university(id, survey_year)通过匹配ref_university.csv列值与id连接并插入survery_year和pid列值。

table_structure.csv -

table_name,attribute_name,attribute_type,Description
,,,
accreditation,accreditation_body,varchar,
,grading,varchar,
,pid,int4, "REFERENCES ref_university(id, survey_year)"
,score,float8,

修改后的CSV文件应如下所示，

新的accreditation.csv： -

"accreditation_body","grading","pid","id","survery_year","score"
"TAAC","","25799","1","2018","2018","4.5"
"TAAC","","25797","2","2016","2018","5.6"
"DAAC","","25798","","","","5.7"

我可以在熊猫中阅读csv

df = pd.read_csv("accreditation.csv")

但是，建议的方法是读取REFERENCES指令并选择列值。如果没有值，则列应为空。我们不能在熊猫功能中核心pid。我们必须阅读table_structure.csv并匹配，如果有一个参考，然后调用提到的列。它不应该合并，只应添加特定的列。

答案

动态解决方案是可行的，但不是那么容易：

df = pd.read_csv("table_structure.csv")

#remove only NaNs rows
df = df.dropna(how='all')
#repalce NaNs by forward filling
df['table_name'] = df['table_name'].ffill()

#create for each table_name one row
df = (df.dropna(subset=['Description'])
       .join(df.groupby('table_name')['attribute_name'].apply(list)
              .rename('cols'), 'table_name'))

#get name of DataFrame and new columns names
df['df1'] = df['Description'].str.extract('REFERENCES\s*(.*)\s*\(')
df['new_cols'] = df['Description'].str.extract('\(\s*(.*)\s*\)')
df['new_cols'] = df['new_cols'].str.split(', ')
#remove unnecessary columns
df = df.drop(['attribute_type','Description'], axis=1).set_index('table_name')
print (df)
table_name                                                                
accreditation            pid  [accreditation_body, grading, pid, score]   

                          df1           new_cols  
table_name                                        
accreditation  ref_university  [id, survey_year]  

#for select by named create dictioanry of DataFrames
data = {'accreditation' : pd.read_csv("accreditation.csv"), 
        'ref_university': pd.read_csv("ref_university.csv")}

---

#seelct by index
v = df.loc['accreditation']
print (v)
attribute_name                                          pid
cols              [accreditation_body, grading, pid, score]
df1                                          ref_university
new_cols                                  [id, survey_year]
Name: accreditation, dtype: object

按字典和Series v选择

df = pd.merge(data[v.name], 
               data[v['df1']][v['new_cols'] + [v['attribute_name']]], 
               on=v['attribute_name'], 
               how='left')

转换为：

df = pd.merge(data['accreditation'], 
               data['ref_university'][['id', 'survey_year'] + ['pid']], 
               on='pid', 
               how='left')

并返回：

print (df)
     pid accreditation_body  score   id  survey_year
0  25799               TAAC    4.5  1.0       2018.0
1  25796               TAAC    5.6  NaN          NaN
2  25798               DAAC    5.7  NaN          NaN

最后由union和reindex添加新列：

df = df.reindex(columns=df.columns.union(v['cols']))
print (df)
  accreditation_body  grading   id    pid  score  survey_year
0               TAAC      NaN  1.0  25799    4.5       2018.0
1               TAAC      NaN  NaN  25796    5.6          NaN
2               DAAC      NaN  NaN  25798    5.7          NaN