如何根据字段合并两个 CSV 文件并在每条记录上保持相同数量的属性？

Posted 2023-02-23

【中文标题】如何根据字段合并两个 CSV 文件并在每条记录上保持相同数量的属性？

【英文标题】：How do I merge two CSV files based on field and keep same number of attributes on each record?

【发布时间】：2014-06-14 03:59:55

【问题描述】：

我正在尝试根据每个文件中的特定字段合并两个 CSV 文件。

file1.csv

id,attr1,attr2,attr3
1,True,7,"Purple"
2,False,19.8,"Cucumber"
3,False,-0.5,"A string with a comma, because it has one"
4,True,2,"Nope"
5,True,4.0,"Tuesday"
6,False,1,"Failure"

file2.csv

id,attr4,attr5,attr6
2,"python",500000.12,False
5,"program",3,True
3,"Another string",-5,False

这是我正在使用的代码：

import csv
from collections import OrderedDict

with open('file2.csv','r') as f2:
    reader = csv.reader(f2)
    fields2 = next(reader,None) # Skip headers
    dict2 = row[0]: row[1:] for row in reader

with open('file1.csv','r') as f1:
    reader = csv.reader(f1)
    fields1 = next(reader,None) # Skip headers
    dict1 = OrderedDict((row[0], row[1:]) for row in reader)

result = OrderedDict()
for d in (dict1, dict2):
    for key, value in d.iteritems():
        result.setdefault(key, []).extend(value)

with open('merged.csv', 'wb') as f:
    w = csv.writer(f)
    for key, value in result.iteritems():
        w.writerow([key] + value)

我得到这样的输出，它适当地合并，但所有行的属性数量不同：

1,True,7,Purple
2,False,19.8,Cucumber,python,500000.12,False
3,False,-0.5,"A string with a comma, because it has one",Another string,-5,False
4,True,2,Nope
5,True,4.0,Tuesday,program,3,True
6,False,1,Failure

file2 不会对file1 中的每个id 都有记录。我希望输出在合并文件中有来自file2 的空字段。例如，id 1 看起来像这样：

1,True,7,Purple,,,

如何将空字段添加到file2 中没有数据的记录中，以便合并 CSV 中的所有记录具有相同数量的属性？

【问题讨论】：

您是否还希望标题行是id,attr1,attr2,attr3,attr4,attr5,attr6？

@s16h 是的。我的示例代码中没有包含该代码。不过，我已经在工作了。

【参考方案1】：

您可以使用pandas 来执行此操作：

import pandas

csv1 = pandas.read_csv('filea1.csv')
csv2 = pandas.read_csv('file2.csv')
merged = csv1.merge(csv2, on='id')
merged.to_csv("output.csv", index=False)

我还没有对此进行测试，但它应该会让您走上正轨，直到我可以尝试。代码是不言自明的；首先您导入pandas 库以便您可以使用它。然后使用 pandas.read_csv 读取 2 个 csv 文件并使用 merge 方法合并它们。 on 参数指定应将哪一列用作“键”。最后将合并后的csv写入output.csv。

【参考方案2】：

如果我们不使用pandas，我会重构为类似

import csv
from collections import OrderedDict

filenames = "file1.csv", "file2.csv"
data = OrderedDict()
fieldnames = []
for filename in filenames:
    with open(filename, "rb") as fp: # python 2
        reader = csv.DictReader(fp)
        fieldnames.extend(reader.fieldnames)
        for row in reader:
            data.setdefault(row["id"], ).update(row)

fieldnames = list(OrderedDict.fromkeys(fieldnames))
with open("merged.csv", "wb") as fp:
    writer = csv.writer(fp)
    writer.writerow(fieldnames)
    for row in data.itervalues():
        writer.writerow([row.get(field, '') for field in fieldnames])

给了

id,attr1,attr2,attr3,attr4,attr5,attr6
1,True,7,Purple,,,
2,False,19.8,Cucumber,python,500000.12,False
3,False,-0.5,"A string with a comma, because it has one",Another string,-5,False
4,True,2,Nope,,,
5,True,4.0,Tuesday,program,3,True
6,False,1,Failure,,,

为了比较，pandas 等价物类似于

df1 = pd.read_csv("file1.csv")
df2 = pd.read_csv("file2.csv")
merged = df1.merge(df2, on="id", how="outer").fillna("")
merged.to_csv("merged.csv", index=False)

这在我看来要简单得多，这意味着您可以花更多时间处理数据，而花更少的时间重新发明***。

【参考方案3】：

使用 dict 的 dict 然后更新它。像这样：

import csv
from collections import OrderedDict

with open('file2.csv','r') as f2:
    reader = csv.reader(f2)
    lines2 = list(reader)

with open('file1.csv','r') as f1:
    reader = csv.reader(f1)
    lines1 = list(reader)

dict1 = row[0]: dict(zip(lines1[0][1:], row[1:])) for row in lines1[1:]
dict2 = row[0]: dict(zip(lines2[0][1:], row[1:])) for row in lines2[1:]

#merge
updatedDict = OrderedDict()
mergedAttrs = OrderedDict.fromkeys(lines1[0][1:] + lines2[0][1:], "?")
for id, attrs in dict1.iteritems():
    d = mergedAttrs.copy()
    d.update(attrs)
    updatedDict[id] = d

for id, attrs in dict2.iteritems():
    updatedDict[id].update(attrs)

#out
with open('merged.csv', 'wb') as f:
    w = csv.writer(f)
    for id, rest in sorted(updatedDict.iteritems()):
        w.writerow([id] + rest.values())