wikicfp.xml解析器python 3

Posted 2021-04-06

我试图解析wikicfp.v1.2008.xml和wikicfp.v1.2009.xml和wikicfp.v1.2010.xml。以下链接中的三个可用版本。 https://github.com/creswick/wikicfp-parser/tree/master/data我尝试使用XML.etree.ElementTree和beautifulsoup，但是我遇到了很多编码错误

格式不正确（令牌无效）：第949行，第40列UnicodeDecodeError：'charmap'编解码器无法解码位置63563中的字节0x9d：字符映射到undefined>

由于错误，我无法前进。我的目标是解析每一行并将其保存在SQL脚本文件或CSV文件中以供以后使用。

from bs4 import BeautifulSoup
out_file = open("final.sql","w")
out_file.write("--DROP TABLE event1;
")
out_file.write("CREATE TABLE event1 (eventid int, fullname TEXT, location TEXT, begindate TINYTEXT , finishdate TINYTEXT , weblink TEXT, info TEXT, PRIMARY KEY (eventid));
")
out_file.close()
infile = open("wikicfp.v1.2009.xml",encoding='utf-8-sig')
contents = infile.read()
soup = BeautifulSoup(contents)
rows = soup.find_all('row')
c = 0
for count in rows:
    tempsoup = rows[c]
    try:
        ei = tempsoup.findAll("field", {"name":"eventid"})
        if not ei[0].contents[0].strip():
            ei = "No info"
        eventid = ei[0].contents[0].strip()

    except Exception:
        eventid = 0

    try:
        fn = tempsoup.findAll("field", {"name":"fullname"})
        s = fn[0].contents[0].strip()
        fullname = s.decode('utf-8')
        fullname = fullname.replace("'","_")
    except Exception:
        fullname = "No info"
    try:
        l = tempsoup.findAll("field", {"name":"location"})
        s = l[0].contents[0].strip()
        location = s.decode('utf-8')

        location = location.replace("'","_")

    except Exception:

        location = "No info"

    try:

        bd = tempsoup.findAll("field", {"name":"begindate"})

        s = bd[0].contents[0].strip()

        begindate = s.decode('utf-8')

    except Exception:

        begindate = "No info"

    try:

        fd = tempsoup.findAll("field", {"name":"finishdate"})

        s = fd[0].contents[0].strip()

        finishdate = s.decode('utf-8')

    except Exception:

        finishdate = "No info"

    try:

        wl = tempsoup.findAll("field", {"name":"weblink"})

        s = wl[0].contents[0].strip()

        weblink = s.decode('utf-8')

    except Exception:

        weblink = "No info"

    try:

        i = tempsoup.findAll("field", {"name":"info"})

        s = i[0].contents[0].strip()

        info = s.decode('utf-8')

        info = info.replace("'","_")

    except Exception:

        info = "No info"
    with open("final.sql","a") as out_file:
        out_file.write("INSERT INTO event VALUES (")
        out_file.write(eventid)
        out_file.write(", '")
        out_file.write(fullname)
        out_file.write("', '")
        out_file.write(location)
        out_file.write("','")
        out_file.write(begindate)
        out_file.write("','")
        out_file.write(finishdate)
        out_file.write("','")
        out_file.write(weblink)
        out_file.write("','")
        out_file.write(info)
        out_file.write("');
")
        c=c+1
out_file.close()
infile.close()

从bs4导入BeautifulSoup的另一个尝试开始

with open("wikicfp.v1.2009.xml") as fp:
    soup = BeautifulSoup(fp, 'xml')
rows = soup.find_all('row')

再试一次

import xml.etree.ElementTree as ET
tree = ET.parse('wikicfp.v1.2009.xml')
root = tree.getroot()

答案

看起来您的xml文件包含无效字符。我尝试了几个不同的文本编辑器（Notepad ++，bracket，Notepad，...），所有这些编辑器都遇到了几个位置，它们无法正常编码（例如，在56964行末尾的2008-xml中）。所以xml-parser无法在那里解析xml。您可以使用lxml及其Parsers recover-option忽略这些字符：

import lxml.etree as ET

tree = ET.parse('wikicfp.v1.2008.xml', 
ET.XMLParser(encoding='ISO-8859-1', ns_clean=True, recover=True))
root = tree.getroot()
rows = root.findall('row')
for row in rows:
    fields = row.findall('field')
    for field in fields:
        print(field)

只需在bash中键入pip install lxml即可获得lxml