将从HTML表格抓取的数据写入CSV文件

Posted 2023-01-14

我正在尝试下一步将Webscrape转换为CSV的步骤。

我已经尝试将每一列放入单独的列表中，但是我觉得这不是解决方案。

from bs4 import BeautifulSoup
import requests

url = 'https://www.pro-football-reference.com/years/2018/passing.htm'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')

for row in tb.find_all('tr'):
    i = row.get_text()
    print(i)

答案

这应该工作

import csv      #quite crucial

final_table = []
for row in tb.findall('tr'):
    next_line = row.get_text()
    final_table.append([next_line])

with open('output.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerows(final_table)

另一答案

使用csv模块。我们将使用soup.find("tr").find_all("th")抓取标题，然后循环遍历正文并将其写入文本文件。每行的第一个单元格是<th>，因此我们需要单独处理该数据并将其添加到<td>数据之前。请注意，每30行交错的标题将被省略。

import csv
import requests
from bs4 import BeautifulSoup

url = "https://www.pro-football-reference.com/years/2018/passing.htm"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

with open("output.csv", "w") as f:
    writer = csv.writer(f)
    writer.writerow([x.get_text() for x in soup.find("tr").find_all("th")])

    for row in soup.find_all("tr"):
        data = [row.find("th").get_text()] + [x.get_text() for x in row.find_all("td")]

        if data: 
            writer.writerow(data)

Run the code

输出（仅前几行）：

Rk,Player,Tm,Age,Pos,G,GS,QBrec,Cmp,Att,Cmp%,Yds,TD,TD%,Int,Int%,Lng,Y/A,AY/A,Y/C,Y/G,Rate,QBR,Sk,Yds,NY/A,ANY/A,Sk%,4QC,GWD
1,Ben Roethlisberger,PIT,36,QB,16,16,9-6-1,452,675,67.0,5129,34,5.0,16,2.4,97,7.6,7.5,11.3,320.6,96.5,71.0,24,166,7.10,7.04,3.4,2,3
2,Andrew Luck*,IND,29,QB,16,16,10-6-0,430,639,67.3,4593,39,6.1,15,2.3,68,7.2,7.4,10.7,287.1,98.7,69.4,18,134,6.79,6.95,2.7,3,3
3,Matt Ryan,ATL,33,QB,16,16,7-9-0,422,608,69.4,4924,35,5.8,7,1.2,75,8.1,8.7,11.7,307.8,108.1,68.5,42,296,7.12,7.71,6.5,1,1
4,Kirk Cousins,MIN,30,QB,16,16,8-7-1,425,606,70.1,4298,30,5.0,10,1.7,75,7.1,7.3,10.1,268.6,99.7,58.2,40,262,6.25,6.48,6.2,1,0

[Check this thread，如果您在Windows的CSV结果中看到额外的换行符。