在for循环中将Dataframes附加在一起

Posted 2021-04-10

我觉得这应该很简单，但我对Python仍然有点新意，并且正在努力弄清楚我应该做些什么。我正在抓取历史股票数据，并希望将它们放入一个Excel电子表格中。它目前只写出最后的库存数据。

我知道它实际上是在每次循环时写入数据帧但是我不确定如何修复它以附加数据帧，或者每次到达那个位置时写入excel表的末尾。任何帮助，将不胜感激。

这是我的代码：

import numpy as np
from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

symbols = ['WYNN', 'FL', 'TTWO']
myColumnHeaders = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume']

for c in range(len(symbols)):
    url = 'https://www.nasdaq.com/symbol/'+symbols[c]+'/historical'
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    historicaldata = soup.find('div', {'id': 'quotes_content_left_pnlAJAX'})
    data_rows = historicaldata.findAll('tr')[2:]
    stock_data = [[td.getText().strip() for td in data_rows[a].findAll('td')]
                 for a in range(len(data_rows))]
    df = pd.DataFrame(stock_data, columns=myColumnHeaders)
    df.set_index('Date')

    df['Volume'].str.replace(',','').astype(int)
    for i in range(5):
        if i == 0:
            df[myColumnHeaders[i]] = pd.to_datetime(df[myColumnHeaders[i]], 'coerce')
        else:
            df[myColumnHeaders[i]] = pd.to_numeric(df[myColumnHeaders[i]], errors='coerce')

df.to_excel('stock data.xlsx',index=False) 

答案

Do not use pd.DataFrame.append in a loop

这是低效的，因为它涉及重复复制数据。一个更好的想法是创建一个数据帧列表，然后在循环外的最后一步将它们连接起来。这是一些伪代码：

symbols = ['WYNN', 'FL', 'TTWO']
cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume']

dfs = []  # empty list which will hold your dataframes

for c in range(len(symbols)):
    # some code

    df = pd.DataFrame(stock_data, columns=cols)
    df = df.set_index('Date')

    df['Volume'] = df['Volume'].str.replace(',', '').astype(int)

    df[cols[0]] = pd.to_datetime(df[cols[0]], errors='coerce')
    df[cols[1:5]] = df[cols[1:5]].apply(pd.to_datetime, errors='coerce')

    dfs.append(df)  # append dataframe to list

res = pd.concat(dfs, ignore_index=True)  # concatenate list of dataframes
res.to_excel('stock data.xlsx', index=False)

请注意，您正在执行许多操作，例set_index，好像它们默认就位了。事实并非如此。您应该分配回变量，例如df = df.set_index('Date')。

另一答案

我已更新您的代码，以便在单个DataFrame中获取所有数据。

import numpy as np
from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

symbols = ['WYNN', 'FL', 'TTWO']
myColumnHeaders = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume']

dfs = []

for c in range(len(symbols)):
    url = 'https://www.nasdaq.com/symbol/'+symbols[c]+'/historical'
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    historicaldata = soup.find('div', {'id': 'quotes_content_left_pnlAJAX'})
    data_rows = historicaldata.findAll('tr')[2:]
    stock_data = [[td.getText().strip() for td in data_rows[a].findAll('td')]
                 for a in range(len(data_rows))]
    df = pd.DataFrame(stock_data, columns=myColumnHeaders)
    df.set_index('Date')
    df['Volume'].str.replace(',','').astype(int)
    for i in range(5):
        if i == 0:
            df[myColumnHeaders[i]] = pd.to_datetime(df[myColumnHeaders[i]], 'coerce')
        else:
            df[myColumnHeaders[i]] = pd.to_numeric(df[myColumnHeaders[i]], errors='coerce')
    df.index = [symbols[c]]*len(df)
    dfs.append(df)

df = dfs[0].append(dfs[1]).append(dfs[2]).reset_index()
writer = pd.ExcelWriter('output.xlsx', engine='xlsxwriter')
df.to_excel(writer, sheet_name='data', index=False)
writer.save()

另一答案

我也是熊猫的新手。我认为这是一个很好的问题，我确信有一个简单的方法可以做到，但这就是我提出来的......

import numpy as np
from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

symbols = ['WYNN', 'FL', 'TTWO']

stocklist = []
for c in range(len(symbols)):
    url = 'https://www.nasdaq.com/symbol/'+symbols[c]+'/historical'
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    historicaldata = soup.find('div', {'id': 'quotes_content_left_pnlAJAX'})
    data_rows = historicaldata.findAll('tr')[2:]
    stock_data = [[td.getText().strip() for td in data_rows[a].findAll('td')]for a in range(len(data_rows))]
    stocklist.append(stock_data)

df0 = pd.DataFrame(stocklist[0], columns = ['Date', 'Open '+symbols[0], 'High '+symbols[0], 'Low '+symbols[0], 'Close '+symbols[0], 'Volume '+symbols[0]])
df0.set_index('Date', inplace=True)
df1 = pd.DataFrame(stocklist[1], columns = ['Date', 'Open '+symbols[1], 'High '+symbols[1], 'Low '+symbols[1], 'Close '+symbols[1], 'Volume '+symbols[1]])
df1.set_index('Date', inplace=True)
df2 = pd.DataFrame(stocklist[2], columns = ['Date', 'Open '+symbols[2], 'High '+symbols[2], 'Low '+symbols[2], 'Close '+symbols[2], 'Volume '+symbols[2]])
df2.set_index('Date', inplace=True)
df3 = df0.merge(df1, left_index=True, right_index=True)
df = df3.merge(df2, left_index=True, right_index=True)

VW = df['Volume WYNN'].str.replace(',','').astype(int)
FL = df['Volume FL'].str.replace(',','').astype(int)
TTWO = df['Volume TTWO'].str.replace(',','').astype(int)
df['Volume WYNN'] = VW
df['Volume FL'] = FL
df['Volume TTWO'] = TTWO 

df.to_excel('stock data.xlsx',index=False)