使用 idx 自动增量将数据帧插入到 postgresql sqlalchemy

Posted 2023-05-08

技术标签:

【中文标题】使用 idx 自动增量将数据帧插入到 postgresql sqlalchemy【英文标题】：Insert dataframe into postgresql sqlalchemy with idx autoincrement 【发布时间】：2016-11-10 02:24:05 【问题描述】：

我是 requests.get() 来获取一些 json。之后，我想将数据插入到 postgresql 中。发生了一些非常有趣的事情，如果我使用df.to_sql(index=False)，数据会毫无问题地附加到 postgresql 中，但 postgresql 中的 Id 不会创建自动增量值；该列完全为空。如果我消除df.to_sql() 中的参数，则会收到以下错误...IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint。这是我的代码...

import requests
import pandas as pd
import sqlalchemy

urls = ['https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22DIA%22%20and%20startDate%20%3D%20%222015-01-01%22%20and%20endDate%20%3D%20%222015-12-31%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22DIA%22%20and%20startDate%20%3D%20%222016-01-01%22%20and%20endDate%20%3D%20%222016-11-08%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22SPY%22%20and%20startDate%20%3D%20%222015-01-01%22%20and%20endDate%20%3D%20%222015-12-31%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22SPY%22%20and%20startDate%20%3D%20%222016-01-01%22%20and%20endDate%20%3D%20%222016-11-08%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22IWN%22%20and%20startDate%20%3D%20%222015-01-01%22%20and%20endDate%20%3D%20%222015-12-31%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22IWN%22%20and%20startDate%20%3D%20%222016-01-01%22%20and%20endDate%20%3D%20%222016-11-08%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=']
df_list = []
for url in urls:
    data = requests.get(url)
    data_json = data.json()
    df = pd.DataFrame(data_json['query']['results']['quote'])
    df_list.append(df)


quote_df = pd.concat(df_list)
engine = sqlalchemy.create_engine('postgresql://postgres:wpc,.2016@localhost:5432/stocks')
quote_df.to_sql('quotes', engine, if_exists='append')

我想使用 postgresql 自动增量索引将 df 插入到 postgresql 中。我怎样才能修复我的代码来做到这一点。

问题更新 10NOV2016 1900

我添加以下代码来修复数据框中的索引...

quote_df = pd.concat(df_list)
quote_df.index.name = 'Index'
quote_df = quote_df.reset_index()
quote_df['Index'] = quote_df.index

engine = create_engine('postgresql://postgres:wpc,.2016@localhost:5432/stocks')

quote_df.to_sql('quotes', engine, if_exists = 'append', index=False) engine.dispose()

现在附加到 postgresql 时出现以下错误...

ProgrammingError: (psycopg2.ProgrammingError) column "Index" of relation "quotes" does not exist LINE 1: INSERT INTO quotes ("Index", "Adj_Close", "Close", "Date", "...

该列确实存在于数据库中。

【问题讨论】：

您可以为您的 PostgreSQL 表发布 create table ... 语句吗？我在 pgadmin4 中创建表。列名与json对象相同。 【参考方案1】：

一种方法（在许多方法中）是：

获取最大的Id 并将其存储到一个变量中（我们称之为max_id）：

select max(Id) from quotes;

现在我们可以这样做了：

原始DF：

In [55]: quote_df
Out[55]:
      Adj_Close       Close        Date        High         Low        Open Symbol   Volume
0    170.572764  173.990005  2015-12-31  175.649994  173.970001  175.089996    DIA  5773400
1    172.347213  175.800003  2015-12-30  176.720001  175.619995  176.570007    DIA  2910000
2     173.50403  176.979996  2015-12-29      177.25      176.00  176.190002    DIA  6145700
..          ...         ...         ...         ...         ...         ...    ...      ...
213   88.252244   89.480003  2016-01-06   90.099998   89.080002   89.279999    IWN  1570400
214   89.297697   90.540001  2016-01-05   90.620003       89.75   90.410004    IWN  2053100
215   88.893319   90.129997  2016-01-04   90.730003   89.360001   90.550003    IWN  2540600

[1404 rows x 8 columns]

现在我们可以将索引增加max_id：

In [56]: max_id = 123456    # <-- you don't need this line... 

In [57]: quote_df.index += max_id

并将索引设置为Id 列：

In [58]: quote_df.reset_index().rename(columns='index':'Id')
Out[58]:
          Id   Adj_Close       Close        Date        High         Low        Open Symbol   Volume
0     123456  170.572764  173.990005  2015-12-31  175.649994  173.970001  175.089996    DIA  5773400
1     123457  172.347213  175.800003  2015-12-30  176.720001  175.619995  176.570007    DIA  2910000
2     123458   173.50403  176.979996  2015-12-29      177.25      176.00  176.190002    DIA  6145700
...      ...         ...         ...         ...         ...         ...         ...    ...      ...
1401  123669   88.252244   89.480003  2016-01-06   90.099998   89.080002   89.279999    IWN  1570400
1402  123670   89.297697   90.540001  2016-01-05   90.620003       89.75   90.410004    IWN  2053100
1403  123671   88.893319   90.129997  2016-01-04   90.730003   89.360001   90.550003    IWN  2540600

[1404 rows x 9 columns]

现在应该可以将此 DF 写入 PostgreSQL，指定 (index=False)

【讨论】：

我无法获取max(Id)，因为此时表是空的；没有任何行。这个数据框将是我要在 postgresql 中插入的第一个数据。 @Gilbert，print(max_id) 在你从空表中取出之后是什么意思？ Jupyter 返回一个带有select max(Id) from quotes 的错误。这是完整的代码行还是我必须包含在 sqlalchamy 语句中的 sudo 代码？ @Gilbert，这是一个必须使用 SQLAlchemy 或 Psycopg2 执行和获取的 SQL ... max_id 正在返回一个 sqlalchemy 对象； <sqlalchemy.engine.result.ResultProxy object at #######> 我想是因为表是空的，所以正在返回该对象。【参考方案2】：

回答我自己的问题 11NOV2016 1112

我发现在df.reset_index() 之后，我可以删除熊猫创建的额外列，并且原始索引列保持重置。现在，如果我在没有index=False 的情况下执行代码，sqlalchemy 会将索引插入到 postgres 中。这是解决我的问题的代码...

import requests
import pandas as pd
from sqlalchemy import create_engine

urls = ['https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22DIA%22%20and%20startDate%20%3D%20%222015-01-01%22%20and%20endDate%20%3D%20%222015-12-31%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22DIA%22%20and%20startDate%20%3D%20%222016-01-01%22%20and%20endDate%20%3D%20%222016-11-11%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22SPY%22%20and%20startDate%20%3D%20%222015-01-01%22%20and%20endDate%20%3D%20%222015-12-31%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22SPY%22%20and%20startDate%20%3D%20%222016-01-01%22%20and%20endDate%20%3D%20%222016-11-11%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22IWN%22%20and%20startDate%20%3D%20%222015-01-01%22%20and%20endDate%20%3D%20%222015-12-31%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=',
    'https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22IWN%22%20and%20startDate%20%3D%20%222016-01-01%22%20and%20endDate%20%3D%20%222016-11-11%22&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=']

df_list = []
for url in urls:
    data = requests.get(url)
    data_json = data.json()
    df = pd.DataFrame(data_json['query']['results']['quote'])
    df_list.append(df)

quote_df = pd.concat(df_list)
quote_df = quote_df.reset_index()
quote_df = quote_df.drop('index', 1)


engine = create_engine('postgresql://postgres:wpc,.2016@localhost:5432/stocks')
quote_df.to_sql('quotes', engine, if_exists='append')

【讨论】：

以上是关于使用 idx 自动增量将数据帧插入到 postgresql sqlalchemy的主要内容，如果未能解决你的问题，请参考以下文章

使用 select 和 autoincrement 插入 postgres 表

mysql：插入数据和自动增量

数据帧考拉到增量表：错误：调用 o237.save 时发生错误

使用 PDO/MySQL 将自动增量编号附加到电子邮件中发送和发送电子邮件

将自动增量主键插入现有表