PYODBC 截断 SQL Server FOR JSON 查询的响应

Posted 2023-02-15

【中文标题】PYODBC 截断 SQL Server FOR JSON 查询的响应

【英文标题】：PYODBC truncates the response of a SQL Server FOR JSON query

【发布时间】：2018-09-03 06:57:23

【问题描述】：

我在尝试让 SQL Server JSON 结果集与 PYODBC 很好地配合时遇到问题。我的 Python/Flask 技能不是最好的，所以我不完全确定这是否是我在做的愚蠢的事情，或者驱动程序不适用于 JSON。

失败的进程正在调用存储过程，该过程使用FOR JSON PATH 返回SELECT 语句的结果。该过程本身很好，结果集在 SSMS 中看起来是正确的。

但是，在以下代码块中，将值分配给 search_results 的行会引发错误。

with DB() as cnxn:
    results = cnxn.query('dbo.getLocationByTrain', params)
    search_results = json.loads(results.fetchone()[0]);

错误是：

Traceback (most recent call last):
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\env\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\env\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\env\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\env\lib\site-packages\flask\_compat.py", line 33, in reraise
    raise value
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\env\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\env\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\Mark\source\repos\CommuteTo\CommuteTo\CommuteTo\views.py", line 81, in search
    search_results = json.loads(results.fetchone()[0]);
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\json\decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 2028 (char 2027)

如果我删除 json.loads 而不是 jsonify 结果，错误显然消失了，但我可以看到返回给 Flask 的结果集是不完整的：

with DB() as cnxn:
    results = cnxn.query('dbo.getLocationByTrain', params)
    search_results = results.fetchone()[0];
return jsonify(search_results) 

Flask 中的结果：

"[\"Name\":\"ABBEY WOOD\",\"Latitude\":\"51.490772\",\"Longitude\":\"0.120326\",\"LatestDepartureTime\":\"08:18:00\",\"MinJourneyTime\":14,\"MaxJourneyTime\":17,\"schools\":[\"Name\":\"Abbey Wood Nursery School\",\"Address1\":\"Dahlia Road\",\"PostCode\":\"SE2 0SX\",\"OverallEffectiveNess\":2,\"Name\":\"Bannockburn Primary School\",\"Address1\":\"Plumstead High Street\",\"PostCode\":\"SE18 1HE\",\"OverallEffectiveNess\":1,\"Name\":\"Gallions Mount Primary School\",\"Address1\":\"Purrett Road\",\"PostCode\":\"SE18 1JR\",\"OverallEffectiveNess\":2,\"Name\":\"Boxgrove Primary School\",\"Address1\":\"Boxgrove Road\",\"PostCode\":\"SE2 9JP\",\"OverallEffectiveNess\":2,\"Name\":\"De Lucy Primary School\",\"Address1\":\"Cookhill Road\",\"PostCode\":\"SE2 9PD\",\"OverallEffectiveNess\":2,\"Name\":\"Heronsgate Primary School\",\"Address1\":\"Whinchat Road\",\"PostCode\":\"SE28 0EA\",\"OverallEffectiveNess\":1,\"Name\":\"Linton Mead Primary School\",\"Address1\":\"Central Way\",\"PostCode\":\"SE28 8DT\",\"OverallEffectiveNess\":2,\"Name\":\"Greenslade Primary School\",\"Address1\":\"Erindale\",\"PostCode\":\"SE18 2QQ\",\"OverallEffectiveNess\":2,\"Name\":\"St Patrick's Catholic Primary School\",\"Address1\":\"Griffin Road\",\"PostCode\":\"SE18 7QG\",\"OverallEffectiveNess\":2,\"Name\":\"St Thomas A Becket Roman Catholic Primary School\",\"Address1\":\"Mottisfont Road\",\"PostCode\":\"SE2 9LY\",\"OverallEffectiveNess\":2,\"Name\":\"St Margaret Clitherow Catholic Primary School\",\"Address1\":\"Cole Close\",\"PostCode\":\"SE28 8GB\",\"OverallEffectiveNess\":2,\"Name\":\"Bishop John Robinson Church of England Primary School\",\"Address1\":\"Hoveton Road\",\"PostCode\":\"SE28 8LW\",\"OverallEffectiveNess\":2,\"Name\":\"Plumstead Manor School\",\"Address1\":\"Old Mill Road\",\"PostCode\":\"SE18 1QF\",\"OverallEffectiveNess\":3,\"Name\":\"Hawksmoor School\",\"Address1\":\"Bentham Road\",\"PostCode\":\"SE28 8AS\",\"OverallEffectiveNess\":1,\"Name\":\"Belmont Primary School\",\"Address1\":\"Belmont Road\",\"PostCode\":\"DA8 1LE\",\"OverallEffectiveNess\":2,\"Name\":\"Parkway Primary School\",\"Address1\":\"Alsike Road\",\"PostCode\":\"DA18 4DP\",\"OverallEffectiveNess\":2,\"Name\":\"Belvedere Infant School\",\"Address1\":\"Mitchell"

SQL 服务器结果：

["Name":"ABBEY WOOD","Latitude":"51.490772","Longitude":"0.120326","LatestDepartureTime":"08:18:00","MinJourneyTime":14,"MaxJourneyTime":17,"schools":["Name":"Abbey Wood Nursery School","Address1":"Dahlia Road","PostCode":"SE2 0SX","OverallEffectiveNess":2,"Name":"Bannockburn Primary School","Address1":"Plumstead High Street","PostCode":"SE18 1HE","OverallEffectiveNess":1,"Name":"Gallions Mount Primary School","Address1":"Purrett Road","PostCode":"SE18 1JR","OverallEffectiveNess":2,"Name":"Boxgrove Primary School","Address1":"Boxgrove Road","PostCode":"SE2 9JP","OverallEffectiveNess":2,"Name":"De Lucy Primary School","Address1":"Cookhill Road","PostCode":"SE2 9PD","OverallEffectiveNess":2,"Name":"Heronsgate Primary School","Address1":"Whinchat Road","PostCode":"SE28 0EA","OverallEffectiveNess":1,"Name":"Linton Mead Primary School","Address1":"Central Way","PostCode":"SE28 8DT","OverallEffectiveNess":2,"Name":"Greenslade Primary School","Address1":"Erindale","PostCode":"SE18 2QQ","OverallEffectiveNess":2,"Name":"St Patrick's Catholic Primary School","Address1":"Griffin Road","PostCode":"SE18 7QG","OverallEffectiveNess":2,"Name":"St Thomas A Becket Roman Catholic Primary School","Address1":"Mottisfont Road","PostCode":"SE2 9LY","OverallEffectiveNess":2,"Name":"St Margaret Clitherow Catholic Primary School","Address1":"Cole Close","PostCode":"SE28 8GB","OverallEffectiveNess":2,"Name":"Bishop John Robinson Church of England Primary School","Address1":"Hoveton Road","PostCode":"SE28 8LW","OverallEffectiveNess":2,"Name":"Plumstead Manor School","Address1":"Old Mill Road","PostCode":"SE18 1QF","OverallEffectiveNess":3,"Name":"Hawksmoor School","Address1":"Bentham Road","PostCode":"SE28 8AS","OverallEffectiveNess":1,"Name":"Belmont Primary School","Address1":"Belmont Road","PostCode":"DA8 1LE","OverallEffectiveNess":2,"Name":"Parkway Primary School","Address1":"Alsike Road","PostCode":"DA18 4DP","OverallEffectiveNess":2,"Name":"Belvedere Infant School","Address1":"Mitchell Close","PostCode":"DA17 6AA","OverallEffectiveNess":2,"Name":"Castilion Primary School","Address1":"Copperfield Road","PostCode":"SE28 8QA","OverallEffectiveNess":1,"Name":"St Thomas More Catholic Primary School","Address1":"Sheldon Road","PostCode":"DA7 4PH","OverallEffectiveNess":1,"Name":"St John Fisher Catholic Primary School","Address1":"Kale Road","PostCode":"DA18 4BA","OverallEffectiveNess":2,"Name":"St Paul's Academy","Address1":"Finchale Road","PostCode":"SE2 9PX","OverallEffectiveNess":2,"Name":"Discovery Primary School","Address1":"Battery Road","PostCode":"SE28 0JN","OverallEffectiveNess":2,"Name":"Alexander McLeod Primary School","Address1":"Fuchsia Street","PostCode":"SE2 0QS","OverallEffectiveNess":2,"Name":"Conway Primary School","Address1":"Gallosson Road","PostCode":"SE18 1QY","OverallEffectiveNess":2,"Name":"Waterside School","Address1":"Robert Street","PostCode":"SE18 7NB","OverallEffectiveNess":2,"Name":"Trinity Church of England School, Belvedere","Address1":"Erith Road","PostCode":"DA17 6HT","OverallEffectiveNess":2,"Name":"East Wickham Primary Academy","Address1":"Wickham Street","PostCode":"DA16 3BP","OverallEffectiveNess":2,"Name":"Welling School","Address1":"Elsa Road","PostCode":"DA16 1LB","OverallEffectiveNess":2,"Name":"Willow Bank Primary School","Address1":"Seacourt Road","PostCode":"SE2 9XB","OverallEffectiveNess":2,"Name":"Belvedere Junior School","Address1":"Mitchell Close","PostCode":"DA17 6AA","OverallEffectiveNess":2,"Name":"St Augustine of Canterbury CofE Primary School","Address1":"St Augustine's Road","PostCode":"DA17 5HP","OverallEffectiveNess":2,"Name":"Brampton Primary Academy","Address1":"Brampton Road","PostCode":"DA7 4SL","OverallEffectiveNess":4,"Name":"Woolwich Polytechnic School","Address1":"Hutchins Road","PostCode":"SE28 8AT","OverallEffectiveNess":1,"Name":"Hillsgrove Primary School","Address1":"Sidmouth Road","PostCode":"DA16 1DR","OverallEffectiveNess":3,"Name":"Windrush Primary School","Address1":"2 Bentham Road","PostCode":"SE28 8AR","OverallEffectiveNess":1,"Name":"Timbercroft Primary School","Address1":"Timbercroft Lane","PostCode":"SE18 2SG","OverallEffectiveNess":2,"Name":"Rockliffe Manor Primary School","Address1":"Bassant Road","PostCode":"SE18 2NP","OverallEffectiveNess":2,"Name":"Willow Dene School","Address1":"Swingate Lane","PostCode":"SE18 2JD","OverallEffectiveNess":2,"Name":"South Rise Primary School","Address1":"Brewery Road","PostCode":"SE18 7PX","OverallEffectiveNess":2,"Name":"Bedonwell Infant and Nursery School","Address1":"Bedonwell Road","PostCode":"DA17 5PF","OverallEffectiveNess":1,"Name":"Bedonwell Junior School","Address1":"Bedonwell Road","PostCode":"DA17 5PF","OverallEffectiveNess":2,"Name":"Pathways Short Stay School","Address1":"Pathways Short Stay School","PostCode":"SE2 9TA","OverallEffectiveNess":2,"Name":"Northwood Primary School","Address1":"Northwood Place","PostCode":"DA18 4HN","OverallEffectiveNess":2,"Name":"St Michael's East Wickham Church of England Voluntary Aided Primary School","Address1":"Wrotham Road","PostCode":"DA16 1LS","OverallEffectiveNess":1,"Name":"Jubilee Primary School","Address1":"Crowden Way","PostCode":"SE28 8JB","OverallEffectiveNess":3]]

在某处看起来好像 PYODBC 被截断为大约 2050 个字符？

我想我不是第一个尝试使用FOR JSON 和 PYODBC 的人，那么我做错了什么？

我的连接字符串（如果重要的话）：

cnxn = r'Driver=SQL Server;Server=MARKSLAPTOP\MSSQLSERVER17;Database=MyDBName;Trusted_Connection=yes;'

而cnxn.query 只是调用cursor.execute(sql, params)。

编辑 这个问题听起来类似于this 查询，但我已经在使用标准的 SQL Server 驱动程序了……

【问题讨论】：

嗯，这非常接近在 2048 字节处终止 - 我数了 2035 个字符。您能想到堆栈中可能将类似文本的字段限制为 2k 的任何地方吗？

这是 pyodbc 中某处的限制吗？我似乎找不到任何记录这一点的东西。我在做的没什么花哨的，所以听起来与驱动程序有关

@FlipperPA 添加了一个可能相关问题的编辑。虽然不太合理。

@NielsBerglund JSON 是来自 SELECT 语句的结果集，因此从技术上讲，不必将其分配给输出参数，因为 pyodbc 可以解释结果集（或者它应该解释！）。不过我会试一试的！

@MarkSinkinson - issue you cited in your edit 是 pypyodbc 的问题，而不是 pyodbc。 DRIVER=SQL Server 已经很老了，只支持 SQL Server 2000 功能（TDS 协议版本 7.1），所以实际上使用 Windows 的 SQL Server ODBC 驱动程序的更新版本可能会更好。

【参考方案1】：

多亏了上述 cmets 的组合，我已经成功地完成了这项工作。

如果其他人遇到我的问题，我会总结一下：

根据@GordThompson 的评论，我将驱动程序从旧的SQL Server 更改为ODBC Driver 13 for SQL Server。升级驱动程序版本后，我的参数传递需要进行一些调整，但总的来说，这似乎是一个明智之举

也许更重要的是，@FlipperPA 的评论建议我将 SQL Server 查询中的 JSON 对象 CAST 设置为 VARCHAR(MAX)。

SELECT CAST( (SELECT ..... FOR JSON PATH) AS VARCHAR(MAX))

这返回了被选中的完整 JSON 对象，现在一切正常。

我只能假设 pyodbc 在 SQL Server 2017/Azure 中支持 JSON 存在一些问题？

【讨论】：

嗨，马克，很高兴你让它工作。快速评论 - JSON 不是 SQL Server 中的数据类型。 JSON 基本上是具有某些特定功能的 nvarchar。

我将 pymssql 用于我的 flask-sqlalchemy 连接并且面临同样的问题。 Select cast 拯救了我的一天。

【参考方案2】：

我今天遇到了同样奇怪的问题。 JSON 结果应该是单行单列，在 2033 个字符处被截断。

但是，结果集就在那里，只是分布在许多行中！你可以重构它：

rows = cursor.fetchall()
json_result = ''
json_result = json_result.join([row[0] for row in rows])

或者：

rows = cursor.fetchall()
search_results = json.loads(''.join([row[0] for row in rows]))

在 ODBC Driver 13 for SQL Server 和 SQL Server 驱动程序之间切换没有帮助。我使用的是 Windows 10、Python 3.7、pyodbc 4.0.26 和 SQL Server 2017。

这似乎是一个错误。