如何将 requests.models.Response(obj) 转换为有用的东西，如列表，然后转换为 DataFrame？
Posted
技术标签:
【中文标题】如何将 requests.models.Response(obj) 转换为有用的东西，如列表，然后转换为 DataFrame？【英文标题】：how to transform requests.models.Response(obj) into something useful like a list, and after into a DataFrame? 【发布时间】：2021-10-24 20:46:47 【问题描述】：我使用 Postman 在网站上进行了 scrape。之后，在同一个程序上，我使用该工具生成了 sn-p 代码，所以我可以在 jupyterNote 上使用它。
代码：
import requests
url = "https://www.tennet.eu/electricity-market/transparency-pages/transparency-germany/reporting-market-relevance/feed-in-management-according-to-par-14/information-on-feed-in-management-levels/?tx_tennetcw_transparency%5Baction%5D=csvDownload&tx_tennetcw_transparency%5Bcontroller%5D=Transparency&cHash=d3d32d8a6d86f1121d287aa727ff5110"
payload='tx_tennetcw_transparency%5B__referrer%5D%5B%40extension%5D=TennetCw&tx_tennetcw_transparency%5B__referrer%5D%5B%40vendor%5D=Arvato&tx_tennetcw_transparency%5B__referrer%5D%5B%40controller%5D=Transparency&tx_tennetcw_transparency%5B__referrer%5D%5B%40action%5D=list&tx_tennetcw_transparency%5B__referrer%5D%5Barguments%5D=YTowOnt96b882eafe83827345698c053f8021fe553f44326&tx_tennetcw_transparency%5B__referrer%5D%5B%40request%5D=a%3A4%3A%7Bs%3A10%3A%22%40extension%22%3Bs%3A8%3A%22TennetCw%22%3Bs%3A11%3A%22%40controller%22%3Bs%3A12%3A%22Transparency%22%3Bs%3A7%3A%22%40action%22%3Bs%3A4%3A%22list%22%3Bs%3A7%3A%22%40vendor%22%3Bs%3A6%3A%22Arvato%22%3B%7D8f91dd11199b4d706eaa7939dfa04d22d7122893&tx_tennetcw_transparency%5B__trustedProperties%5D=a%3A5%3A%7Bs%3A7%3A%22current%22%3Bi%3A1%3Bs%3A3%3A%22uid%22%3Bi%3A1%3Bs%3A5%3A%22sDate%22%3Bi%3A1%3Bs%3A5%3A%22eDate%22%3Bi%3A1%3Bs%3A5%3A%22email%22%3Bi%3A1%3B%7D172537dd2ccfc75d9cd1f075558d525c95ff58e2&tx_tennetcw_transparency%5Bcurrent%5D=FeedInManagementActivities&tx_tennetcw_transparency%5Buid%5D=55594&tx_tennetcw_transparency%5BsDate%5D=2021-08-01%2000%3A00&tx_tennetcw_transparency%5BeDate%5D=2021-08-24%2023%3A59&type=7788'
headers =
'Content-Type': 'application/x-www-form-urlencoded'
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
输出为：
"Action ID";"feed-in source";"level [%]";Start;"Time of publication"
TOG2021-0126;"Veja Mate AC105";100,0;"2021-08-13 20:11";"2021-08-13 20:20"
TOG2021-0126;"Veja Mate AC105";85,2;"2021-08-13 19:52";"2021-08-13 19:54"
TOG2021-0126;"Veja Mate AC105";75,7;"2021-08-13 16:14";"2021-08-13 16:22"
TOG2021-0126;"Veja Mate AC104";100,0;"2021-08-13 20:11";"2021-08-13 20:20"
TOG2021-0126;"Veja Mate AC104";79,8;"2021-08-13 19:52";"2021-08-13 19:54"
TOG2021-0126;"Veja Mate AC104";71;"2021-08-13 16:14";"2021-08-13 16:22"
我尝试过的：
-
将响应解析为 JSON 对象。
转换成字符串或字符串列表。
对我来说，这看起来像是 word 中的 excel 表格。
我想要的是 5 列标题，以及每个 Action-ID 的连续行。然后，我终于可以把它放到一个DataFrame中了。
在此先感谢您，对任何菜鸟表示抱歉
【问题讨论】：
【参考方案1】：您可以使用io.StringIO + pd.read_csv 将响应读入 panda DataFrame：
import requests
import pandas as pd
from io import StringIO
url = "https://www.tennet.eu/electricity-market/transparency-pages/transparency-germany/reporting-market-relevance/feed-in-management-according-to-par-14/information-on-feed-in-management-levels/?tx_tennetcw_transparency%5Baction%5D=csvDownload&tx_tennetcw_transparency%5Bcontroller%5D=Transparency&cHash=d3d32d8a6d86f1121d287aa727ff5110"
payload = "tx_tennetcw_transparency%5B__referrer%5D%5B%40extension%5D=TennetCw&tx_tennetcw_transparency%5B__referrer%5D%5B%40vendor%5D=Arvato&tx_tennetcw_transparency%5B__referrer%5D%5B%40controller%5D=Transparency&tx_tennetcw_transparency%5B__referrer%5D%5B%40action%5D=list&tx_tennetcw_transparency%5B__referrer%5D%5Barguments%5D=YTowOnt96b882eafe83827345698c053f8021fe553f44326&tx_tennetcw_transparency%5B__referrer%5D%5B%40request%5D=a%3A4%3A%7Bs%3A10%3A%22%40extension%22%3Bs%3A8%3A%22TennetCw%22%3Bs%3A11%3A%22%40controller%22%3Bs%3A12%3A%22Transparency%22%3Bs%3A7%3A%22%40action%22%3Bs%3A4%3A%22list%22%3Bs%3A7%3A%22%40vendor%22%3Bs%3A6%3A%22Arvato%22%3B%7D8f91dd11199b4d706eaa7939dfa04d22d7122893&tx_tennetcw_transparency%5B__trustedProperties%5D=a%3A5%3A%7Bs%3A7%3A%22current%22%3Bi%3A1%3Bs%3A3%3A%22uid%22%3Bi%3A1%3Bs%3A5%3A%22sDate%22%3Bi%3A1%3Bs%3A5%3A%22eDate%22%3Bi%3A1%3Bs%3A5%3A%22email%22%3Bi%3A1%3B%7D172537dd2ccfc75d9cd1f075558d525c95ff58e2&tx_tennetcw_transparency%5Bcurrent%5D=FeedInManagementActivities&tx_tennetcw_transparency%5Buid%5D=55594&tx_tennetcw_transparency%5BsDate%5D=2021-08-01%2000%3A00&tx_tennetcw_transparency%5BeDate%5D=2021-08-24%2023%3A59&type=7788"
headers = "Content-Type": "application/x-www-form-urlencoded"
response = requests.request("POST", url, headers=headers, data=payload)
df = pd.read_csv(StringIO(response.text), sep=";")
print(df)
打印：
Action ID feed-in source level [%] Start Time of publication
0 TOG2021-0126 Veja Mate AC105 100,0 2021-08-13 20:11 2021-08-13 20:20
1 TOG2021-0126 Veja Mate AC105 85,2 2021-08-13 19:52 2021-08-13 19:54
2 TOG2021-0126 Veja Mate AC105 75,7 2021-08-13 16:14 2021-08-13 16:22
3 TOG2021-0126 Veja Mate AC104 100,0 2021-08-13 20:11 2021-08-13 20:20
4 TOG2021-0126 Veja Mate AC104 79,8 2021-08-13 19:52 2021-08-13 19:54
...and so on.
【讨论】：
以上是关于如何将 requests.models.Response(obj) 转换为有用的东西，如列表，然后转换为 DataFrame？的主要内容，如果未能解决你的问题，请参考以下文章