从 AWS 定价中抓取标签表

Posted 2023-02-19

【中文标题】从 AWS 定价中抓取标签表

【英文标题】：Scraping tabbed table from AWS pricing

【发布时间】：2022-01-23 04:55:48

【问题描述】：

我正在尝试构建刮板来刮掉此页面中的表格 (https://aws.amazon.com/sagemaker/pricing/) 我只对 training、processing 和其他一些数据感兴趣。

req = requests.get(url)
soup = bs4.BeautifulSoup(req.content)
tables = soup.find_all("table")

inst_table = str(tables[0])

但看起来我必须使用某种动态机制来获取选项卡式开关。

假设我们点击了训练选项卡，我的目标是构建一个存储抓取数据的文件

  "ml.t2.medium": 
    "vCPU": 2.0,
    "mem_GiB": 4.0,
    "price": 0.15,
    "category": "Standard",
    "task": "training",
  

【问题讨论】：

您希望在哪一步涉及 Selenium？

//li[contains(@class,'lb-tabs-trigger')]//div[contains(text(),'Training')] 适用于单击训练元素。

@DebanjanB 基本上我想从主页开始，然后动态执行其余的抓取，不知道该怎么做。我还没有真正做太多的硒。

@ArundeepChohan 我需要使用硒吗？你能举个例子吗？

@add-semi-colons 我在哪里可以找到这些ml.t2.medium、"vCPU": 2.0、"category": "Standard"、"task": "training" 数据？

【参考方案1】：

好消息是你不需要selenium 的重炮。

与 AWS 一样，您几乎总是可以查询返回所需数据的 API。

这是您需要的以及如何获得它：

import json
import time

import requests

headers = 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0",


endpoint = f"https://b0.p.awsstatic.com/pricing/2.0/meteredUnitMaps/" \
           f"sagemaker/USD/current/sagemaker-instances.json?" \
           f"timestamp=int(time.time())"
response = requests.get(endpoint, headers=headers).json()

for region, region_data in response["regions"].items():
    if region == "EU (Frankfurt)":
        for instance_type, instance_data in region_data.items():
            print(json.dumps(instance_data, indent=2))

EU (Frankfurt) 的示例输出（为简洁而缩短）：

  "rateCode": "X7Z5CZBN2ZY5QED6.JRTCKXETXF.6YS6EN2CT7",
  "price": "6.1120000000",
  "Instance": "ml.g4dn.12xlarge",
  "Clock Speed": "2.5 GHz",
  "Instance Type": "ml.g4dn.12xlarge-AsyncInf",
  "Component": "AsyncInf",
  "VCPU": "48",
  "Memory": "192 GiB"


  "rateCode": "F926HEYB3SV5TQ3Y.JRTCKXETXF.6YS6EN2CT7",
  "price": "6.8000000000",
  "Instance": "ml.g4dn.16xlarge",
  "Clock Speed": "2.5 GHz",
  "Instance Type": "ml.g4dn.16xlarge-AsyncInf",
  "Component": "AsyncInf",
  "VCPU": "64",
  "Memory": "256 GiB"


  "rateCode": "7SMSS7DTJHR8UWN7.JRTCKXETXF.6YS6EN2CT7",
  "price": "1.8810000000",
  "Instance": "ml.g4dn.4xlarge",
  "Clock Speed": "2.5 GHz",
  "Instance Type": "ml.g4dn.4xlarge-AsyncInf",
  "Component": "AsyncInf",
  "VCPU": "16",
  "Memory": "64 GiB"


and much more ...