在python中执行异步并发api请求

Posted 2023-03-30

【中文标题】在python中执行异步并发api请求

【英文标题】：performing async concurrent api requests in python

【发布时间】：2022-01-06 10:07:29

【问题描述】：

我有这个代码：

import requests
import json
import psycopg2
import time
import csv

epics = []
with open('cmd.csv', newline='') as inputfile:
    for row in csv.reader(inputfile):
        epics.append(row[0])

# print(epics)

for epic in epics:
    url = 'https://demo-api.ig.com/gateway/deal/positions/otc'
    params = 
        "epic": epic,
        "expiry": "DFB",
        "direction": "BUY",
        "size": "1",
        "orderType": "MARKET",
        # "timeInForce":'null',
        # "level":'null',
        "guaranteedStop": "false",
        # "stopLevel": 'null',
        # "stopDistance":'null',
        # "trailingStop":'null',
        # "trailingStopIncrement":'null',
        "forceOpen": "true",
        # "limitLevel": 'null',
        # "limitDistance":'null',
        # "quoteId":'null',
        "currencyCode": "GBP"
    

    # Trade submission
    time.sleep(1.5)
    resp = requests.post(url, headers=headers, json=params)
    print(url)
    result = resp.json()
    print(result)

epics 只是一个 csv 值列表，循环遍历并为每个史诗发送一个 post 请求。但是，当我使用请求时，它需要很长时间才能遍历史诗列表。我想每秒发布 100 个并发请求，因为这是来自网站的 SLA。每个请求都应该针对一个独特的史诗。

有人可以提供有关如何执行此操作的指导

【问题讨论】：

Python 默认运行单线程。由于sleep 的时间为 1.5 秒，因此每秒无法执行超过 1 个请求。如果你想要并行，你应该尝试使用threading 或multiprocessing 模块。

谢谢@white，您能否提供一个代码示例来说明如何做到这一点。

【参考方案1】：

你可以试试这样的。

首先将您的代码包装成这样的函数。

def my_function(epic):
    url = 'https://demo-api.ig.com/gateway/deal/positions/otc'
    params = 
        "epic": epic,
        "expiry": "DFB",
        "direction": "BUY",
        "size": "1",
        "orderType": "MARKET",
        # "timeInForce":'null',
        # "level":'null',
        "guaranteedStop": "false",
        # "stopLevel": 'null',
        # "stopDistance":'null',
        # "trailingStop":'null',
        # "trailingStopIncrement":'null',
        "forceOpen": "true",
        # "limitLevel": 'null',
        # "limitDistance":'null',
        # "quoteId":'null',
        "currencyCode": "GBP"
    

    # Trade submission
    time.sleep(1.5)
    resp = requests.post(url, headers=headers, json=params)
    print(url)
    result = resp.json()
    print(result)

接下来你需要将你想要的列表传递给函数。

使用多处理模块和语法将列表提供给函数。这使用了多处理。

import multiprocessing

with multiprocessing.Pool() as pool:
    pool.map(my_function, epics)

您也可以使用多线程进行多处理。

from multiprocessing.pool import Pool, ThreadPool
from multiprocessing import cpu_count

max_requests = 100

# compute number of processes in our pool
# the lesser of length of list to process and the number of cores we have
pool_size = min(max_requests, cpu_count(), len(epics))

# create process pool based on number of cpu cores
process_pool = Pool(pool_size)

# create thread pool
thread_pool = ThreadPool(int(max_requests/pool_size))
worker = partial(my_function, process_pool)

# run the final function
thread_pool.map(worker, epics)