Pythongevent 中协程池和线程池的简单使用

Posted 2022-05-30 Xavier Jiezou

引言

gevent 不仅实现了协程池（gevent.pool.Pool），同时也实现了线程池（gevent.threadpool.ThreadPool），本文简单介绍它们的用法。

安装

pip install gevent

用法

协程池

• 简单示例（设置最大协程数为 10）

import gevent
from gevent import socket
from gevent.pool import Pool

N = 1000
# limit ourselves to max 10 simultaneous outstanding requests
pool = Pool(10)
finished = 0


def job(url):
    global finished
    try:
        try:
            ip = socket.gethostbyname(url)
            print("url = ip")
        except socket.gaierror as e:
            print(f"url failed with e")
    finally:
        finished += 1


with gevent.Timeout(2, False):
    for x in range(10, 10 + N):
        pool.spawn(job, f"x.com")
    pool.join()

print(f"finished within 2 seconds: finished/N")

• 近似期望输出

10.com failed with [Errno 11001] getaddrinfo failed
11.com failed with [Errno 11001] getaddrinfo failed
12.com = 185.53.177.52
13.com failed with [Errno 11001] getaddrinfo failed
14.com failed with [Errno 11001] getaddrinfo failed
15.com failed with [Errno 11001] getaddrinfo failed
16.com failed with [Errno 11001] getaddrinfo failed
17.com = 125.76.247.133
18.com failed with [Errno 11001] getaddrinfo failed
19.com = 23.99.120.193
20.com = 47.106.214.40
21.com = 104.22.7.153
22.com = 121.36.197.129
23.com = 144.48.124.149
24.com = 104.17.11.52
25.com failed with [Errno 11001] getaddrinfo failed
26.com failed with [Errno 11001] getaddrinfo failed
27.com = 5.22.145.121
28.com = 123.56.153.176
29.com failed with [Errno 11001] getaddrinfo failed
30.com failed with [Errno 11001] getaddrinfo failed
31.com failed with [Errno 11001] getaddrinfo failed
32.com failed with [Errno 11001] getaddrinfo failed
33.com failed with [Errno 11001] getaddrinfo failed
finished within 2 seconds: 24/1000

线程池

• 简单示例（设置最大线程数为 3）

import gevent
from gevent.threadpool import ThreadPool
import time

pool = ThreadPool(3)
start = time.time()
for i in range(4):
    pool.spawn(time.sleep, 1)
gevent.wait()
delay = time.time() - start
print(
    f'Running "time.sleep(1)" 4 times with 3 threads. Should take about 2 seconds: delay:.3fs'
)

• 近似期望输出

Running "time.sleep(1)" 4 times with 3 threads. Should take about 2 seconds: 2.024s

参考

协程池示例：Example dns_mass_resolve.py
线程池示例：Example threadpool.py