Python网络异步爬虫（aiohttp库）

Posted 2022-11-14 web安全工具库

0x01 单线程访问网页

import requests
import time
num=20
url = "https://www.dandanzan10.top/dianying/index.html"
ks=time.time()
for _ in range(1,num+1):
    res=requests.get(url)
    print(res.text[:10])
    js = time.time()
print(js-ks)


运行结果： <!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
<!DOCTYPE 
24.051593542099


进程已结束,退出代码0

访问网页20次大概需要24秒左右

0x02 加入协程访问网页

import asyncio
import aiohttp
import time


ks=time.time()


async def get(a,url):
    async with a.get(url) as res:
        return await res.text()


async def main():
    async with aiohttp.ClientSession() as s:
        res = await get(s,"https://www.dandanzan10.top/dianying/index.html")
        print(res[:10])


if __name__=='__main__':


    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    taks=[asyncio.ensure_future(main(),loop=loop) for _ in range(20)]
    loop.run_until_complete(asyncio.wait(taks))
    js = time.time()
    print(js-ks)
    
    运行结果： <!DOCTYPE *20.。。。2.0055174827575684
    

通过协程方法，访问该页面20次，仅用2秒左右

1、定义一个get请求方法，返回获取的字符串

2、async：协程定义方法必须加上这个关键字

3、await：返回的对象都必须加上这个关键字

0x03 声明

仅供安全研究与学习之用，若将工具做其他用途，由使用者承担全部法律及连带责任，作者不承担任何法律及连带责任。

欢迎关注公众号编程者吧