创建单个驱动程序以使用代理遍历多个站点

Posted 2021-04-02

我在python中编写了一个脚本与selenium一起使用代理来解析不同站点的标题。如果我坚持为每个站点创建单独的驱动程序实例，我的脚本就可以做到。但是，我的目标是使用相同的浏览器（使用不同的代理），无论它遍历多少个站点。

如何使用具有不同代理的相同浏览器遍历多个站点来解析它们的标题？

到目前为止我的尝试：

import random
from itertools import cycle
from selenium import webdriver

proxyList = ['103.110.37.244:36022', '180.254.218.229:8080', '110.74.197.207:50632', '1.20.101.95:49001', '200.10.193.90:8080', '173.164.26.117:3128', '103.228.118.66:43002', '178.128.231.201:3128', '1.2.169.54:55312', '181.52.85.249:31487', '97.64.135.4:8080', '190.96.214.123:53251', '52.144.107.142:31923', '45.5.224.145:52035', '89.218.22.178:8080', '192.241.143.186:80', '113.53.29.218:38310', '36.78.131.182:39243']

def get_title(url):
    random.shuffle(proxyList)
    proxy = next(cycle(proxyList))
    print(f'proxy in use {proxy}')
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument(f'--proxy-server={proxy}')
    driver = webdriver.Chrome(options=chrome_options)
    try:
        driver.get(url)
        print(driver.title)
    except Exception:
        if proxy in proxyList:
            proxyList.remove(proxy)
        return get_title(url)

if __name__ == '__main__':
    links = ['http://www.google.com/','https://stackoverflow.com/','https://www.yahoo.com/']
    for link in links:
        get_title(link)

答案

创建一次驱动程序并将其传递给函数，而不是在函数中创建它。这也意味着始终使用相同的代理...

由于代理在chromedriver启动之前作为参数传递，因此如果不创建新驱动程序，则无法更改代理。

import random
from itertools import cycle
from selenium import webdriver

proxyList = ['103.110.37.244:36022', '180.254.218.229:8080', '110.74.197.207:50632', '1.20.101.95:49001', '200.10.193.90:8080', '173.164.26.117:3128', '103.228.118.66:43002', '178.128.231.201:3128', '1.2.169.54:55312', '181.52.85.249:31487', '97.64.135.4:8080', '190.96.214.123:53251', '52.144.107.142:31923', '45.5.224.145:52035', '89.218.22.178:8080', '192.241.143.186:80', '113.53.29.218:38310', '36.78.131.182:39243']

def get_title(url, driver):
    driver.get(url)
    print(driver.title)

if __name__ == '__main__':
    random.shuffle(proxyList)
    proxy = next(cycle(proxyList))
    print(f'proxy in use {proxy}')
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument(f'--proxy-server={proxy}')
    driver = webdriver.Chrome(options=chrome_options)
    links = ['http://www.google.com/','https://stackoverflow.com/','https://www.yahoo.com/']
    for link in links:
        get_title(link, driver)