如何在 Python 上使用 selenium webdriver 和 browsermob 代理捕获网络流量？

Posted 2023-02-23

【中文标题】如何在 Python 上使用 selenium webdriver 和 browsermob 代理捕获网络流量？

【英文标题】：How to capture network traffic using selenium webdriver and browsermob proxy on Python?

【发布时间】：2015-03-25 23:50:45

【问题描述】：

我想在 Python 上使用 Selenium Webdriver 来捕获网络流量。因此，我必须使用代理（如 BrowserMobProxy）

当我使用 webdriver.Chrome 时：

from browsermobproxy import Server

server = Server("~/browsermob-proxy")

server.start()
proxy = server.create_proxy()

from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server=host:port'.format(host='localhost', port=proxy.port))

driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)

proxy.new_har
driver.get(url)
proxy.har # returns a HAR 

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

网页已正确加载，所有请求都可以在 HAR 文件中访问。 但是当我使用 webdriver.Firefox 时：

# The same as above
# ...
from selenium import webdriver
profile  = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile, proxy = proxy.selenium_proxy())

proxy.new_har
driver.get(url)
proxy.har # returns a HAR

for ent in proxy.har['log']['entries']:
    print ent['request']['url']

网页无法正常加载，HAR 文件中的请求数小于应有的请求数。

你知道第二个代码中代理设置的问题吗？我应该如何修复它以正确使用 webdriver.Firefox？

【参考方案1】：

偶然发现了这个项目https://github.com/derekargueta/selenium-profiler。吐出 URL 的所有网络数据。应该不难破解并集成到您正在运行的任何测试中。

原文来源：https://www.openhub.net/p/selenium-profiler

【参考方案2】：

对我来说，以下代码组件可以正常工作。

profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

【讨论】：

在使用 https 时这对您有用吗？它不适合我（即使设置profile.accept_untrusted_certs = True）。

【参考方案3】：

I am sharing my solution, this would not write any logs to any file
but you can collect all sort of messages such as Errors, 
Warnings, Logs, Info, Debug , CSS, XHR as well as Requests(traffic)

1. We are going to create Firefox profile so that we can enable option of 
"Persist Logs" on Firefox (you can try it to enable on your default browser and see 
if it launches with "Persist Logs" without creating firefox profile )

2. we need to modify the Firefox initialize code 
where this line will do magic : options.AddArgument("--jsconsole");

so complete Selenium Firefox code would be,  this will open Browser Console
everytime you execute your automation :

 else if (browser.Equals(Constant.Firefox))
            

                var profileManager = new FirefoxProfileManager();
                FirefoxProfile profile = profileManager.GetProfile("ConsoleLogs");
                FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(DrivePath);
                service.FirefoxBinaryPath = DrivePath;                
                profile.SetPreference("security.sandbox.content.level", 5);
                profile.SetPreference("dom.webnotifications.enabled", false);
                profile.AcceptUntrustedCertificates = true;
                FirefoxOptions options = new FirefoxOptions();
                options.AddArgument("--jsconsole");
                options.AcceptInsecureCertificates = true;
                options.Profile = profile;
                options.SetPreference("browser.popups.showPopupBlocker", false);
                driver = new FirefoxDriver(service.FirefoxBinaryPath, options, TimeSpan.FromSeconds(100));
                driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);               

            

3. Now you can write your logic since you have traffic/ logging window open so don't 
go to next execution if test fails. That way Browser Console will keep your errors 
messages and help you to troubleshoot further 

Browser : Firefox  v 61 

How can you launch Browser Console for firefox:
1. open firefox (and give any URL )
2. Press Ctrl+Shift+J (or Cmd+Shift+J on a Mac) 

Link : https://developer.mozilla.org/en-US/docs/Tools/Browser_Console