Python selenium 测试卡在 urlopen 中

Posted 2023-02-23

【中文标题】Python selenium 测试卡在 urlopen 中

【英文标题】：Python selenium test gets stuck in urlopen

【发布时间】：2015-10-11 14:06:42

【问题描述】：

我的应用依赖于：

Python 3 Django 1.8 Weasyprint 硒

它可以在开发和生产环境中完美运行，但在使用 selenium 进行测试时却无法正常运行。

使用 weasyprint，我从 html 创建了一个 PDF，这个库使用 urllib 下载 CSS（例如 http://localhost:8081/static/lib/bootstrap/css/bootstrap.min.css），但是在打开这些时它挂起（没有错误，只是卡住）。

如果我在挂起时直接在浏览器中输入此网址，则会显示 CSS。

使用的命令：

./manage.py test tests.test_account.HomeNewVisitorTest

测试的相关部分：

from selenium import webdriver

class HomeNewVisitorTest(StaticLiveServerTestCase):
    def setUp(self):
        if TEST_ENV_FIREFOX:
            self.driver = webdriver.Firefox()
        else:
            self.driver = webdriver.PhantomJS()
        self.driver.set_window_size(1440, 900)

    def tearDown(self):
        try:
            path = 'worksites/' + self.worksite_name.lower()
            os.rmdir(settings.MEDIA_ROOT + path)
        except FileNotFoundError:
            pass
        super().tearDown()

    def test(self):
        d = self.driver
        d.get(self.get_full_url('home'))
        d.find_element_by_css_selector('.btn-success[type=submit]').click()

在我看来：

    # Generate PDF for contact directory
    template = get_template("pdf/annuaire.html")
    context = "worksite": worksite
    html = template.render(RequestContext(self.request, context))
    base_url = self.request.build_absolute_uri("/")
    pdf = weasyprint.HTML(string=html, base_url=base_url)
    pdf.write_pdf(directory + '/annuaire.pdf')

这里是卡住时的线程转储：

Fatal Python error: Aborted

Thread 0x0000000106f92000 (most recent call first):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374 in readinto
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313 in _read_status
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351 in begin
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171 in getresponse
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1185 in do_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1210 in http_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441 in _call_chain
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 481 in _open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 463 in open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 161 in urlopen
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/urls.py", line 276 in default_url_fetcher
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/urls.py", line 311 in fetch
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/contextlib.py", line 59 in __enter__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 297 in _select_source
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/contextlib.py", line 59 in __enter__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 223 in __init__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/css/__init__.py", line 198 in find_stylesheets
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/css/__init__.py", line 448 in get_all_computed_styles
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/document.py", line 312 in _render
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 132 in render
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/weasyprint/__init__.py", line 164 in write_pdf
  File "/Users/sebcorbin/Sites/planexo/worksite/views.py", line 111 in done
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 357 in render_done
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 730 in render_done
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 300 in post
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 686 in post
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/views/generic/base.py", line 89 in dispatch
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/formtools/wizard/views.py", line 237 in dispatch
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/views/generic/base.py", line 71 in view
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/contrib/auth/decorators.py", line 22 in _wrapped_view
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/handlers/base.py", line 132 in get_response
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 189 in __call__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/testcases.py", line 1099 in __call__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/contrib/staticfiles/handlers.py", line 63 in __call__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/wsgiref/handlers.py", line 137 in run
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/servers/basehttp.py", line 182 in handle
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 673 in __init__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/servers/basehttp.py", line 102 in __init__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 344 in finish_request
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 331 in process_request
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 305 in _handle_request_noblock
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 238 in serve_forever
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/testcases.py", line 1182 in run
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 920 in _bootstrap_inner
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 888 in _bootstrap

Current thread 0x00007fff7996a300 (most recent call first):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374 in readinto
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313 in _read_status
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351 in begin
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171 in getresponse
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1185 in do_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1210 in http_open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441 in _call_chain
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 481 in _open
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 463 in open
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 457 in _request
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 389 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 191 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/webelement.py", line 447 in _execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/selenium/webdriver/remote/webelement.py", line 68 in click
  File "/Users/sebcorbin/Sites/planexo/tests/test_account.py", line 203 in _test_worksite_form
  File "/Users/sebcorbin/Sites/planexo/tests/test_account.py", line 36 in test
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 577 in run
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 625 in __call__
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/testcases.py", line 186 in __call__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/suite.py", line 122 in run
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/suite.py", line 84 in __call__
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/runner.py", line 168 in run
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/runner.py", line 178 in run_suite
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/test/runner.py", line 211 in run_tests
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/commands/test.py", line 90 in handle
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/base.py", line 441 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/commands/test.py", line 74 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/base.py", line 390 in run_from_argv
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/commands/test.py", line 30 in run_from_argv
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/__init__.py", line 330 in execute
  File "/Users/sebcorbin/.virtualenvs/planexo-py3/lib/python3.4/site-packages/django/core/management/__init__.py", line 338 in execute_from_command_line
  File "./manage.py", line 10 in <module>

【参考方案1】：

感谢大家指出问题，我以这篇文章http://nedbatchelder.com/blog/201103/quick_and_dirty_multithreaded_django_dev_server.html 为灵感并修改了我的manage.py 文件：

#!/usr/bin/env python
from http import server
import os
from socketserver import ThreadingMixIn
import sys


def monkey_patch_test_server():
    # This monkey-patches HTTPServer to create a base HTTPServer class that
    # supports multithreading
    originalhttpserver = server.HTTPServer

    class ThreadedHTTPServer(ThreadingMixIn, originalhttpserver):
        def __init__(self, server_address, RequestHandlerClass=None):
            originalhttpserver.__init__(self, server_address,
                                        RequestHandlerClass)

    server.HTTPServer = ThreadedHTTPServer


if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "planexo.settings")

    from django.core.management import execute_from_command_line

    if sys.argv[1] == 'test':
        monkey_patch_test_server()

    execute_from_command_line(sys.argv)

现在可以了！

【讨论】：

干得好！希望 Django 有一天会收到一个合适的开发服务器。

【参考方案2】：

使用 Python faulthandler 模块打印所有挂起线程的回溯。这将查明发生挂起的实际功能。

如果需要，编辑您的问题并添加相关信息。

否则我的猜测是您使用 Django 测试/开发服务器的方式是一次只能处理一个请求。您对开发服务器的初始 HTTP 请求会触发向自身请求数据的请求，这是 Django 开发服务器无法做到的。但这只是猜测。

问题中也缺少相关的源代码。

【讨论】：

我已经用你的建议更新了我的问题，但不知道如何解释它，应该有 3 个线程......

是的，我认为 Django 开发服务器无法为自己服务：***.com/a/10636757/315168 - 我建议您在另一个 localhost 端口中启动另一个开发服务器并将其用作 base_url 的目标

使 base_url 在 settings.py 或 OS 环境变量中可配置。然后启动一个特定的 Django 开发服务器实例python manage.py runserver 127.0.0.0:anotherport 并让它在本地开发运行中为 weasyprint 服务。

【参考方案3】：

我将分享我的案例：

我在使用 ChromeDriver 2.21.371459 和 Google Chrome 48.0.2564.116 的 Django selenium 测试（django==1.7.12 和 selenium==2.53.1）时遇到了这个问题。

我能够隔离问题。在我的情况下，它只发生在我的自定义本地 cdn 域上引用静态文件的页面（HTML <img> 标记中的图像，例如 http://cdn.local.myproject.net/static/myimage.png）。如果我使用相对路径“/static/myimage.png”或本地主机“http://127.0.0.1/static/myimage.png”，则问题不存在，所以我认为这是一个 DNS 问题。

我可以通过使用 chrome 的 --dns-prefetch-disable 选项绕过这个问题。

Python 示例：

from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--dns-prefetch-disable')
driver = Chrome(chrome_options=options)

我不知道这是否是一般情况，但希望它可以帮助你们中的一些人。

【参考方案4】：

使用faulthandler 模块。也可能是 Django，只能接受 1 个请求。