wsgi 协议

Posted 2021-02-02 zuanzuan

wsgi 协议

前言

本来没打算这么早就学习 wsgi 的，因为想要学习python 是如何处理网络请求的绕不开 wsgi，所以只好先学习一下 wsgi。先对 wsgi 有个印象，到了学习 Django 运行方式以及如何处理网络请求数据的时候就会感觉很顺畅了。本文参考

什么是 WSGI

wsgi 的全称是Web Server Gateway Interface，这是一个规范，描述了 web server 如何与 web application 交互、web application 如何处理请求。该规范的具体描述在 PEP3333。WSGI 既要实现 web server，也要实现 web application。在 Django 中的 app 其实就是 web application，而 web server其实在使用命令行输入python manage.py runserver或者使用 pycharm 开启 Django 项目的时候就把runserver当做参数传给了 manage.py里面

经过判断然后执行execute_from_command_line(sys.argv)，sys.argv就是 runserver命令，进入该函数，发现执行了utility.execute()函数，进入函数查看源码：

def execute(self):
    """
    Given the command-line arguments, this figures out which subcommand is
    being run, creates a parser appropriate to that command, and runs it.
    """
    try:
        subcommand = self.argv[1]
    except IndexError:
        subcommand = ‘help‘  # Display help if no arguments were given.

    # Preprocess options to extract --settings and --pythonpath.
    # These options could affect the commands that are available, so they
    # must be processed early.
    parser = CommandParser(None, usage="%(prog)s subcommand [options] [args]", add_help=False)
    parser.add_argument(‘--settings‘)
    parser.add_argument(‘--pythonpath‘)
    parser.add_argument(‘args‘, nargs=‘*‘)  # catch-all
    try:
        options, args = parser.parse_known_args(self.argv[2:])
        handle_default_options(options)
    except CommandError:
        pass  # Ignore any option errors at this point.

    try:
        settings.INSTALLED_APPS
    except ImproperlyConfigured as exc:
        self.settings_exception = exc

    if settings.configured:
        # Start the auto-reloading dev server even if the code is broken.
        # The hardcoded condition is a code smell but we can‘t rely on a
        # flag on the command class because we haven‘t located it yet.
        if subcommand == ‘runserver‘ and ‘--noreload‘ not in self.argv:
            try:
                autoreload.check_errors(django.setup)()
            except Exception:
                # The exception will be raised later in the child process
                # started by the autoreloader. Pretend it didn‘t happen by
                # loading an empty list of applications.
                apps.all_models = defaultdict(OrderedDict)
                apps.app_configs = OrderedDict()
                apps.apps_ready = apps.models_ready = apps.ready = True

                # Remove options not compatible with the built-in runserver
                # (e.g. options for the contrib.staticfiles‘ runserver).
                # Changes here require manually testing as described in
                # #27522.
                _parser = self.fetch_command(‘runserver‘).create_parser(‘django‘, ‘runserver‘)
                _options, _args = _parser.parse_known_args(self.argv[2:])
                for _arg in _args:
                    self.argv.remove(_arg)

        # In all other cases, django.setup() is required to succeed.
        else:
            django.setup()

    self.autocomplete()

    if subcommand == ‘help‘:
        if ‘--commands‘ in args:
            sys.stdout.write(self.main_help_text(commands_only=True) + ‘
‘)
        elif len(options.args) < 1:
            sys.stdout.write(self.main_help_text() + ‘
‘)
        else:
            self.fetch_command(options.args[0]).print_help(self.prog_name, options.args[0])
    # Special-cases: We want ‘django-admin --version‘ and
    # ‘django-admin --help‘ to work, for backwards compatibility.
    elif subcommand == ‘version‘ or self.argv[1:] == [‘--version‘]:
        sys.stdout.write(django.get_version() + ‘
‘)
    elif self.argv[1:] in ([‘--help‘], [‘-h‘]):
        sys.stdout.write(self.main_help_text() + ‘
‘)
    else:
        self.fetch_command(subcommand).run_from_argv(self.argv)

源码太长了。。。我把关键地方抠出来：

if settings.configured:
    # Start the auto-reloading dev server even if the code is broken.
    # The hardcoded condition is a code smell but we can‘t rely on a
    # flag on the command class because we haven‘t located it yet.
    if subcommand == ‘runserver‘ and ‘--noreload‘ not in self.argv:
        try:
            autoreload.check_errors(django.setup)()
        except Exception:
            # The exception will be raised later in the child process
            # started by the autoreloader. Pretend it didn‘t happen by
            # loading an empty list of applications.
            apps.all_models = defaultdict(OrderedDict)
            apps.app_configs = OrderedDict()
            apps.apps_ready = apps.models_ready = apps.ready = True

            # Remove options not compatible with the built-in runserver
            # (e.g. options for the contrib.staticfiles‘ runserver).
            # Changes here require manually testing as described in
            # #27522.
            _parser = self.fetch_command(‘runserver‘).create_parser(‘django‘, ‘runserver‘)
            _options, _args = _parser.parse_known_args(self.argv[2:])
            for _arg in _args:
                self.argv.remove(_arg)

    # In all other cases, django.setup() is required to succeed.
    else:
        django.setup()

这里也是注释最多的地方，可以看到有runserver这条命令，然后这里面在经过一系列的判断最后要执行最后一行代码：

self.fetch_command(subcommand).run_from_argv(self.argv)

这行代码等学习 Django 处理流程的时候在详细解释，反正只要知道目前经过这个函数的执行，Django 的 web server 成功运行了。

实现了 WSGI 的模块/库有 wsgiref（python 内置，下面也是用这个来举例）、werkzeug.serving、twisted.web等。

当前运行在 wsgi 之上的 web 框架有 Bottle、Flask、Django 等。WSGI server 所做的工作仅仅是将客户端收到的请求传递给 WSGI application，然后将 WSGI application 的返回值作为相应传给客户端。WSGI application 可以是栈式的，这个栈的中间部分叫做中间件，两端是必须要实现的 application 和 server。所以对客户端来说，中间件扮演服务器；对服务器来说，中间件扮演客户端。在 Django 中wsgi 收到的数据用 request对象表示，要传给客户端的数据用 Httpresponse对象表示。

搭建一个 wsgi 服务

在上章节说了 python 有个内置的 WSGI 库叫 wsgiref。

首先看下项目结构：

# templates为模板（html）文件夹
# start.py 为项目入口，
# urls.py 为路由配置
# views.py 为具体处理路由逻辑代码

start 文件

# start.py文件
from wsgiref.simple_server import make_server
from urls import urls

def app(env, response):
    # 在这里，
    print(env)
    route = env[‘PATH_INFO‘]
    print(response)
    
    # 设置状态码与响应头
    response(‘200 OK‘, [(‘Content-type‘, ‘text/html‘)])
    
    # 设置错误处理
    data = urls[‘/error‘]()
    
    # 设置路由处理
    if route in urls:
        data = urls[route]()
        
    # 返回二进制响应体
    return [data]

if __name__ == ‘__main__‘:
    
    # 创建服务器对象
    server = make_server(‘‘, 8808, app)
    print(‘服务：http://localhost:8808‘)
    
    # 服务保持运行状态
    server.serve_forever()
    
    # WSGI server 是一个 web server，其处理一个 HTTP 请求的逻辑如下：
    # iterable = app(env, response)
    #     for date in iterable:
    #         send data to client

其实这个模块底层使用了 sockserver 模块，我前面的博客也有介绍。经过 make_server就成功开启了wsgi server，然后server_forever()是为了将服务器持续接收客户端请求，采用的是轮询方法，该方法里面的参数 poll_interval=0.5，采用的是0.5秒轮询一次，轮询采用的是 selector学名叫多路复用技术。

urls 文件

# urls.py文件
from views import *
urls = {
    ‘/index‘: index, # 函数地址
    ‘/error‘: error
}

该文件就是处理路由的，然后将对应的路由映射到相应的逻辑处理函数。

views 文件

# 处理请求的功能函数(处理结果返回的都是页面 => 功能函数)
# 利用 jinja2来渲染模板，将后台数据传给前台

from jinjia2 import Template

# 处理主页请求
def index():
    with open(‘templates/index.html‘, ‘r‘) as f:
        dt = f.read()
    tem = Template(dt)
    
    # 将后台数据通过模板渲染功能渲染传给前台页面
    resp = tem.render(name=‘主页‘)
    return resp.encode(‘utf-8‘)

# 处理图标请求
def ico():
    with open(‘favicon.ico‘, ‘rb‘) as f:
        dt = f.read()
    return dt

# 处理错误请求
def error():
    return b‘404‘

templates

该文件夹里面放的伪要返回给前端相关资源，比如index.html

测试

• index 测试

• error 测试

WSGI application接口

在上面wsgi 服务中的 app 就是 wsgi 中的 application，该接口应该实现为一个可调用对象，例如函数、方法、类、含__call__方法的实例。这个可调用对象可以接收两个参数：

• 一个字典，该字典可以包含了客户端请求的信息以及其他信息，可以认为是请求上下文，一般叫做 environment（在这里我取名为 env）；
• 一个用于发送 HTTP 状态码与响应头的回调函数。（具体怎么回调的还不清楚）

同时，可调用对象的返回值是响应体（response body），响应正文是可迭代的、并包含了多个字符串。（加了中括号可以减少迭代次数，提高效率）

把上面的 app 代码拷下来：

def app(env, response):
    # 在这里，
    print(env)
    route = env[‘PATH_INFO‘]
    print(response)
    
    # 设置状态码与响应头
    response(‘200 OK‘, [(‘Content-type‘, ‘text/html‘)])
    
    # 设置错误处理
    data = urls[‘/error‘]()
    
    # 设置路由处理
    if route in urls:
        data = urls[route]()
        
    # 返回二进制响应体
    return [data]

当我对服务端发起请求时，会打印出 env，如下：

{‘PATH‘: ‘/Users/jingxing/Virtualenv/py3-env1/bin:/Users/jingxing/.nvm/versions/node/v4.9.1/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/python_study/mongodb/bin://Volumes/python_study/mongodb/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/python_study/Applications/mongodb-osx-x86_64-3.6.3/bin::/usr/local/mysql/bin‘, ‘PS1‘: ‘(py3-env1) ‘, ‘VERSIONER_PYTHON_VERSION‘: ‘2.7‘, ‘LS_OPTIONS‘: ‘--color=auto‘, ‘LOGNAME‘: ‘jingxing‘, ‘XPC_SERVICE_NAME‘: ‘com.jetbrains.pycharm.23248‘, ‘PWD‘: ‘/Users/jingxing/django_project/day01‘, ‘PYCHARM_HOSTED‘: ‘1‘, ‘NODE_PATH‘: ‘/Users/jingxing/.nvm/versions/node/v4.9.1/lib/node_modules‘, ‘PYCHARM_MATPLOTLIB_PORT‘: ‘62845‘, ‘PYTHONPATH‘: ‘/Users/jingxing/django_project/day01:/Users/jingxing/django_project/day04:/Users/jingxing/django_project/day02:/Users/jingxing/PycharmProjects/youku/youkusecond:/Users/jingxing/django_project/day03:/Applications/PyCharm.app/Contents/helpers/pycharm_matplotlib_backend‘, ‘NVM_CD_FLAGS‘: ‘‘, ‘NVM_DIR‘: ‘/Users/jingxing/.nvm‘, ‘SHELL‘: ‘/bin/bash‘, ‘LSCOLORS‘: ‘CxfxcxdxbxegedabagGxGx‘, ‘PYTHONIOENCODING‘: ‘UTF-8‘, ‘VERSIONER_PYTHON_PREFER_32_BIT‘: ‘no‘, ‘USER‘: ‘jingxing‘, ‘CLICOLOR‘: ‘Yes‘, ‘TMPDIR‘: ‘/var/folders/yl/3drd7wf93f90sfkgpc2zg9cr0000gn/T/‘, ‘SSH_AUTH_SOCK‘: ‘/private/tmp/com.apple.launchd.ujA3r16JUC/Listeners‘, ‘VIRTUAL_ENV‘: ‘/Users/jingxing/Virtualenv/py3-env1‘, ‘XPC_FLAGS‘: ‘0x0‘, ‘PYTHONUNBUFFERED‘: ‘1‘, ‘__CF_USER_TEXT_ENCODING‘: ‘0x1F5:0x0:0x0‘, ‘Apple_PubSub_Socket_Render‘: ‘/private/tmp/com.apple.launchd.gOrXw3Il2u/Render‘, ‘LC_CTYPE‘: ‘en_US.UTF-8‘, ‘NVM_BIN‘: ‘/Users/jingxing/.nvm/versions/node/v4.9.1/bin‘, ‘HOME‘: ‘/Users/jingxing‘, ‘SERVER_NAME‘: ‘jingxingdeMacBook-Pro.local‘, ‘GATEWAY_INTERFACE‘: ‘CGI/1.1‘, ‘SERVER_PORT‘: ‘8808‘, ‘REMOTE_HOST‘: ‘‘, ‘CONTENT_LENGTH‘: ‘‘, ‘SCRIPT_NAME‘: ‘‘, ‘SERVER_PROTOCOL‘: ‘HTTP/1.1‘, ‘SERVER_SOFTWARE‘: ‘WSGIServer/0.2‘, ‘REQUEST_METHOD‘: ‘GET‘, ‘PATH_INFO‘: ‘/‘, ‘QUERY_STRING‘: ‘‘, ‘REMOTE_ADDR‘: ‘127.0.0.1‘, ‘CONTENT_TYPE‘: ‘text/plain‘, ‘HTTP_HOST‘: ‘127.0.0.1:8808‘, ‘HTTP_CONNECTION‘: ‘keep-alive‘, ‘HTTP_UPGRADE_INSECURE_REQUESTS‘: ‘1‘, ‘HTTP_USER_AGENT‘: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36‘, ‘HTTP_ACCEPT‘: ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8‘, ‘HTTP_ACCEPT_ENCODING‘: ‘gzip, deflate, br‘, ‘HTTP_ACCEPT_LANGUAGE‘: ‘en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7‘, ‘HTTP_COOKIE‘: ‘csrftoken=YjPgsyb6TW4fen2fxjy6DHzZYFlBU4SsAuE9AVqWRjLIhymeAlukqjVBpL7KTPPH‘, ‘wsgi.input‘: <_io.BufferedReader name=7>, ‘wsgi.errors‘: <_io.TextIOWrapper name=‘<stderr>‘ mode=‘w‘ encoding=‘UTF-8‘>, ‘wsgi.version‘: (1, 0), ‘wsgi.run_once‘: False, ‘wsgi.url_scheme‘: ‘http‘, ‘wsgi.multithread‘: True, ‘wsgi.multiprocess‘: False, ‘wsgi.file_wrapper‘: <class ‘wsgiref.util.FileWrapper‘>}

这些参数值得关注的为：

• PATH_INFO：路由信息；
• SERVER_PORT：端口；
• HTTP_HOST：ip；
• SERVER_PROTOCOL：服务器端通信协议

可迭代响应

在 app 中向客户端返回数据时，写的为

return [data]，如果改为return date，这将会导致 WSGI 程序的响应变慢。原因是字符串date也是可迭代的，它的每一次迭代只能得到 1bytes 的数据量，这也意味着每一次只向客户端发送1bytes 的数据，直到发送完毕为止。所以推荐使用return [data]。这里的数据是怎么返回的目前还不清楚，保留疑问。。。

如果可迭代响应含有多个字符串，那么Content-Length应该是这些字符串长度之和。

解析 GET 请求

运行 start.py文件，在浏览器中访问http://localhost:8808/?id=1&name=musibii，可以在响应内容中找到到：

‘QUERY_STRING‘: ‘id=1&name=musibii‘
‘REQUEST_METHOD‘: ‘GET‘

cgi.parse_qs()函数可以很方便的处理 QUERY_STRING，同时需要cgi.escape()处理特殊字符以防止脚本注入，如下：

from cgi import parse_qs, escape

QUERY_STRING = ‘id=1&name=musibii‘
d = parse_qs(QUERY_STRING)
print(d.get(‘id‘, [‘‘])[0]) # [‘‘]是默认值，如果在QUERY_STRING中没找到则返回默认值
print(d.get(‘name‘,[]))

print(escape(‘<script>alert(123);</script>‘))

运行结果：

1
[‘musibii‘]
<script>alert(123);</script>

处理 GET 请求的动态网页

from wsgiref.simple_server import make_server
from cgi import parse_qs, escape

# html中 form 的 method 默认为 get，action 是当前页面
html = ‘‘‘
<html>
<body>
   <form method="get" action="">
        <p>
           Age: <input type="text" name="age" value="%(age)s">
        </p>
        <p>
            Hobbies:
            <input
                name="hobbies" type="checkbox" value="software"
                %(checked-software)s
            > Software
            <input
                name="hobbies" type="checkbox" value="tunning"
                %(checked-tunning)s
            > Auto Tunning
        </p>
        <p>
            <input type="submit" value="Submit">
        </p>
    </form>
    <p>
        Age: %(age)s<br>
        Hobbies: %(hobbies)s
    </p>
</body>
</html>
‘‘‘

def app(env, response):
    
    # 解析QUERY_STRING
    d = parse_qs(env[‘QUERY_STRING‘])
    
    age = d.get(‘age‘, [‘‘])[0] # 返回 age 对应的值
    hobbies = d.get(‘hobbies‘, []) # 以 list 形式返回所有的 hobbies
    
    # 防止脚本注入
    age = escape(age)
    hobbies = [escape(hobby) for hobby in hobbies]
    
    response_body = html% {
        ‘checked-software‘: (‘‘, ‘checket‘)[‘software‘ in hobbies],
        ‘checked-tunning‘: (‘‘, ‘checked‘)[‘tunning‘ in hobbies],
        ‘age‘: age or ‘Empty‘,
        ‘hobbies‘: ‘,‘.join(hobbies or [‘No Hobbies?‘])
    }
    
    status = ‘200 OK‘
    
    response_body = [
        (‘Content-Type‘, ‘text/html‘),
        (‘Content-Length‘, str(len(response_body)))
    ]
    
    start_response(status, response_headers)
    return [response_body]

httpd = make_server(‘‘, 8088, app)

httpd.serve_forever()

处理 POST 请求的动态网页

对于POST 请求，查询字符串是放在 HTTP 请求正文（request body）末尾的，不是显式在 url 中。请求正文在 env 字典变量中键为wsgi.input对应的值中，这是一个类似 file 的变量：

‘wsgi.input‘: <_io.BufferedReader name=7>

我看源码看晕了还是没找到这个 name 具体是什么意思，经过 google 猜测这个应该是个标识符。

from wsgiref.simple_server import make_server
from cgi import parse_qs, escape

# html中form的method是post
html = """
<html>
<body>
   <form method="post" action="">
        <p>
           Age: <input type="text" name="age" value="%(age)s">
        </p>
        <p>
            Hobbies:
            <input
                name="hobbies" type="checkbox" value="software"
                %(checked-software)s
            > Software
            <input
                name="hobbies" type="checkbox" value="tunning"
                %(checked-tunning)s
            > Auto Tunning
        </p>
        <p>
            <input type="submit" value="Submit">
        </p>
    </form>
    <p>
        Age: %(age)s<br>
        Hobbies: %(hobbies)s
    </p>
</body>
</html>
"""

def application(environ, start_response):

    # CONTENT_LENGTH 可能为空，或者没有
    try:
        request_body_size = int(environ.get(‘CONTENT_LENGTH‘, 0))
    except (ValueError):
        request_body_size = 0

    request_body = environ[‘wsgi.input‘].read(request_body_size)
    d = parse_qs(request_body)

    # 获取数据
    age = d.get(‘age‘, [‘‘])[0] 
    hobbies = d.get(‘hobbies‘, []) 

    # 转义，防止脚本注入
    age = escape(age)
    hobbies = [escape(hobby) for hobby in hobbies]

    response_body = html % { 
        ‘checked-software‘: (‘‘, ‘checked‘)[‘software‘ in hobbies],
        ‘checked-tunning‘: (‘‘, ‘checked‘)[‘tunning‘ in hobbies],
        ‘age‘: age or ‘Empty‘,
        ‘hobbies‘: ‘, ‘.join(hobbies or [‘No Hobbies?‘])
    }

    status = ‘200 OK‘

    response_headers = [
        (‘Content-Type‘, ‘text/html‘),
        (‘Content-Length‘, str(len(response_body)))
    ]

    start_response(status, response_headers)
    return [response_body]

httpd = make_server(‘localhost‘, 8051, application)

httpd.serve_forever()

中间件

中间件位于 WSGI server 和 WSGI application 之间。所以对客户端来说，中间件扮演服务器；对服务器来说，中间件扮演客户端。在 Django 中wsgi 收到的数据用 request对象表示，要传给客户端的数据用 Httpresponse对象表示。

示例：

from wsgiref.simple_server import make_server

def application(environ, start_response):

    response_body = ‘hello world!‘

    status = ‘200 OK‘

    response_headers = [
        (‘Content-Type‘, ‘text/plain‘),
        (‘Content-Length‘, str(len(response_body)))
    ]

    start_response(status, response_headers)
    return [response_body]

# 中间件
class Upperware:
   def __init__(self, app):
      self.wrapped_app = app

   def __call__(self, environ, start_response):
      for data in self.wrapped_app(environ, start_response):
        yield data.upper()

wrapped_app = Upperware(application)

httpd = make_server(‘localhost‘, 8051, wrapped_app)

httpd.serve_forever()