使用 nginx+gunicorn+django 的间歇性 502s
Posted
技术标签:
【中文标题】使用 nginx+gunicorn+django 的间歇性 502s【英文标题】：Intermittent 502s with nginx+gunicorn+django 【发布时间】：2012-08-20 06:49:45 【问题描述】：在过去的几周里，我们收到了越来越多的 502 错误。目前我们的堆栈是 nginx + gunicron + django 在一个 m1.large EC2 实例上，由一个小型 RDS 实例支持。
随着请求负载的增加，它们似乎变得更加频繁。我会在使用浏览器时看到随机的 502，但是我们命中 api (Tasty Pie) 的命令行脚本通常会在第二次或第三次请求时失败。但是，如果我在脚本发出请求之前将睡眠功能添加到脚本中，那么该请求就可以了，但接下来会出现 502。请注意，我们将摘要身份验证与请求库和睡眠包装器一起使用——因此是 401、200 模式。
为了使调试更加棘手，当 Gunicorn 使用 --debug 选项运行时，问题会自行解决。如果我删除 --debug 选项但将我的 Gunicorn 工作人员明确限制为 1，则错误仍然存在。
我的 nginx.conf：
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
events
worker_connections 768;
# multi_accept on;
http
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
gzip_proxied any;
gzip_types application/x-ghi-packedschemafeatures-v1
gzip_http_version 1.1;
gzip_comp_level 1;
gzip_min_length 500;
proxy_buffering on;
proxy_http_version 1.1;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
虚拟主机文件：
server
listen 80;
server_name pipeline.ourdomain.com;
location /
rewrite ^ https://$server_name$request_uri permanent;
server
listen 443;
server_name pipeline.ourdomain.com;
ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
ssl_session_cache shared:SSL:10m;
ssl_certificate /etc/ssl/certs/ourdomain.com.combined.crt;
ssl_certificate_key /etc/ssl/private/ourdomain.com.key;
root /var/www/;
location /static/
alias /var/www/production/pipeline/public/;
location /
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-Protocol https;
proxy_connect_timeout 240;
proxy_read_timeout 280;
proxy_pass http://localhost:8000/;
error_page 500 502 503 504 /static/50x.html;
独角兽指挥部
#!/bin/bash
set -e
LOGFILE=/var/log/gunicorn/ea_pipeline.log
LOGDIR=$(dirname $LOGFILE)
SETTINGS=production_settings
# user/group to run as
USER=ubuntu
GROUP=ubuntu
DJANGO_PATH=$(dirname $(readlink -f $0))/../
cd $DJANGO_PATH
echo $(pwd)
. ../env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec ../env/bin/gunicorn_django \
--user=$USER --group=$GROUP --log-level=debug \
--preload \
--workers=4 \
--timeout=90 \
--settings=$SETTINGS \
--limit-request-line=8190 \
--limit-request-field_size 0 \
--pythonpath=$DJANGO_PATH \
--log-file=$LOGFILE production_settings.py 2>>$LOGFILE
访问日志示例：
67.134.170.194 - - [24/Aug/2012:00:28:17 +0000] "GET /api/v1/storage/ HTTP/1.1" 401 5 "-" "python-requests/0.13.8 CPython/2.7.3 Linux/3.2.0-29-generic"
67.134.170.194 - - [24/Aug/2012:00:28:18 +0000] "GET /api/v1/storage/ HTTP/1.1" 200 326 "-" "python-requests/0.13.8 CPython/2.7.3 Linux/3.2.0-29-generic"
67.134.170.194 - - [24/Aug/2012:00:28:18 +0000] "GET /api/v1/customer/?client_id=lamb_01 HTTP/1.1" 502 18 "-" "python-requests/0.13.8 CPython/2.7.3 Linux/3.2.0-29-generic"
67.134.170.194 - - [24/Aug/2012:00:29:41 +0000] "GET /api/v1/storage/ HTTP/1.1" 502 18 "-" "python-requests/0.13.8 CPython/2.7.3 Linux/3.2.0-29-generic"
Nginx 错误日志：
2012/08/24 00:28:18 [error] 16490#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 67.134.170.194, server: pipeline.ourdomain.com, request: "GET /api/v1/customer/?client_id=lamb_01 HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v1/customer/?client_id=lamb_01", host: "pipeline.ourdomain.com"
2012/08/24 00:29:41 [error] 16490#0: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 67.134.170.194, server: pipeline.ourdomain.com, request: "GET /api/v1/storage/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v1/storage/", host: "pipeline.ourdomain.com"
Gunicorn 日志样本：
2012-08-24 17:03:13 [8716] [INFO] Starting gunicorn 0.14.3
2012-08-24 17:03:13 [8716] [DEBUG] Arbiter booted
2012-08-24 17:03:13 [8716] [INFO] Listening at: http://127.0.0.1:8000 (8716)
2012-08-24 17:03:13 [8716] [INFO] Using worker: sync
2012-08-24 17:03:13 [8735] [INFO] Booting worker with pid: 8735
2012-08-24 17:03:13 [8736] [INFO] Booting worker with pid: 8736
2012-08-24 17:03:13 [8737] [INFO] Booting worker with pid: 8737
2012-08-24 17:03:13 [8738] [INFO] Booting worker with pid: 8738
2012-08-24 17:03:21 [8738] [DEBUG] GET /api/v1/storage/
Assertion failed: ok (mailbox.cpp:84)
2012-08-24 17:03:21 [8738] [INFO] Parent changed, shutting down: <Worker 8738>
2012-08-24 17:03:21 [8738] [INFO] Worker exiting (pid: 8738)
Error in sys.exitfunc:
2012-08-24 17:03:21 [8737] [DEBUG] GET /api/v1/storage/
2012-08-24 17:03:22 [8838] [INFO] Starting gunicorn 0.14.3
2012-08-24 17:03:22 [8838] [ERROR] Connection in use: ('127.0.0.1', 8000)
2012-08-24 17:03:22 [8838] [ERROR] Retrying in 1 second.
2012-08-24 17:03:22 [8737] [INFO] Parent changed, shutting down: <Worker 8737>
2012-08-24 17:03:22 [8737] [INFO] Worker exiting (pid: 8737)
Error in sys.exitfunc:
2012-08-24 17:03:22 [8736] [DEBUG] GET /api/v1/customer/
2012-08-24 17:03:23 [8736] [INFO] Parent changed, shutting down: <Worker 8736>
2012-08-24 17:03:23 [8736] [INFO] Worker exiting (pid: 8736)
Error in sys.exitfunc:
2012-08-24 17:03:23 [8838] [ERROR] Connection in use: ('127.0.0.1', 8000)
2012-08-24 17:03:23 [8838] [ERROR] Retrying in 1 second.
2012-08-24 17:03:24 [8735] [DEBUG] GET /api/v1/upload_action/
2012-08-24 17:03:24 [8838] [ERROR] Connection in use: ('127.0.0.1', 8000)
2012-08-24 17:03:24 [8838] [ERROR] Retrying in 1 second.
2012-08-24 17:03:24 [8735] [INFO] Parent changed, shutting down: <Worker 8735>
2012-08-24 17:03:24 [8735] [INFO] Worker exiting (pid: 8735)
Error in sys.exitfunc:
2012-08-24 17:03:25 [8838] [DEBUG] Arbiter booted
2012-08-24 17:03:25 [8838] [INFO] Listening at: http://127.0.0.1:8000 (8838)
2012-08-24 17:03:25 [8838] [INFO] Using worker: sync
2012-08-24 17:03:25 [8907] [INFO] Booting worker with pid: 8907
2012-08-24 17:03:25 [8908] [INFO] Booting worker with pid: 8908
2012-08-24 17:03:25 [8909] [INFO] Booting worker with pid: 8909
2012-08-24 17:03:25 [8910] [INFO] Booting worker with pid: 8910
【问题讨论】：
@VBart Nginx 正在记录到 access.log 但在您的配置中：error_log /var/log/nginx/error.log;。而且，nginx错误日志的格式不一样，不是你这里展示的那种。
@VBart 是的。在上面添加。
好的，现在很清楚 nginx 无法连接到 gunicorn。 gunicorn 日志中的这些错误似乎是罪魁祸首。也许您应该尝试向 gunicorn 开发人员报告错误。
@VBart 感谢您的信息。我不知道是 gunicorn 还是 nginx 有问题。我正在设置 uWSGI 并且情况似乎更好。
【参考方案1】：
这是一个非常古老的帖子。但是我在 NGinx+Gunicorn+Flask 设置中遇到了完全相同的问题。在每 300 个左右的请求中，我也会遇到与您相同的日志的 502 错误。将 gunicorn 工作类型更改为异步为我解决了这个问题（我选择了 gthread）。希望这个答案对某人有所帮助。
如何更改设置：http://docs.gunicorn.org/en/stable/settings.html#worker-class
如何选择您的工人类型： http://docs.gunicorn.org/en/latest/design.html#choosing-a-worker-type
这里有一个很好的解释： How many concurrent requests does a single Flask process receive?
【讨论】：
以上是关于使用 nginx+gunicorn+django 的间歇性 502s的主要内容，如果未能解决你的问题，请参考以下文章
使用 Django、Nginx、Gunicorn 和主管进行空日志记录
django-gunicorn-nginx：502 网关错误
Django 不使用 NGINX + GUNICORN 提供静态文件
使用 nginx 和 gunicorn 运行多个 django 项目