记一次nginx 504排查

Posted Java_Chuck

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了记一次nginx 504排查相关的知识,希望对你有一定的参考价值。

有一个导出接口,由于里面组装数据会很慢,直接表现为504 Gateway Time-out,时间为2min,梳理了下调用的链路,nginx -> spring gateway -> business server,里面打开了nginx的配置,如下:

#user  nobody;
worker_processes  4;

error_log  /app/openresty/nginx/logs/error.log  error;
pid        /app/openresty/nginx/logs/nginx.pid;
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;


events {
    worker_connections 65536 ;
}


http {
    include       mime.types;
    include       custom_upstream.conf;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr|$remote_user|[$time_local]|"$request"'
                      '|$status|$request_time|$body_bytes_sent|"$http_referer"'
                      '|"$http_user_agent"|"$http_x_forwarded_for"|$upstream_response_time|$upstream_status';    
    access_log  /app/openresty/nginx/logs/access.log  main;
    
    server_tokens   off;
    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;
    client_max_body_size 100m;
    client_header_buffer_size 100m;
    keepalive_timeout  120;
    proxy_send_timeout 120;
    proxy_read_timeout 120;
    proxy_buffering off;
    gzip on;

    server {
        listen       80;
        server_name  localhost;

        listen 443 ssl;
        ssl_certificate_key /app/openresty/nginx/conf/key/sf.key;
        ssl_certificate /app/openresty/nginx/conf/key/sf.crt;

        #charset koi8-r;

        include       custom_location.conf;

        #location / {
        #    root   html;
        #    index  index.html index.htm;
        #}

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
		location = /50x.html {
            root   html;
        }

        location = /nstats {
            check_status;
            access_log      off;
            allow           10.116.0.0/16;
            allow           10.110.0.0/16;
            allow           10.117.0.0/16;
            allow           10.150.0.0/16;
            allow           100.0.0.0/8;
            allow           10.0.0.0/8;
            deny            all;
        }
    }
}
	location / {
		root   html;
		index  index.html index.htm;
	}
	

	location ^~ /foo {
		proxy_pass http://bar:8080;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header Cookie $http_cookie;
		index index.html index.jsp login.jsp index.htm;    
	}

配置经过处理,已替换成foo,bar

果然在配置中找到了2min的配置,我们的接口是一个导出的接口,自然是跟响应有关,说干就干,那就简单了,改成30min分钟。

    keepalive_timeout  120;
    proxy_send_timeout 120;
    proxy_read_timeout 120;

为了不影响其他接口,这里我们新增了一个精确匹配的路由

    location = /foo/downloaddatafile {

        proxy_pass http://bar:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Cookie $http_cookie;
        keepalive_timeout  120;
        proxy_send_timeout 120;
        proxy_read_timeout 1800;        
    }

重启一波,点击导出,依然是504,这次的超时竟然还变短了,1min.what???这是什么情况。

只能打开浏览器的respose,很幸运是有东西的,提示我们去检查nginx的error.log.

好的,那我们就去看看吧,关键信息:[error] 33#0: *703 upstream timed out (110: Connection timed out)

度娘告诉我们需要改缓冲配置。

 

    location = /aom-asset-repair/repair/downloaddatafile {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Cookie $http_cookie;
        keepalive_timeout  120;
        proxy_send_timeout 120;
        proxy_read_timeout 1800;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffer_size 64k;
        proxy_buffers   4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;
        proxy_pass http://bar:8080;
    }

写在最后:

1. proxy_pass配置的顺序为啥会影响其他配置项生效?

2.导出这种慢操作,建议优化成异步处理,后台异步处理时,可以使用多线程。

 

以上是关于记一次nginx 504排查的主要内容,如果未能解决你的问题,请参考以下文章

记一次nginx 504排查

记一次nginx 504排查

记一次 nginx 504 Gateway Time-out

记一次nginx 500

记一次生产环境Nginx日志骤增的问题排查过程

记一次nginx部署yii2项目时502 bad gatewary错误的排查