Haproxy 和间歇性 503 问题

Posted 2023-03-10

【中文标题】Haproxy 和间歇性 503 问题

【英文标题】：Haproxy and intermittent 503 issues

【发布时间】：2015-10-24 23:01:29

【问题描述】：

我在 VirtualBox VM (Boot2docker) 中使用 Haproxy 1.5.14，它加载的资产间歇性显示 503，没有任何真正的韵律或原因，在集群启动时尤其如此。

集群看起来像这样，1 个带有端口 80 和 443 的前端到 2 个分别为静态资源和 websocket 服务的后端。

代理

FE（前端，用于静态资源） BE（后端，用于 websocket 连接）

例如，前端服务的静态资产可能是

https://local.dev.myproject.com/assets/images/back.png

尽管前端服务器已启动，但没有任何改变，点击刷新并查看 chrome 调试器，我会看到许多状态 503 或 OK 200 304，但这不是确定性的。它可以在任何资产上从 503 到 OK 再回到 503。当直接连接到网络服务器时，资产返回正常，所以看起来有点像 haproxy。

我能想到的最好的结果是运行状况检查无法正常工作，并且 FE/BE 服务器暂时从 haproxy 的内部名册中删除，但这没有意义，它每半秒检查一次，我可以看到调用 haproxy 正在发送由 FE/BE 终端输出窗口正常返回，按预期每半秒一次。

如果我查看 haproxy 统计报告，我可以看到服务器周期性地来来往往，闪烁，尽管在终端窗口中 haproxy 仍在不间断地调用健康检查，并且服务器按预期返回它们。

附件是我正在使用的当前 haproxy 配置，不胜感激。

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    #log         127.0.0.1 local2
    # log /lnl_zoom_shared/log    local0
    # log /lnl_zoom_shared/log    local1 notice

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # SSL
    #ca-base /etc/ssl
    #crt-base /etc/ssl
    ca-base  /myproject_shared/SECURITY/local.dev.myproject.com/
    crt-base  /myproject_shared/SECURITY/local.dev.myproject.com/
    tune.ssl.default-dh-param 1024

    # turn on stats unix socket
    #stats socket /var/lib/haproxy/stats

    # Exposes the stat socket so we can manage the proxy through node.js
    stats socket /tmp/haproxy.sock level admin

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  http-server-close
    option                  http-pretend-keepalive
    option                  dontlognull
    option                  redispatch
    option                  contstats
    option forwardfor       except 127.0.0.0/8


    retries                 3
    backlog                 10000
    timeout client          25s
    timeout connect         10s
    timeout server          25s
    #long timeoutfor websocket connections
    timeout tunnel          3600s
    timeout http-keep-alive 1s
    timeout http-request    15s
    timeout queue           30s
    timeout tarpit          60s
    default-server inter 3s rise 2 fall 3
    #timeout check           10s

    maxconn                 256

#---------------------------------------------------------------------
# Haproxy's internal stats on the servers below: password protected
#---------------------------------------------------------------------
 stats enable
    stats auth admin:myadminpassword
    stats uri /haproxy
    stats refresh 5s


#---------------------------------------------------------------------
#
#---------------------------------------------------------------------
frontend public
   # HTTP
   bind *:80

     # Redirect all HTTP traffic to HTTPS
       redirect scheme https if ! ssl_fc 

      # HTTPS
      # Example with CA certificate bundle
      # bind :443 ssl crt cert.pem ca-file bundle.crt
      # Example without CA certification bunch
       bind *:443 ssl crt /myproject_shared/SECURITY/local.dev.myproject.com/local.dev.myproject.com.pem


      acl url_static_BE    path_beg    -i /BE /primus
      use_backend BE          if url_static_BE

      # FRONT END (aka FE) STATIC ASSETS SERVER
      # if path is a static asset, assume the front end server to handle it
      acl url_static    path_beg    -i /static /images /javascript /stylesheets
      acl url_static    path_end    -i .jpg .gif .png .css .js .html .ico
      use_backend FE          if url_static



      # GIT HOOKS for UPDATE on the git repo changes
      acl url_githook       path_beg       -i /gitupdate
      use_backend HACNTL          if url_githook

      #BACK END (aka BE) 



  default_backend BE




#---------------------------------------------------------------------
# controller for haproxy
#---------------------------------------------------------------------
backend HACNTL
   # Tell the backend that this is a secure connection,
   # even though it's getting plain HTTP.
   option forwardfor
   http-request add-header X-Forwarded-Proto https if  ssl_fc 

    server      SELF 127.0.0.1:3300

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend FE
   # Tell the backend that this is a secure connection,
   # even though it's getting plain HTTP.
   option forwardfor
   http-request add-header X-Forwarded-Proto https if  ssl_fc 
   option httpchk GET /haproxy/getstatus
   option httpchk HEAD /
   balance     roundrobin

   #server      FE1 11.22.33.44:8000  maxconn 256
     server FE_172.17.0.2  172.17.0.2:8000 maxconn 256 check inter 500ms

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend BE
  # Tell the backend that this is a secure connection,
  # even though it's getting plain HTTP.
  option forwardfor
  http-request add-header X-Forwarded-Proto https if  ssl_fc 
  #http-request set-header X-Custom-Header %[url]
  #http-request set-header Connection upgrade
  #http-request set-header Upgrade websocket
  option httpchk GET /haproxy/getstatus
  cookie SRVNAME insert nocache
  balance     roundrobin


     server BE_172.17.0.3 172.17.0.3:8888 maxconn 256 cookie       BE_172.17.0.3 check inter 500ms

【参考方案1】：

虽然不是绝对的修复，但允许每台服务器一次启动一个已经解决了这个问题。基本上在 docker run 命令之间添加一个睡眠

【讨论】：

我有同样的情况，haproxy->docker nginx 和 uwsgi 应用程序，当页面刷新时我经常得到 503。您能否详细说明您的解决方案，“允许每台服务器一次启动一个”是什么意思？

@ksirg 有几个不同的 503 问题，来自不同来源的相同错误。我的应用程序有几个用于 Web 服务的 docker 容器（例如 1）数据库，2）前端服务器 3）haproxy to loadbalance）当我一次启动它们时，会在启动时立即导致 503 错误。一次启动一个，它们之间有几秒钟的延迟减少了启动时的 503，但在刷新时，在其他时候它们仍然出现，原因我不明白。但我认为这与网络有关。