Haproxy+Heartbeat 高可用集群方案操作记录
Posted kevingrace
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Haproxy+Heartbeat 高可用集群方案操作记录相关的知识，希望对你有一定的参考价值。
之前详细介绍了haproxy的基础知识点, 下面记录下Haproxy+Heartbeat高可用web集群方案实现过程, 以加深理解.
架构草图如下:
1) 基本环境准备 (centos6.9系统)
172.16.60.208(eth0) HA主节点(ha-master) haproxy,heartbeat 172.16.60.207(eth0) HA备节点(ha-slave) haproxy,heartbeat 172.16.60.229 VIP地址 172.16.60.204(eth0) 后端节点1(rs-204) nginx/tomcat 172.16.60.205(eth0) 后端节点2(rs-205) nginx/tomcat 1) 关闭防火墙和selinux (四台节点机都操作) [[email protected] ~]# /etc/init.d/iptables stop [[email protected] ~]# setenforce 0 [[email protected] ~]# vim /etc/sysconfig/selinux SELINUX=disabled 2) 设置主机名和绑定hosts (两台HA节点机器都操作) 主节点操作 [[email protected] ~]# hostname ha-master [[email protected] ~]# vim /etc/sysconfig/network HOSTNAME=ha-master [[email protected] ~]# vim /etc/hosts 172.16.60.208 ha-master 172.16.60.207 ha-slave 备节点操作 [[email protected] ~]# hostname ha-slave [[email protected] ~]# vim /etc/sysconfig/network HOSTNAME=ha-slave [[email protected] ~]# vim /etc/hosts 172.16.60.208 ha-master 172.16.60.207 ha-slave
2) 安装后端两个realserver节点的web环境 (即172.16.60.204/205两台机器都要安装nginx)
采用yum方式在两台realserver节点上安装nginx (先安装nginx的yum源) [[email protected] ~]# rpm -ivh http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm [[email protected] ~]# yum install -y nginx rs-204的nginx配置 [[email protected] ~]# cd /etc/nginx/conf.d/ [[email protected] conf.d]# cat default.conf [[email protected] conf.d]# >/usr/share/nginx/html/index.html [[email protected] conf.d]# vim /usr/share/nginx/html/index.html this is test page of realserver01:172.16.60.204 [[email protected] conf.d]# /etc/init.d/nginx start Starting nginx: [ OK ] [[email protected] conf.d]# lsof -i:80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nginx 31944 root 6u IPv4 91208 0t0 TCP *:http (LISTEN) nginx 31945 nginx 6u IPv4 91208 0t0 TCP *:http (LISTEN) rs-205的nginx配置 [[email protected] src]# cd /etc/nginx/conf.d/ [[email protected] conf.d]# cat default.conf [[email protected] conf.d]# >/usr/share/nginx/html/index.html [[email protected] conf.d]# vim /usr/share/nginx/html/index.html this is test page of realserver02:172.16.60.205 [[email protected] conf.d]# /etc/init.d/nginx start Starting nginx: [ OK ] [[email protected] conf.d]# lsof -i:80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nginx 20839 root 6u IPv4 289527645 0t0 TCP *:http (LISTEN) nginx 20840 nginx 6u IPv4 289527645 0t0 TCP *:http (LISTEN) 访问http://172.16.60.204/, 访问结果为"this is test page of realserver01:172.16.60.204" 访问http://172.16.60.205/, 访问结果为"this is test page of realserver02:172.16.60.205"
3) 安装配置Haproxy (两台HA节点机进行同样操作)
1) 先安装haproxy [[email protected] ~]# yum install gcc gcc-c++ make openssl-devel kernel-devel [[email protected] ~]# cd /usr/local/src/ #下载haproxy软件到/usr/local/src目录下 [[email protected] src]# ls haproxy-1.8.12.tar.gz haproxy-1.8.12.tar.gz [[email protected] src]# tar -zvxf haproxy-1.8.12.tar.gz [[email protected] src]# cd haproxy-1.8.12 [[email protected] haproxy-1.8.12]# make TARGET=linux26 CPU=x86_64 PREFIX=/usr/local/haprpxy USE_OPENSSL=1 ADDLIB=-lz 参数说明： TARGET=linux26 #使用 uname -r 查看内核，如：2.6.32-642.el6.x86_64，此时该参数就为linux26 CPU=x86_64 #使用 uname -r 查看系统信息，如 x86_64 GNU/Linux，此时该参数就为 x86_64 PREFIX=/usr/local/haprpxy #haprpxy 安装路径 [[email protected] haproxy-1.8.12]# ldd haproxy | grep ssl libssl.so.10 => /usr/lib64/libssl.so.10 (0x00000031d0400000) [[email protected] haproxy-1.8.12]# make install PREFIX=/usr/local/haproxy [[email protected] haproxy-1.8.12]# mkdir -p /usr/local/haproxy/conf [[email protected] haproxy-1.8.12]# mkdir -p /etc/haproxy [[email protected] haproxy-1.8.12]# cp /usr/local/src/haproxy-1.8.12/examples/option-http_proxy.cfg /usr/local/haproxy/conf/haproxy.cfg [[email protected] haproxy-1.8.12]# ln -s /usr/local/haproxy/conf/haproxy.cfg /etc/haproxy/haproxy.cfg [[email protected] haproxy-1.8.12]# cp -r /usr/local/src/haproxy-1.8.12/examples/errorfiles /usr/local/haproxy/errorfiles [[email protected] haproxy-1.8.12]# ln -s /usr/local/haproxy/errorfiles /etc/haproxy/errorfiles [[email protected] haproxy-1.8.12]# mkdir -p /usr/local/haproxy/log [[email protected] haproxy-1.8.12]# touch /usr/local/haproxy/log/haproxy.log [[email protected] haproxy-1.8.12]# ln -s /usr/local/haproxy/log/haproxy.log /var/log/haproxy.log [[email protected] haproxy-1.8.12]# cp /usr/local/src/haproxy-1.8.12/examples/haproxy.init /etc/rc.d/init.d/haproxy [[email protected] haproxy-1.8.12]# chmod +x /etc/rc.d/init.d/haproxy [[email protected] haproxy-1.8.12]# chkconfig haproxy on [[email protected] haproxy-1.8.12]# ln -s /usr/local/haproxy/sbin/haproxy /usr/sbin 2) haroxy.cfg文件进行负载配置 [[email protected] haproxy-1.8.12]# cd /usr/local/haproxy/conf/ [[email protected] conf]# cp haproxy.cfg haproxy.cfg.bak [[email protected] conf]# > haproxy.cfg [[email protected] conf]# vim haproxy.cfg global log 127.0.0.1 local3 info maxconn 65535 chroot /usr/local/haproxy uid 99 gid 99 daemon defaults log global mode http retries 3 option redispatch stats uri /haproxy stats refresh 30s stats realm haproxy-status stats auth admin:dxInCtFianKtL]36 stats hide-version maxconn 65535 timeout connect 5000 timeout client 50000 timeout server 50000 frontend http-in mode http maxconn 65535 bind :80 log global option httplog option httpclose acl is_01 hdr_beg(host) www.kevin.com use_backend web-server if is_01 backend web-server mode http balance roundrobin cookie SERVERID insert indirect nocache option httpclose option forwardfor server web01 172.16.60.204:80 weight 1 cookie 3 check inter 2000 rise 2 fall 5 server web02 172.16.60.205:80 weight 1 cookie 4 check inter 2000 rise 2 fall 5 3) 配置HAProxy日志 [[email protected] conf]# vim /etc/rsyslog.conf ....... $ModLoad imudp #取消注释 ，这一行不注释，日志就不会写 $UDPServerRun 514 #取消注释 ，这一行不注释，日志就不会写 ....... local3.* /var/log/haproxy.log #这一行必须要写,因为在haproxy.cfg里global全局定义好的日志级别 [[email protected] conf]# vim /etc/sysconfig/rsyslog SYSLOGD_OPTIONS="-r -m 0" #接收远程服务器日志 重启syslog日志服务 [[email protected] conf]# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ] 4) 设置haproxy负载均衡的最大并发连接数 查看内核 [[email protected] conf]# sysctl -a | grep file fs.file-nr = 992 0 386459 fs.file-max = 386459 查看应用层面的需求 [[email protected] conf]# cat /usr/local/haproxy/conf/haproxy.cfg global #全局参数设置 maxconn 65535 #设置最大连接数 更改系统层面 [[email protected] conf]# vim /etc/security/limits.conf #最后一行增加 * - nofile 65535 5) 重启两台HA机器的haproxy [[email protected] conf]# /etc/init.d/haproxy start Starting haproxy: [ OK ] [[email protected] conf]# ps -ef|grep haproxy nobody 13080 1 0 16:43 ? 00:00:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid root 13083 11940 0 16:43 pts/0 00:00:00 grep haproxy [[email protected] conf]# lsof -i:80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME haproxy 13080 nobody 4u IPv4 428975 0t0 TCP *:http (LISTEN) 将www.kevin.com域名解析到两个HA节点上, 即172.16.60.208 和 172.16.60.207上 接着访问http://www.kevin.com/, 则发现访问结果是"this is test page of realserver01:172.16.60.204" , 不断刷新, 访问结果也是这个. 只有当172.16.60.204这个节点的nginx挂了, 访问结果才变成"this is test page of realserver02:172.16.60.205", 即请求转发到正常的realserver节点上. 从haproxy.cfg文件中可以看出, 虽然配置了"balance roundrobin"这个选项, 即客户端每一次访问, 都跳转到后端不同的服务器上. 但是并没有生效! 因为又配置了"cookie SERVERID insert indirect nocache", 即保持客户端session会话同步的配置, 所以客户端请求会一直转发到同一个realserver节点上,直至 这个节点发生故障才会转发到另外正常的节点上. 把"cookie SERVERID insert indirect nocache" 这个配置去掉或注释掉, 再次访问http://www.kevin.com/, 就会发现每刷新一次, 请求都会转发到不同的realserver 节点上, 即"balance roundrobin" 配置生效! 访问http://www.kevin.com/haproxy, 输入haproxy.cfg文件中配置的用户名和密码admin:dxInCtFianKtL]36， 即可打开haproxy监控页面
从上图可以看出， 此时监控的后端两个realserver节点的服务都是OK的（配置文件中定义的web01和web02此时都是绿色状态）。 现在尝试关闭rs-205的nginx服务， 刷新http://www.kevin.com/haproxy监控页面， 发现web02变成红色，即此时该节点服务是故障状态！然后重启rs-205的nginx服务，再次刷出监控页面， 发现web02就又恢复到正常的绿色状态了！
4) 安装配置Heartbeat (两台HA节点机进行同样操作)
以上是关于Haproxy+Heartbeat 高可用集群方案操作记录的主要内容，如果未能解决你的问题，请参考以下文章
keepalived,heartbeat,lvs,haproxy
Linux高可用集群方案之配置heartbeat v2基于haresources配置文件的httpd高可用集群