Linux集群架构

Posted jack的网络日志

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Linux集群架构相关的知识,希望对你有一定的参考价值。

负载均衡集群介绍

 LVS介绍

 

 

 

优点:在NAT模式中,只需要分发器有公网Ip,分发器和rs之间可以用内网通信,所以比较节省公网ip资源

缺点:分发器成为瓶颈,请求量不能太大。规模保持在10台左右。

我们可以想象分发器和每个rs之间建立了一个虚拟通道

这种模式,需要有一个公共的IP配置在分发器和所有rs上,我们把它叫做vip,

原理是客户端请求的目标IP为vip,分发器接收到请求数据包后,会对数据包做一个交工,会把目标IP由vip更改为rs的IP,这样数据包就到了rs上。

rs接受数据包后,会还原原始数据包,这样目标IP为vip,因为所有的rs上配置了这样一个vip,所以它会认为是它自己。

nat模式iptables转发,ip tunneling将目标ip做了一个更改,dr模式和ip tunneling类似,不同点dr更改的是mac地址

LVS的调度算法

常用的4种

轮询:Round-Robin rr                        (任务均衡的分发到rs上)

加权轮询:Weight Round-Robin wrr                 (带权重的轮询,例如一个rs的配置好,你可以将它权重设置高一点)

最小连接:Least-Connection lc                     (把新的任务发送到请求数少的rs上)

加权最小连接:Weight Least-Connection wlc           (带权重的最小连接)

不常用的4种

基于局部性的最小连接:Locality-Based Least Connections lblc    

带复制的基于局部性最小链接:Locality-Based Least Connections with Replication lblcr

目标地址散列调度:Destination Hashing dh

源地址散列调度:Source Hashing sh

LVS NAT模式搭建

1.nat是通过iptables实现的,所以必须配置些iptables规则,首先准备三台机器,其中有一台作为分发器(dir),另外两台为read seruver(rs1,rs2)

2.现在我们有两台虚拟机了,还需要克隆一台(用户centos01或centos02克隆)或重新装一台centos03,克隆(如何克隆前面有讲)完成后开启centos03,修改ip地址,设置好网段,我们这里设置为192.168.133.0网段,三台机器ip分别为01(192.168.133.44)02(192.168.133.88)03(192.168.133.66),其中分发器需要有两个网卡一个链接内网一个链接外网(也就是01服务器)02和03服务器只需要有一个内网网卡就行了

3.修改03服务器的ip地址为66

4.重启网卡

5.用ssh工具链接测试,链接03服务器成功

Connecting to 192.168.133.66:22...
Connection established.
To escape to local shell, press \'Ctrl+Alt+]\'.

Last login: Mon Mar  5 21:52:13 2018 from 192.168.133.1
[root@centos-03 ~]# 
[root@centos-03 ~]# 

6.修改主机名,主机名没变的话输入bash命令进一个子shell或重新链接ssh

[root@centos-03 ~]# hostnamectl set-hostname centos-03
[root@centos-03 ~]# bash

7.开启02服务器

8.查看01有几个网卡,如果有一个需要再添加一个网卡选择仅主机模式  

[root@centos-01 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.133.44  netmask 255.255.255.0  broadcast 192.168.133.255
        inet6 fe80::22b5:c83f:c27e:7a9f  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:f9:65:0b  txqueuelen 1000  (Ethernet)
        RX packets 1220  bytes 98455 (96.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 178  bytes 28363 (27.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens37: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.23.128  netmask 255.255.255.0  broadcast 192.168.23.255
        inet6 fe80::e29e:ee3f:2dfe:7ca1  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:f9:65:15  txqueuelen 1000  (Ethernet)
        RX packets 26  bytes 4100 (4.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 37  bytes 5979 (5.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

9.点击虚拟机编辑查看外网网段为23

编辑配置文件ip修改为110重启网络服务(如果没有新网卡配置文件需要拷贝一份ens33的,修改ip为23.110)

[root@centos-01 network-scripts]# cp ifcfg-ens33 ifcfg-ens37
[root@centos-01 network-scripts]# vim ifcfg-ens37
[root@centos-01 network-scripts]# 

TYPE=Ethernet
BOOTPROTO=static
IPADDR=192.168.23.110
NETMASK=255.255.255.0
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens37
DEVICE=ens37
ONBOOT=yes

重启网卡  

[root@centos-01 network-scripts]# systemctl restart network
[root@centos-01 ~]# ifup ens37

 

查看ip变为110了

[root@centos-01 network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.133.44  netmask 255.255.255.0  broadcast 192.168.133.255
        inet6 fe80::22b5:c83f:c27e:7a9f  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:f9:65:0b  txqueuelen 1000  (Ethernet)
        RX packets 2196  bytes 219838 (214.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1665  bytes 232258 (226.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens37: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.23.110  netmask 255.255.255.0  broadcast 192.168.23.255
        inet6 fe80::ebbe:cf4f:f8be:96bd  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:f9:65:15  txqueuelen 1000  (Ethernet)
        RX packets 216  bytes 25268 (24.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 175  bytes 25477 (24.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  

10.Windows上可以ping通

C:\\Users\\Administrator>ping 192.168.23.110

正在 Ping 192.168.23.128 具有 32 字节的数据:
来自 192.168.23.128 的回复: 字节=32 时间=1ms TTL=64
来自 192.168.23.128 的回复: 字节=32 时间<1ms TTL=64
来自 192.168.23.128 的回复: 字节=32 时间<1ms TTL=64

192.168.23.128 的 Ping 统计信息:
    数据包: 已发送 = 3,已接收 = 3,丢失 = 0 (0% 丢失),
往返行程的估计时间(以毫秒为单位):
    最短 = 0ms,最长 = 1ms,平均 = 0ms
来自 192.168.23.128 的回复: Control-C
^C
C:\\Users\\Administrator>

11.三台机器都关掉防火墙

[root@centos-01 ~]# 
[root@centos-01 ~]# systemctl stop firewalld
[root@centos-01 ~]# systemctl disable firewalld(这样下一次就不会开机启动了)
[root@centos-01 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 149 packets, 10516 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 96 packets, 9320 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@centos-01 ~]# 

12.查看一个包安装了那些服务

[root@centos-01 ~]# rpm -ql iptables-services

三台机器使用centos6中iptables的形式

如果yum安装的很慢,可以临时取消掉epel(是国外的资源),重新执行yum安装

[root@centos-03 ~]# mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.1 

13.开启iptables服务

[root@centos-02 ~]# systemctl start iptables
[root@centos-02 ~]# systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.
[root@centos-02 ~]# 

 

[root@centos-03 ~]# systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.
[root@centos-03 ~]# systemctl start iptables
[root@centos-03 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   62  4628 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
   20   800 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 57 packets, 4696 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@centos-03 ~]# 

  

为了让它调用一个空规则

[root@centos-03 ~]# iptables -F
[root@centos-03 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  确定  ]
[root@centos-03 ~]# 

 

[root@centos-02 ~]# iptables -F
[root@centos-02 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  确定  ]
[root@centos-02 ~]# 

 

[root@centos-01 network-scripts]# iptables -nvL
Chain INPUT (policy ACCEPT 1765 packets, 167K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 2 packets, 656 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1151 packets, 139K bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@centos-01 network-scripts]# 

 

[root@centos-02 ~]# getenforce 
Permissive
[root@centos-02 ~]# setenforce 0

 

改为disabled

[root@centos-03 ~]# vim /etc/selinux/config 
[root@centos-03 ~]# 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted  

 

设置02、03的网关为01的内网IP44

[root@centos-02 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 
[root@centos-02 ~]# 
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=63d602d6-c8ae-4350-b149-aad17fc44e98
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.133.88
NETWORK=255.255.255.0
GATEWAY=192.168.133.44
DNS1=119.29.29.29

 

[root@centos-02 ~]# systemctl restart network
[root@centos-02 ~]# 

 

查看网关

[root@centos-02 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.133.44  0.0.0.0         UG    100    0        0 ens33
192.168.133.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
[root@centos-02 ~]# 
[root@centos-03 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.133.44  0.0.0.0         UG    100    0        0 ens33
192.168.133.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
[root@centos-03 ~]# 

1.在dir上安装ipvsadm (这个工具是实现lvs的重要工具,只用在分发器上安装,如果没网可以挂载cdroom rpm方式安装)

mount /dev/cdroom/   /mnt/

[root@centos-01 Packages]# rpm -ivh ipvsadm-1.27-7.el7.x86_64.rpm 
准备中...                          ################################# [100%]
正在升级/安装...
   1:ipvsadm-1.27-7.el7               ################################# [100%]
[root@centos-01 Packages]# ipvsadm -v

  

[root@centos-01 network-scripts]# yum install -y ipvsadm

2.在dir上编辑脚本,在ipvsadm上最好用脚本去执行命令,方便简洁:

[root@centos-01 Packages]# vim /usr/local/sbin/lvs.nat.sh
[root@centos-01 Packages]# 

  

#! /bin/bash

#! /bin/bash

# director 服务器上开启路由转发功能

echo 1 > /proc/sys/net/ipv4/ip_forward

# director 服务器上开启路由转发功能

echo 1 > /proc/sys/net/ipv4/ip_forward

# 关闭icmp的重定向

echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects

echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects

# 注意区分网卡名字,阿铭的两个网卡分别为ens33和ens37

echo 0 > /proc/sys/net/ipv4/conf/ens33/send_redirects

echo 0 > /proc/sys/net/ipv4/conf/ens37/send_redirects

# director 设置nat防火墙

iptables -t nat -F

iptables -t nat -X

#同网段的内网可以上网

iptables -t nat -A POSTROUTING -s 192.168.133.0/24  -j MASQUERADE

# director设置ipvsadm

IPVSADM=\'/usr/sbin/ipvsadm\'

$IPVSADM -C

$IPVSADM -A -t 192.168.23.110:80 -s lc -p 3

$IPVSADM -a -t 192.168.23.110:80 -r 192.168.133.88:80 -m -w 1

$IPVSADM -a -t 192.168.23.110:80 -r 192.168.52.66:80 -m -w 1

[root@centos-01 Packages]# sh /usr/local/sbin/lvs.nat.sh 
[root@centos-01 Packages]# 

测试:两台机子上都要安装nginx/

rs1用Yum安装nginx,修改主页:

[root@centos-02 ~]# ps aux|grep nginx
root      51748  0.0  0.2 122892  2104 ?        Ss   3月07   0:00 nginx: master process /usr/sbin/nginx
nginx     51749  0.0  0.3 123356  3568 ?        S    3月07   0:00 nginx: worker process
root     106788  0.0  0.0 112664   932 pts/1    S+   00:15   0:00 grep --color=auto nginx
[root@centos-02 ~]# netstat -nlpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      51748/nginx: master 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      918/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1480/master         
tcp6       0      0 :::80                   :::*                    LISTEN      51748/nginx: master 
tcp6       0      0 :::22                   :::*                    LISTEN      918/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1480/master         
[root@centos-02 ~]# 
[root@centos-02 ~]# echo 02 > /usr/share/nginx/html/index.html 
[root@centos-02 ~]# curl localhost
02
[root@centos-02 ~]# 

rs2

[root@centos-03 ~]# systemctl start nginx
[root@centos-03 ~]# echo 03 > /usr/share/nginx/html/index.html 
[root@centos-03 ~]# curl localhost
03
[root@centos-03 ~]# 

ipvsadm查看下

[root@centos-01 Packages]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.23.110:80 lc persistent 3
  -> 192.168.133.66:80            Masq    1      0          0         
  -> 192.168.133.88:80            Masq    1      0          0         
[root@centos-01 Packages]# 

 

修改下算法

[root@centos-01 Packages]# vim /usr/local/sbin/lvs.nat.sh 
[root@centos-01 Packages]# 
$IPVSADM -A -t 192.168.23.110:80 -s rr     

$IPVSADM -a -t 192.168.23.110:80 -r 192.168.133.88:80 -m -w 1

$IPVSADM -a -t 192.168.23.110:80 -r 192.168.133.66:80 -m -w 1
[root@centos-01 Packages]# sh /usr/local/sbin/lvs.nat.sh 
[root@centos-01 Packages]#   

 

哈哈哈成功了,负载均衡了

[root@centos-01 Packages]# curl 192.168.23.110
03
[root@centos-01 Packages]# curl 192.168.23.110
02
[root@centos-01 Packages]# curl 192.168.23.110
03
[root@centos-01 Packages]# curl 192.168.23.110
02
[root@centos-01 Packages]# curl 192.168.23.110
03
[root@centos-01 Packages]# curl 192.168.23.110
02
[root@centos-01 Packages]# 

LVS DR模式搭建  

准备工作

三台机器:

分发器,dir:192.168.133.44

rs1:192.168.133.88  (网关改回来133.2,之前nat模式改成100)

rs2: 192.168.133.66  (网关改回来133.2,之前nat模式改成100)

vip:192.168.133.200   (这个虚拟ip需要绑定在每台机器上)

rs1和rs2上修改配置文件gateway为原来的133.2网关,并重启网卡 systemctl restart network

1.编辑dr模式的配置文件

[root@centos-01 ~]# vim /usr/local/sbin/lvs.dr.sh
[root@centos-01 ~]# 

#! /bin/bash

echo 1 > /proc/sys/net/ipv4/ip_forward

ipv=/usr/sbin/ipvsadm

vip=192.168.133.200

rs1=192.168.133.88

rs2=192.168.133.66

#注意这里的网卡名字

ifdown ens33

ifup ens33

ifconfig ens33:2 $vip broadcast $vip netmask 255.255.255.255 up

route add -host $vip dev ens33:2

$ipv -C

$ipv -A -t $vip:80 -s rr

$ipv -a -t $vip:80 -r $rs1:80 -g -w 1

$ipv -a -t $vip:80 -r $rs2:80 -g -w 1  

2.执行shell脚本

[root@centos-01 ~]# sh /usr/local/sbin/lvs.dr.sh
成功断开设备 \'ens33\'。
成功激活的连接(D-Bus 激活路径:/org/freedesktop/NetworkManager/ActiveConnection/14)
[root@centos-01 ~]# 

3.配置r1

[root@centos-02 ~]# vim /usr/local/sbin/lvs.rs.sh
[root@centos-02 ~]# 

#/bin/bash
vip=192.168.133.200
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端

ifdown lo

ifup lo

ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up

route add -host $vip lo:0

#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端

#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
[root@centos-02 ~]# sh !$
sh /usr/local/sbin/lvs.rs.sh
[root@centos-02 ~]# 

4.配置r2

[root@centos-03 ~]# vim /usr/local/sbin/lvs.dr.sh
#/bin/bash
vip=192.168.133.200
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端

ifdown lo

ifup lo

ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up

route add -host $vip lo:0

#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端

#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
[root@centos-03 ~]# sh !$

5.查看网关

[root@centos-02 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.133.2   0.0.0.0         UG    100    0        0 ens33
192.168.133.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.133.200 0.0.0.0         255.255.255.255 UH    0      0        0 lo
[root@centos-02 ~]# 
[root@centos-03 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.133.2   0.0.0.0         UG    100    0        0 ens33
192.168.133.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.133.200 0.0.0.0         255.255.255.255 UH    0      0        0 lo
[root@centos-03 ~]# 

6.查看nat iptables规则

[root@centos-01 ~]# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 84 packets, 9395 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 25 packets, 3787 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 49 packets, 3517 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 12 packets, 808 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    7   633 MASQUERADE  all  --  *      *       192.168.133.0/24     0.0.0.0/0           
[root@centos-01 ~]# 

7.关闭nat iptables

[root@centos-01 ~]# iptables -t nat -F

8.测试:DR模式是没有办法用浏览器显著的看出结果,也不能用rs1,rs2访问,因为vip绑定在lo上,他们会默认访问自己,最好的办法就是从新开启一个虚拟机,ip设置在同一网段,用curl查看。

 

9.用ipvsadm查看

[root@centos-01 ~]# ipvsadm  -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.133.200:80 rr
  -> 192.168.133.66:80            Route   1      0          0         
  -> 192.168.133.88:80            Route   1      0          0         
[root@centos-01 ~]# 

二、DR模式只用一个IP实现

 

201006241277382974737.jpg

在前面加一个路由器,原理就是让 路由器把所有的80端口请求都分给VIP,分发器再分给每个web服务器,而web服务器处理完请求后跟客户连接就不走分发器了,直接通过路由器去外网了,这样就实现了只用一个公网IP也能用DR模式

先从内网找了三台服务器分别是:

192.168.1.166 web1

192.168.1.167 web2

192.168.1.160 分发器

192.168.1.169 VIP

192.168.1.1   路由器内网ip(网关) 路由器是随便找的一台tplink adal路由器,凑合着测试用的

211.83.113.119 路由器的WAN口IP (随便蒙的,重复莫怪)

 

先安装ipvsadm 直接yum install ipvsadm就行了,不多说

分发器上的设置

global_defs {

   notification_email {

        ufo@xman.com

   }

 

notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server smtp.qq.com

   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_sync_group VG1 {

        group{

        VI_1

                }

                    }

vrrp_instance VI_1 {

        state MASTER

        interface eth0

        virtual_router_id 51

        priority 100

        advert_int 1

 

        authentication {

        auth_type PASS

        auth_pass 33210

                        }

 

        virtual_ipaddress {

        192.168.1.169

                          }

 

        virtual_server 192.168.1.169 80 {

        delay_loop 6

        lb_algo rr

        lb_kind DR

        protocol TCP

 

                real_server 192.168.1.166 80 {

                weight 1

                inhibit_on_failure

                TCP_CHECK {

                connect_timeout 5

                nb_get_retry 3

                delay_before_retry 3

                connect_port 80

                          }

                                              }

                real_server 192.168.1.167 80 {

                weight 1

                inhibit_on_failure

                TCP_CHECK {

                connect_timeout 5

                nb_get_retry 3

                delay_before_retry 3

                connect_port 80

                          }

                                              }

配置文件写完了,然后就是

mkdir /etc/keepalived  #系统默认会到这里去找配置文件

cp /usr/local/keepalive/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/keepalive/etc/rc.d/init.d/keepalived /etc/init.d/

cp /usr/local/keepalive/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalive/sbin/keepalived /bin/ #将可执行程序放入sbin 或者 bin目录里

 

vim /etc/sysctl.conf

net.ipv4.ip_forward = 1

保存退出 后执行sysctl -p

route add defaule gw 192.168.1.1 把路由内网地址添加为默认网关

web服务器设置

两台web服务器也要修改 /etc/sysctl.conf 修改内容如下

vim /etc/sysctl.conf

# LVS

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.lo.arp_ignore = 1

net.ipv4.conf.lo.arp_announce = 2

sysctl -p

之后还要增加vip

ifconfig lo:1 192.168.1.169 netmask 255.255.255.255 别忘了加到rc.local里面

route add defaule gw 192.168.1.1 把路由内网地址添加为默认网关

 

路由器设置

路由器的设置没什么好说的,除了上网设置以外还要做一个端口映射,就是把80端口映射到 vip上也就是192.168.1.169

 

现在启动keepalived吧

/etc/init.d/keepalived start

开始的时候比较慢,大概1分钟后系统日志里面出现下面这条记录就OK了

avahi-daemon[3012]: Registering new address record for 192.168.1.169 on eth0

 

我们访问一下 http://211.83.113.119

我把我们的应用程序放到了上面跑了一下,呵呵测试成功

ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.1.169:80 rr

  -> 192.168.1.166:80             Route   1      5          6         

  -> 192.168.1.167:80             Route   1      3          9

 

   后来遇到了一个问题,由于这套应用处在一个大网站的后台,所以大部分的请求都来自同一个IP地址,而有一部分程序需要给每个连接做session粘滞,

这样我就不能用lvs 的-p参数来设置ip粘滞时间,如果用lvs的粘滞时间的话大部分的请求都将分给同一台web服务器(注意:这里是session粘滞而不是IP粘滞),

lvs可做不到这点,怎么办呢?

在cu论坛上询问后得知有很多朋友做过类似的项目,他们的解决办法是 将session共享,共享到什么地方就有很多选择了

我们是把所有web服务器的php session都给memcached ,这样你不管分发器把 ip连接分给哪个web服务器都不会有问题了,配置方法很简单,就在php的配置文件内

增加一条语句就可以了,不过前提你需要装好memcache模块

[Session]

; Handler used to store/retrieve data.

session.save_handler = memcache

session.save_path = "tcp://192.168.1.161:11213"

CentOS7/RHEL7安装EPEL扩展仓库步骤详解

方法一 : 命令安装

yum -y install epel-release

2.2 方法二 : 手动安装


针对系统架构选择相应的类型 : http://dl.fedoraproject.org/pub/epel/7/

我们使用的 x86_64, 就要进入该目录下寻找相应包, 安装方法如下 :

rpm -vih http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
  • 1

或者:

wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
rpm -vih epel-release-7-2.noarch.rpm
  • 1
  • 2

接着更新元数据缓存(非必须):

yum clean all && yum makecache
  • 1

3 验证 EPEL 仓库


此时我们发现 /etc/yum.repos.d/ 下多了两个 epel 的 repo 文件:

epel.repo 和 epel-testing.repo

ll /etc/yum.repos.d/
  • 1

yum_list

w3m 是基础包中没有的, 而 epel 包中才有的软件包

sudo yum search w3m
  • 1

search_w3m

4 参照


清华源–EPEL 镜像使用帮助

Centos安装EPEL扩展源

Centos7.x RHEL 配置EPEL第三方YUM源

Extra Packages for Enterprise Linux (EPEL)

How To Install EPEL Repo on a CentOS and RHEL 7.x



知识共享许可协议

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可

版权声明:本文为博主原创文章 && 转载请著名出处 @ http://blog.csdn.net/gatieme http://blog.csdn.net/gatieme/article/details/70232358
 

keepalived lvs  

keepalived有一个较好的功能,如果后台一台rs宕机,便不会再把请求发送过去两台rs上的脚本还是要编写和执行的。

两台rs上的脚本还是要编写和执行lvs.r.sh脚本的。

keepalived有一个较好的功能,如果后台一台rs宕机,便不会再把请求发送过去。

在进行实验之前,需要执行2条命令,把之前ipvsadm规则清空掉,重新加载网卡,把之前设置的vip清空掉。

[root@centos-01 ~]# ipvsadm -C
[root@centos-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@centos-01 ~]# 

  

1.安装keepalived

[root@centos-01 ~]# yum install -y keedalived

  

 2.编辑配置文件

[root@centos-01 ~]# vim /etc/keepalived/keepalived.conf 

vrrp_instance VI_1 {

    #备用服务器上为 BACKUP

    state MASTER

    #绑定vip的网卡为ens33,你的网卡可能不一样

    interface ens33

    virtual_router_id 51

    #备用服务器上为90

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass aminglinux

    }

   #设置vip

    virtual_ipaddress {

        192.168.133.200

    }

}

virtual_server 192.168.133.200 80 {

    #(每隔10秒查询realserver状态)

    delay_loop 10

    #(lvs 算法)

    lb_algo rr

    #(DR模式)

    lb_kind DR

    #(同一IP的连接60秒内被分配到同一台realserver)

    persistence_timeout 60

    #(用TCP协议检查realserver状态)

 protocol TCP

    real_server 192.168.133.88 80 {

        #(权重)

        weight 100

        TCP_CHECK {

        #(10秒无响应超时)

        connect_timeout 10

        nb_get_retry 3

        delay_before_retry 3

        connect_port 80

        }

    }

    real_server 192.168.133.66 80 {

        weight 100

        TCP_CHECK {

        connect_timeout 10

        nb_get_retry 3

        delay_before_retry 3

        connect_port 80

        }

     }

}
[root@centos-01 ~]# systemctl start keepalived
[root@centos-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.133.200:80 rr persistent 60
  -> 192.168.133.66:80            Route   100    0          0         
  -> 192.168.133.88:80            Route   100    0          0         
[root@centos-01 ~]# 

 

keepalived有踢出的功能,当关掉r2上的nginx显示

[root@centos-03 ~]# systemctl stop nginx
[root@centos-03 ~]# 

[root@centos-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.133.200:80 rr persistent 60
  -> 192.168.133.88:80            Route   100    0          0         
[root@centos-01 ~]# 

 

当再次开启r2上的nginx显示

[root@centos-03 ~]# systemctl start nginx
[root@centos-03 ~]# 

[root@centos-01 ~]# 
[root@centos-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.133.200:80 rr persistent 60
  -> 192.168.133.66:80            Route   100    0          0         
  -> 192.168.133.88:80            Route   100    0          0         
[root@centos-01 ~]# 

配置两台rs的配置文件如下:

rs1:192.168.133.88和rs2:192.168.133.66都执行以下脚本

[root@centos-02 ~]# vim /usr/local/sbin/lvs.rs.sh 
[root@centos-02 ~]# 
[root@centos-03 ~]# vim /usr/local/sbin/lvs.dr.sh 
[root@centos-03 ~]# 

  

#/bin/bash

vip=192.168.133.200

#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端

ifdown lo

ifup lo

ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up

route add -host $vip lo:0

#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端

#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

dr上需要执行  

[root@centos-01 ~]# vim /usr/local/sbin/lvs.dr.sh 
[root@centos-01 ~]# 

#! /bin/bash

echo 1 > /proc/sys/net/ipv4/ip_forward

以上是关于Linux集群架构的主要内容,如果未能解决你的问题,请参考以下文章

了解Linux运维要用到的web集群架构知识

Linux集群架构概述

Linux系统集群架构线上项目配置实战

Linux企业运维——K8s高可用集群架构搭建详解

Linux企业运维——K8s高可用集群架构搭建详解

Linux云计算-MySQL-高可用集群架构-MHA 架构