Linux Virtual Server
Posted ㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤ0
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Linux Virtual Server相关的知识,希望对你有一定的参考价值。
文章目录
lvs(Linux Virtual Server)
lvs(Linux Virtual Server),即虚拟服务器,是一个虚拟的服务器集群系统。工作在OSI第四层中,在服务器承受不了业务需求量的时候,使用均衡负载的方式来使服务器能够给用户正常提供服务。
lvs工作流程:
当用户发起请求时,lvs 调度器先将请求接收,调度器会根据收到请求的服务和根据自己预设的工作模式和算法来判断哪台服务器的负载较低,然后将此条请求转发给负载较低提供服务的服务器,由这台提供服务的服务器来处理此条请求,最后提供服务的服务器根据预设的工作模式来决定资源返回给客户端的方式。
lvs工具:
ipvsadm:工作在用户空间的命令行工具,用于管理集群服务
ipvs:工作于内核中netfilter INPUT钩子上
支持TCP,UDP,AH,EST,AH_EST,SCTP等多种协议
lvs专业名词:
调度器:director,dispatcher,balancer
RS:Real Server
Client IP:CIP
Director Virutal IP:VIP
Director IP:DIP
Real Server IP:RIP
lvs四种工作模式
NAT
特点
- RS和DIP必须使用私网地址,且RS的网关要指向DIP。
- 请求和响应报文都要经过director转发,在负载较高的场景种,director的配置会成为系统性能的瓶颈。
- 因为请求和响应报文都要经过director的转发,所以可以支持端口映射。
- RS可以使用任意OS。
- RIP和DIP必须在同一个网络中。
工作流程
如上图:首先客户端向VIP发送请求,报文首部为①,director收到请求后通过预设的算法来判断将请求转发给哪一台RS并且将报文首部修改为②,请求通过DIP转发到RIP,服务器返回资源时,报文首部为③,资源通过RIP转发到DIP,director收到资源后将报文首部修改为④,最后通过VIP返回到CIP客户端。
dr(默认模式)
特点
- dr模式通过修改请求报文的目标MAC地址进行转发。
- director要配置VIP和DIP。
- 所有的RS都要配置RIP和VIP。
- RIP可以使用私有地址,也可以使用公网地址。
- RS跟director必须在同一个物理网络中,中间不能有路由器相隔。
- 请求报文经过director,并且由director调度,但响应报文一定不能经过director。
- 因为响应报文不经过director,所以不支持端口映射。
- 因为响应报文不经过director,所以RS网关不能指向DIP。
- RS可以是大多数OS。
因为所有RS都要配置VIP为了确保客户端将请求发送给director有以下三种解决方案:
- 静态绑定:在客户端上将VIP和director的MAC地址绑定。(可以实现功能,但是不现实)。
- arptables:在所有RS主机上配置arp防火墙,使RS主机即使收到arp包,也不响应,这样只有director收到apr包才会响应,客户端自然只会将请求发送给director。
- 修改RS主机内核的2个参数:
- arp_announce:是否接收并记录别人的通告以及是否通告自己的MAC地址给别人,arp_announce参数有三个值:
- 0:(default,默认为0):通告自己所拥有的所有地址
- 1:尽量不通告与自己不在同一个网段的地址,比如不同高2.0网段的地址给1.0网段,但是是尽量不通告,有可能还是会通告。
- 2:总是不通告与自己不在同一个网段的地址,绝对不会通告。(配置dr模式时,应该将arp_announce参数设置为2)
- arp_ignore:是否响应arp请求,arp_ignore参数有9个可选值,0-8:
- 0:(default,默认为0)回应任何网络接口上对任何本地IP地址的arp查询请求。
- 1:请求报文从那个接口进入的且请求的目标地址就是此接口配置的地址才响应,否则不响应。(只回答目标IP地址是来访网络接口本地地址的arp查询请求)
- 2:只回答目标IP地址是来访问网络接口本地地址的arp查询请求,且来访IP必须在该网络接口的子网段内。
- 3:不回应该网络接口的arp请求,而只对设置的唯一和连接地址做出回应。
- 4-7:保留未使用。
- 8:不回应所有(本地地址)的arp查询。(配置dr模式时,应该将arp_ignore参数设置为1)
- arp_announce:是否接收并记录别人的通告以及是否通告自己的MAC地址给别人,arp_announce参数有三个值:
工作流程
如上图:首先客户端向VIP发送请求,报文首部为①,因为RS的VIP都限制了arp包的回应和通告并且VIP是配置到lo网卡上,所以客户端发送的请求一定是发送到director,director收到请求后通过预设的算法来判断将请求转发给哪一台RS并且将报文首部修改为②,由MAC地址来判断由哪台服务器提供服务,RS收到请求后,通过lo网卡上的VIP将资源返回给客户端,报文首部为③。
tun
特点
- 不修改请求报文的ip首部,而是在原有的ip首部再封装一个ip首部。
- RIP、DIP、VIP必须是公网地址。
- RS的网关不是指向DIP(因为响应报文是不通过director的)。
- 请求报文必须经过director,且由director调度,但响应报文不能经过director。
- 不支持端口映射(因为响应报文是不通过director的)。
- RS的OS必须支持隧道功能(因为要解封装ip首部两次,若不支持则不能识别tun模式的报文首部)。
工作流程
如上图:客户端将请求发送给VIP,报文首部为①,因为RIP是公网地址,而VIP是配置再lo网卡上,客户端从公网访问是找不到RS的VIP的,所以客户端发送的请求一定是发送到director,director收到请求后通过预设的算法来判断将请求转发给哪一台RS并且将报文首部再封装一层报文首部,此时报文首部修改成②,通过外面一层报文首部访问到RS,RS收到请求后,再解封装内层报文首部得知,是客户端通过CIP到director的VIP来访问的,于是通过lo网卡的VIP和客户端的CIP来封装报文首部,并且将资源返回给客户端,报文首部为③
fullnat
特点
- director通过同时修改请求报文的目标地址和源地址进行转发。
- VIP是公网地址,RIP和DIP是私网地址,RIP与DIP无须在用一个网络中。
- RS接收到请求报文的源地址为DIP,因此要响应给DIP。
- 请求报文和响应报文都必须经过director。
- 支持端口映射。
- RS可以使用任意OS。
工作流程
如上图:客户端通过CIP将请求发送给director的VIP,报文首部为①,director收到请求后通过预设的算法来判断将请求转发给哪一台RS,因为RIP和DIP不在同一个网络中,需要经过路由器,所以报文首部为②,RS收到请求后,返回资源时,因为中间需要经过路由器,所以报文首部为③,最后又director将资源返回给客户端,报文首部为④。
lvs scheduler调度算法
静态
仅根据算法本身进行调度。
RR:round robin,论调,平均分配请求。
WRR:weighted rr,加权的rr,根据一定的用户设置比例进行轮调,比如每次RS1给2个请求,RS2给1个请求。
SH:source hash,源地址hash,实现session保持的机制,将来自于同一个IP的请求始终调度至同一RS,每个服务单独调度。
DH:destination hash,目标地址hash,将对同一个目标(资源)的请求始终发往同一个RS。
动态
根据算法及各RS的当前负载状态进行调度,根据指定的算法算出overhead(负载),最终挑选出overhead值最小的则为被选中的RS。
LC:Least Connection,最少连接数,算法如下:
overhead = Active * 256 + Inactive
WLC:Weighted LC,加权的LC,算法如下:
overhead=(Active*256+Inactive)/ weight
SED:Shortest Expection Delay,最短期望延迟,算法如下:
overhead = (Active + 1) * 256 / weight
NQ:Nevel Queue,是SED算法的改进,根据SED算法每台主机第一次至少要均分配一次,然后再按SED算法来挑选。
LBLC:Locality-Based LC,基于本地的最少连接数,即为动态的DH算法。
正向代理情形下的cache server调度。
LBLCR:Locality-Based Least-Connection with Replication,带复制功能的LBLC算法。
lvs配置方法
nat模式
要求:
nat模型实现http和https两种负载均衡集群,注意,各RS都要提供同一个私钥和同一个证书。
ip类型 | ip地址 |
---|---|
VIP | 192.168.159.128 |
DIP | 192.168.247.214 |
R1IP | 192.168.247.211 |
R2IP | 192.168.247.100 |
调度器上添加仅主机模式的网卡配置VIP
安装配置httpd并且生成证书后使用同一个证书
\\\\安装配置过程略
[root@R1 apache]# mkdir /etc/pki/CA
[root@R1 apache]# cd /etc/pki/CA
[root@R1 CA]# mkdir private
[root@R1 CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus (2 primes)
........+++++
....................................................+++++
e is 65537 (0x010001)
[root@R1 CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:HB
Locality Name (eg, city) [Default City]:WH
Organization Name (eg, company) [Default Company Ltd]:ha
Organizational Unit Name (eg, section) []:ha
Common Name (eg, your name or your server's hostname) []:hanao.com
Email Address []:1@1.com
[root@R1 CA]# mkdir certs newcerts crl
[root@R1 CA]# touch index.txt && echo 01 > serial
[root@R1 CA]# (umask 077;openssl genrsa -out httpd.key 2048)
Generating RSA private key, 2048 bit long modulus (2 primes)
..............................+++++
.................+++++
e is 65537 (0x010001)
[root@R1 CA]# openssl req -new -key httpd.key -days 365 -out httpd.csr
Ignoring -days; not generating a certificate
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:HB
Locality Name (eg, city) [Default City]:WH
Organization Name (eg, company) [Default Company Ltd]:ha
Organizational Unit Name (eg, section) []:ha
Common Name (eg, your name or your server's hostname) []:hanao.com
Email Address []:1@1.com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@R1 CA]# openssl ca -in httpd.csr -out httpd.crt -days 365
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Jun 12 06:14:14 2021 GMT
Not After : Jun 12 06:14:14 2022 GMT
Subject:
countryName = CN
stateOrProvinceName = HB
organizationName = ha
organizationalUnitName = ha
commonName = hanao.com
emailAddress = 1@1.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
B1:16:DF:69:E3:BE:CC:DA:F7:46:6F:98:33:4F:2F:52:87:47:D6:FC
X509v3 Authority Key Identifier:
keyid:DC:5C:6E:05:C8:3A:37:ED:74:81:D8:C1:66:52:2A:AB:F0:80:8A:78
Certificate is to be certified until Jun 12 06:14:14 2022 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@R1 CA]# mv httpd.key httpd.crt /etc/httpd/
[root@R1 CA]# scp /etc/httpd/httpd.key 192.168.247.100:/etc/httpd/
The authenticity of host '192.168.247.100 (192.168.247.100)' can't be established.
ECDSA key fingerprint is SHA256:IDtC7Ky/bDYQyep8Z+pxtsm/gAoqEexlJEkkZlNwMLc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.247.100' (ECDSA) to the list of known hosts.
root@192.168.247.100's password:
httpd.key 100% 1675 614.9KB/s 00:00
[root@R1 CA]# scp /etc/httpd/httpd.crt 192.168.247.100:/etc/httpd/
root@192.168.247.100's password:
httpd.crt 100% 4495 1.4MB/s 00:00
[root@R1 ~]# echo "R1" >/usr/local/apache/htdocs/index.html
[root@R2 ~]# echo "R2" >/usr/local/apache/htdocs/index.html
关闭防火墙和selinux
[root@R1 ~]# systemctl disable --now firewalld
[root@R1 ~]# setenforce 0
[root@R2 ~]# systemctl disable --now firewalld
[root@R2 ~]# setenforce 0
[root@lb ~]# systemctl disable --now firewalld
[root@lb ~]# setenforce 0
在调度器上开启 IP转发 功能
[root@lb ~]# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
[root@lb ~]# sysctl -p
net.ipv4.ip_forward = 1
在调度器上添加并保持规则
[root@lb ~]# ipvsadm -A -t 192.168.159.128:80 -s rr
[root@lb ~]# ipvsadm -a -t 192.168.159.128:80 -r 192.168.247.211:80 -m
[root@lb ~]# ipvsadm -a -t 192.168.159.128:80 -r 192.168.247.100:80 -m
[root@lb ~]# ipvsadm -A -t 192.168.159.128:443 -s rr
[root@lb ~]# ipvsadm -a -t 192.168.159.128:443 -r 192.168.247.100:443 -m
[root@lb ~]# ipvsadm -a -t 192.168.159.128:443 -r 192.168.247.211:443 -m
[root@lb ~]# ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@lb ~]# cat /etc/sysconfig/ipvsadm
-A -t 192.168.159.128:80 -s rr
-a -t 192.168.159.128:80 -r 192.168.247.100:80 -m -w 1
-a -t 192.168.159.128:80 -r 192.168.247.211:80 -m -w 1
-A -t 192.168.159.128:443 -s rr
-a -t 192.168.159.128:443 -r 192.168.247.100:443 -m -w 1
-a -t 192.168.159.128:443 -r 192.168.247.211:443 -m -w 1
设置开机自动读取规则
[root@lb ~]# echo "ipvsadm -R < /etc/sysconfig/ipvsadm" >>/etc/rc.d/rc.local
将RIP的网关都指向DIP
[root@R2 apache]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.247.100
PREFIX=24
GATEWAY=192.168.247.214
DNS1=114.114.114.114
[root@R1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.247.211
PREFIX=24
GATEWAY=192.168.247.214
DNS1=114.114.114.114
测试效果
dr模式
要求:
dr模型实现http和https两种负载均衡集群,注意,各RS都要提供同一个私钥和同一个证书。
IP类型 | IP地址 |
---|---|
VIP | 192.168.247.5 |
DIP | 192.168.247.214 |
R1IP | 192.168.247.211 |
R2IP | 192.168.247.100 |
安装配置httpd并且生成证书后使用同一个证书
\\\\安装配置过程略
[root@R1 apache]# mkdir /etc/pki/CA
[root@R1 apache]# cd /etc/pki/CA
[root@R1 CA]# mkdir private
[root@R1 CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus (2 primes)
........+++++
....................................................+++++
e is 65537 (0x010001)
[root@R1 CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:HB
Locality Name (eg, city) [Default City]:WH
Organization Name (eg, company) [Default Company Ltd]:ha
Organizational Unit Name (eg, section) []:ha
Common Name (eg, your name or your server's hostname) []:hanao.com
Email Address []:1@1.com
[root@R1 CA]# mkdir certs newcerts crl
[root@R1 CA]# touch index.txt && echo 01 > serial
[root@R1 CA]# (umask 077;openssl genrsa -out httpd.key 2048)
Generating RSA private key, 2048 bit long modulus (2 primes)
..............................+++++
.................+++++
e is 65537 (0x010001)
[root@R1 CA]# openssl req -new -key httpd.key -days 365 -out httpd.csr
Ignoring -days; not generating a certificate
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:HB
Locality Name (eg, city) [Default City]:WH
Organization Name (eg, company) [Default Company Ltd]:ha
Organizational Unit Name (eg, section) []:ha
Common Name (eg, your name or your server's hostname) []:hanao.com
Email Address []:1@1.com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@R1 CA]# openssl ca -in httpd.csr -out httpd.crt -days 365
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Jun 12 06:14:14 2021 GMT
Not After : Jun 12 06:14:14 2022 GMT
Subject:
countryName = CN
stateOrProvinceName = HB
organizationName = ha
organizationalUnitName = ha
commonName = hanao.com
emailAddress = 1@1.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
B1:16:DF:69:E3:BE:CC:DA:F7:46:6F:98:33:4F:2F:52:87:47:D6:FC
X509v3 Authority Key Identifier:
keyid:DC:5C:6E:05:C8:3A:37:ED:74:81:D8:C1:66:52:2A:AB:F0:80:8A:78
Certificate is to be certified until Jun 12 06:14:14 2022 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@R1 CA]# mv httpd.key httpd.crt /etc/httpd/
[root@R1 CA]# scp /etc/httpd/httpd.key 192.168.247.100:/etc/httpd/
The authenticity of host '192.168.247.100 (192.168.247.100)' can't be established.
ECDSA key fingerprint is SHA256:IDtC7Ky/bDYQyep8Z+pxtsm/gAoqEexlJEkkZlNwMLc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.247.100' (ECDSA) to the list of known hosts.
root@192.168.247.100's password:
httpd.key 100% 1675 614.9KB/s 00:00
[root@R1 CA]# scp /etc/httpd/httpd.crt 192.168.247.100:/etc/httpd/
root@192.168.247.100's password:
httpd.crt 100% 4495 1.4MB/s 00:00
[root@R1 ~]# echo "<h1>R1</h1>" >/usr/local/apache/htdocs/index.html
[root@R2 ~]# echo "<h1>R2</h1>" >/usr/local/apache/htdocs/index.html
关闭防火墙和selinux
[root@R1 ~]# systemctl disable --now firewalld
[root@R1 ~]# setenforce 0
[root@R2 ~]# systemctl disable --now firewalld
[root@R2 ~]# setenforce 0
[root@lb ~]# systemctl disable --now firewalld
[root@lb ~]# setenforce 0
在调度器上配置VIP和DIP
[root@lb ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR0=192.168.247.214
PREFIX0=24
IPADDR1=192.168.247.5
PREFIX1=24
GATEWAY=192.168.247.2
DNS1=114.114.114.114
在RS上先关闭arp包的通告和响应然后配置RIP和VIP
[root@R1 httpd]# cat >> /etc/sysctl.conf <<EOF
> net.ipv4.conf.all.arp_ignore=1
> net.ipv4.conf.all.arp_announce=2
> EOF
[root@R1 httpd]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@R2 apache]# cat >> /etc/sysctl.conf <<EOF
> net.ipv4.conf.all.arp_ignore=1
> net.ipv4.conf.all.arp_announce=2
> EOF
[root@R2 apache]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@R1 httpd]# ifconfig lo:0 192.168.247.5/32 broadcast 192.168.247.5 up
[root@R1 httpd]# route add -host 192.168.247.5 dev lo:0
[root@R1 httpd]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.247.211
PREFIX=24
GATEWAY=192.168.247.2
DNS1=114.114.114.114
[root@R2 apache]# ifconfig lo:0 192.168.247.5/32 broadcast 192.168.247.5 up
[root@R2 apache]# route add -host 192.168.247.5 dev lo:0
[root@R2 apache]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=static
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.247.100
PREFIX=24
GATEWAY=192.168.247.2
DNS1=114.114.114.114
在调度器上添加、保存规则并且设置开机自动读取规则
[root@lb ~]# ipvsadm -A -t 192.168.247.5:80 -s rr
[root@lb ~]# ipvsadm -a -t 192.168.247.5:80 -r 192.168.247.211:80 -g
[root@lb ~]# ipvsadm -a -t 192.168.247.5:80 -r 192.168.247.100:80 -g
[root@lb ~]# ipvsadm -A -t 192.168.247.5:443 -s rr
[root@lb ~]# ipvsadm -a -t 192.168.247.5:443 -r 192.168.247.211:443 -g
[root@lb ~]# ipvsadm -a -t 192.168.247.5:443 -r 192.168.247.100:443 -g
[root@lb ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.247.5:80 rr
-> 192.168.247.100:80 Route 1 0 0
-> 192.168.247.211:80 Route 1 0 0
TCP 192.168.247.5:443 rr
-> 192.168.247.100:443 Route 1 0 0
-> 192.168.247.211:443 Route 1 0 0
ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@lb ~]# echo "ipvsadm -Sn > /etc/sysconfig/ipvsadm" >>/etc/rc.d/rc.local
测试效果
发送ARP包时,本地ARP表上R2的MAC绑定了192.168.247.5的IP于是访问不到R1,所以用的RS的IP访问
以上是关于Linux Virtual Server的主要内容,如果未能解决你的问题,请参考以下文章
使用piranha搭建Linux Virtual Server (LVS)集群环境
负载均衡:Linux Virtual Server(lvs)
lvs(linux virtual server)keepalived