LVS高可用集群搭建

Posted 2020-10-01

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了LVS高可用集群搭建相关的知识，希望对你有一定的参考价值。

最近公司重整架构,前端使用LVS做负债均衡,虽然之前也研究过集群,看过很多LVS原理和架构,但毕竟没有真正操作过，以下案例是在虚拟机中完成实验，记录一下，方便以后在服务器中实施搭建。

架构图如下：

前提介绍：本案例采用Centos7+Keepalived1.3.5+Tomcat9+Mysql5.6+Redis3.2.8+Rabbitmq3.6.10

集群实现的功能有：

1）：实现单点访问，利用keepalived的vip实现对不同的后端服务器进行访问；

2）：健康检查，利用keepalived的多种检查机制TCP、URL、HTTP_GET等；

3）：利用nginx实现负债均衡和静动分离；

4）：tomcat链接mysql做集群的web服务器，结合nginx静动分离，实现高可用集群；

5）：rabbitmq实现消息队列；

6）：redis实现数据库缓存服务器；

7）：mysql数据存储。（这里没做集群，业务不大）

一：集群搭建

前端LVS采用俩主互为备模式，nginx做后端web服务器负债均衡，mysql做数据存储使用主从模式切换。

服务器环境如下

nginx+LVS_master 192.168.23.14

nginx+LVS_backup 192.168.23.15

VIP 192.168.23.20

Tomcat1 192.168.23.16

2 192.168.23.17

3 192.168.23.18

rabbitmq 192.168.23.19

redis 192.168.23.21

Mysql 192.168.23.22

安装说明：前端俩台keepalived+nginx做前段服务器每个服务器各俩块网卡，有路由器的情况下，只需要路由器提供的外网ip即可，使用路由器映射功能，这样集群就变为全部是后端服务器，提高服务器的安全性。

1：安装前调试以及优化

安装centos7选择默认最小化

停止firewalld运行

systemctl stop firewalld.service

systemctl disable firewalld.service

内核优化vi/etc/sysctl.conf

    ##关闭ipv6
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    ##路由转发
    net.ipv4.ip_forward = 1
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0

防火墙安装iptables

yum -y installiptables-services iptables

修改selinux

vi/etc/selinux/config 修改selinux为disabled

重启reboot

2：安装master和backup

安装前准备

yum -y installopenssl openssl-devel popt popt-devel gcc* wget vim ipvsadm pcre pcre-devel

是否运行ipvsadm

查看内核模块是否存在lsmod|grep ip_vs

makdir /data

下载wgethttp://www.keepalived.org/software/keepalived-1.3.5.tar.gz

tar zxfkeepalived-1.3.5.tar.gz -C /data

./configure--prefix=/usr/local/keepalived-1.3.5 && make && make install

复制keepalived文件和脚本

cp/usr/local/keepalived-1.3.5/sbin/keepalived /usr/sbin/

cp/usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp/usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp/data/keepalived-1.3.5/./keepalived/etc/init.d/keepalived /etc/init.d/

chmod 755/etc/init.d/keepalived

mkdir/etc/keepalived/

设置外网的VIP

ifconfig ens33:0 ip 192.168.23.20 netmask 255.255.255.0 up

重启网卡

systemctl restart network.service

设置开启启动

chkconfig --add keepalived

chkconfig --level 345 keepalived on

设置/etc/keepalived/文件

! Configuration File for keepalived

global_defs {

router_id LVS1

}

vrrp_scriptchk_http_port { #编写nginx脚本在etc/keepalived/目录，下面会给出脚本

script "/etc/keepalived/nginx.sh" #监控服务脚步
interval 2 #检测时间间隔
weight 2
}

vrrp_instance VI_1 {

state MASTER #master服务器名称

interface ens33

virtual_router_id 51

priority 100 #优先级越大越高

advert_int 1

authentication {

auth_type PASS

auth_pass lmnginx #与backup的认证密码

}

virtual_ipaddress {

192.168.23.20

}

virtual_server 192.168.23.20 80 {

delay_loop 1

lb_algo rr

lb_kind DR

protocol TCP

real_server 192.168.23.14 80 {

weight 1

TCP_CHECK {

connect_timeout 3

retry 2

delay_before_retry 3

connect_port 80

}

real_server 192.168.23.15 80 {

weight 1

TCP_CHECK {

connect_timeout 3

retry 3

delay_before_retry 3

connect_port 80

}

keepalived配置文件backup设置

注：backup和master配置一致，这里不在重复介绍，主要介绍配置文件的差异

router_id LVS2

state BACKUP

interface ens37 （改为vip所在的网卡设置名）

priority 99

nginx监控脚本

#!/bin/sh

nginx=`ps -C nginx --no-header |wc -l`

if [ $nginx -eq 0 ];then

killall keepalived

/usr/local/keepalived/sbin/keepalived -D

else

exit 0

keepalived监控脚本。

#!/bin/bash

master=`ps -C keepalived --no-header |wc -l`

ip=`ifconfig ens33 |grep "inet" |awk -F " " ‘{print $2}‘`

error=/etc/keepalived/error.txt

cb=`df -hT |grep "root" |awk -F " " ‘{print $6}‘`

free=`free -m |grep Mem |awk -F " " ‘{print $3}‘`

cpu=`uptime |awk -F "," ‘{print $3,$4,$5}‘`

date=`date +%F-%r`

while true

if [ $master -eq 0 ]; then

echo -e "$date $ip master down\n已用磁盘空间$cb\n已用cpu格式为:1分钟,5分钟,15分钟.大于0.7负债率请联系管理员$cpu\n已用内存空间$free m" >$error

echo "####################################keepalived报错信息如下 #######################################" >>$error

/etc/init.d/keepalived status >>$error

cat $error |/bin/mail -s master [email protected] >/dev/null 2>&1

else

echo "$ip $date Check keepalived correctly" >$error

sleep 10

done

加入计划任务，每分钟执行一次脚本

附：mail设置

安装mail

yum -y install mailx

设置mail文件

vim /etc/mail.rc

set from=***@163.com（你的邮箱名，建议和账号一致） smtp=smtp.163.com （邮箱的stmp服务器）

set smtp-auth-user=***@163.com（你的邮箱账号） smtp-auth-password=password（你的邮箱密码，这个密码是stmp认证用户的密码，不懂可以百度） smtp-auth=login （确定登陆）

3：nginx编译安装

安装依赖包

yum install -y vim wget lsof gcc gcc-c++ bzip2 net-tools bind-utilszlib-devel openssl-devel libxml2-devel libxslt-devel perl-devel perl-ExtUtils-Embed
wget

创建nginx用户

groupadd nginx

useradd -s /sbin/nologin -g nginx -M nginx

安装Pcre
下载 wget http://www.pcre.org/pcre-8.41.tar.gz

cd /data

tar zxf pcre.tar.gz

./configure --prefix=/usr/local/pcre &&make && make install

安装openssl
下载 wget https://www.openssl.org/source/openssl-1.1.0f.tar.gz

cd /data

tar zxf openssl.tar.gz

./config --prefix=/usr/local/openssl &&make && make install

安装zlib
下载 wget www.zlib.net/zlib-1.2.11.tar.gz

cd /data

tar zxf zlib.tar.gz

./configure --prefix=/usr/local/zlib &&make && make install

mkdir -p/var/tmp/nginx/{client,proxy,fastcgi,uwsgi,scgi}

mkdir /var/run/nginx

mkdir /var/log/nginx

mkdir /etc/nginx

安装nginx
下载 wget http://nginx.org/download/nginx-1.13.3.tar.gz

./configure --prefix=/usr/local/nginx--sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf--error-log-path=/var/log/nginx/error.log--http-log-path=/var/log/nginx/access.log --lock-path=/var/lock/nginx.lock--user=nginx --group=nginx --with-http_ssl_module --with-http_dav_module--with-http_flv_module --with-http_realip_module --with-http_addition_module--with-http_xslt_module --with-http_stub_status_module --with-http_sub_module--with-http_random_index_module --with-http_degradation_module--with-http_secure_link_module --with-http_gzip_static_module--with-http_perl_module --with-http_v2_module --with-threads --with-stream--with-stream_ssl_module --with-pcre=/data/pcre-8.41--with-zlib=/data/zlib-1.2.11 --with-openssl=/data/openssl-1.1.0f--with-file-aio --with-mail --with-mail_ssl_module--http-client-body-temp-path=/var/tmp/nginx/client_body--http-proxy-temp-path=/var/tmp/nginx/proxy--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi--http-scgi-temp-path=/var/tmp/nginx/scgi --with-ld-opt="-Wl,-E"

make && make install

配置nginx
实现静动分离和负债均衡

user nginx;

worker_processes auto;

events {

worker_connections 65535;

use epoll;

multi_accept on;

}

http {

include mime.types;

include conf.d/*;

default_type application/octet-stream;

log_format main ‘$remote_addr - $remote_user [$time_local] "$request" ‘

‘$status $body_bytes_sent "$http_referer" ‘

‘"$http_user_agent" "$http_x_forwarded_for"‘;

sendfile on;

tcp_nopush on;

tcp_nodelay on;

charset UTF-8;

keepalive_timeout 65;

gzip on;

gzip_proxied any;

gzip_min_length 1000;

gzip_comp_level 4;

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

upstream www.limeng.com {

server 192.168.23.16;

server 192.168.23.17;

server 192.168.23.18;

}

server {

listen 80;

server_name www.limeng.com;

location / {

index index;

}

location ~ .*\.(js|css|ico|png|jpg|eot|svg|ttf|woff) {

index index.html index.htm;

proxy_pass http://www.limeng.com;

proxy_cache cache_one;

proxy_cache_valid 200 302 304 5d;

proxy_cache_valid any 5d;
proxy_cache_key ‘$host:$server_port$request_uri‘

add_header X-Cache ‘$upstream_cache_status from $host‘

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

expires 7d;

}

location ~ .*$ {

index index.jsp;

proxy_pass http://www.limeng.com;

}

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root html;

}

注：俩台nginx配置一样

nginx启动（脚本自行谷歌）

chmod 755 /etc/init.d/nginx

启动nginx

/etc/init.d/nginx start

Nginx安装完成！

4：后端web服务器Tomcat安装

jdk安装

安装前准备：

yum -y install gcc*wget vim

检查命令

java -version

java -version 有就卸载rpm -e --nodeps jdk

解压安装

wget http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1502180263_4da80e17ef2af74cdbd6c02a779d1db6

tar -zxvf jdk-8u144-linux-x64.tar.gz

移动目录

mv ./jdk1.8.0_121/usr/local/java/

最后再删除安装文件：

rm -rf jdk-8u144-linux-x64.gz

配置环境变量

vi /etc/profile加入以下变量

exportJAVA_HOME=/usr/local/java/

exportCLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

exportPATH=$PATH:$JAVA_HOME/bin

加载变量

source /etc/profile

tomcat安装

解压

wget http://mirror.bit.edu.cn/apache/tomcat/tomcat-9/v9.0.0.M22/bin/apache-tomcat-9.0.0.M18.tar.gz

tar -zxvf apache-tomcat-9.0.0.M18.tar.gz

解压完整后删除安装包

rm -rf apache-tomcat-9.0.0.M18.tar.gz

移动目录

mkdir /usr/local/tomcat/

mv ./apache-tomcat-9.0.0.M18//usr/local/tomcat/

首先我们来配置一下tomcat的环境变量和内存设置，进入tomcat的bin目录后通过vim命令打开catalina.sh文件，并在其中加入如下配置：

JAVA_OPTS="-Dfile.encoding=UTF-8 -server -Xms1024m -Xmx1024m -Xmn512m

-XX:SurvivorRatio=10

-XX:MaxTenuringThreshold=15

-XX:NewRatio=2

-XX:+DisableExplicitGC"

export TOMCAT_HOME=/tmp/tomcat/apache-tomcat-9.0.0.M18

export CATALINA_HOME=/tmp/tomcat/apache-tomcat-9.0.0.M18

export JRE_HOME=/tmp/java/jdk1.8.0_121

export JAVA_HOME=/tmp/java/jdk1.8.0_121

server.xml配置优化

maxThreads="1000"

minSpareThreads="100"

maxSpareThreads="1000"

acceptCount="1000"

URIEncoding="utf-8"

server="Neo App Srv 1.0"

connectionTimeout="20000"

redirectPort="8443" />

启动

./startup.sh

5：mysql安装配置

安装前准备

yum -y install makegcc cmake bison-devel ncurses-devel libaio libaio-devel per-Data-Dumpernet-tools gcc* wget vim autoconf

查看是否有mysql的rpm包

rpm -qa |grep mysql

rpm -e --nodeps mysql

查看是否有mariadb-libs的rpm包

rpm -qa |grep mariadb-libs

yum -y install mariadb-libs

解压文件
wget https://www.mysql.com/downloads/mysql-5.6.23.tar.gz

tar -zxf /data/mysql-5.6.23.tar.gz

编译安装

cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql-DMYSQL_DATADIR=/usr/local/mysql/data -DSYSCONFDIR=/etc-DWITH_MYISAM_STORAGE_ENGINE=1 -DWITH_INNOBASE_STORAGE_ENGINE=1-DWITH_MEMORY_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DMYSQL_UNIX_ADDR=/var/lib/mysql/mysql.sock-DMYSQL_TCP_PORT=3306 -DENABLED_LOCAL_INFILE=1-DWITH_PARTITION_STORAGE_ENGINE=1 -DEXTRA_CHARSETS=all -DDEFAULT_CHARSET=utf8-DDEFAULT_COLLATION=utf8_general_ci

make &&make install

建立用户和组

groupadd mysql

useradd mysql -s/sbin/nologin -g mysql

给权限

chown -Rmysql:mysql /usr/local/mysql/

mysql下的初始化

/usr/local/mysql/scripts/mysql_install_db--user=mysql --basedir=/usr/local/mysql/

拷贝配置文件my.cnf

cd support-files/

cp my-default.cnf /etc/my.cnf

配置mysql

[client]

port=3306

socket=/var/lib/mysql/mysql.sock

[mysql]

no-auto-rehash

[mysqld]

user=mysql

port=3306

basedir=/usr/local/mysql

datadir=/var/lib/mysql

open_files_limit=10240

back_log=600

max_connections = 3000

#max_connect_errors = 6000

external-locking = FALSE

max_allowed_packet = 32M

thread_cache_size = 300

query_cache_size = 64M

query_cache_limit = 4M

key_buffer_size = 2048M

read_rnd_buffer_size = 16M

bulk_insert_buffer_size = 64M

myisam_sort_buffer_size = 128M

innodb_lock_wait_timeout = 120

skip-name-resolve

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

# Settings user and group are ignored when systemd is used.

# If you need to run mysqld under a different user or group,

# customize your systemd unit file for mariadb according to the

# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]

log-error=/var/log/mysql_error.log

pid-file=/var/run/mysql/mysql.pid

# include all files from the config directory

!includedir /etc/my.cnf.d

添加变量

echo "exportPATH=$PATH:/usr/local/mysql/bin" >> /etc/profile

source /etc/profile

添加开机启动

cp/usr/local/mysql/./support-files/mysql.server /etc/rc.d/init.d/mysqld

chmod +x/etc/init.d/mysqld

chkconfig --addmysqld

chkconfig mysqld on

启动mysql

systemctl startmysqld.service

查看端口

netstat -anpt |grepmysql

编辑etc下的my.cnf文件

设置mysql密码

mysqladmin -uroot password

7：redis安装

解压安装
wget https://redis.io/download/redis-3.2.8.tar.gz

tar -zxf /data/redis-3.2.8.tar.gz

make

make install/usr/local/redis

执行完，进入目录/usr/local/redis

建立文件夹并拷贝

mkdir /etc/redis

cp -rf/usr/local/redis/redis.conf /etc/redis/6379.conf

编辑配置文件vim/etc/redis/6379.conf

daemonize yes

bind 192.168.23.21 127.0.0.1 (链接ip和本地）

logfile "/var/logs/redis.log"(自定义)

databases 16 (默认数据库数量）

dbfilename dump.rdb （默认存储名）

dir/tmp/redis/data/redisdata （默认数据存放目录）

requirepassyour password (自己设置redis的密码)

启动redis

redis_server /usr/local/redis/redis.conf

8：rabbitmq安装

安装前准备

yum -y install makegcc gcc-c++ kernel-devel m4 ncurses-devel openssl-devel

解压 zlang

tar -zxf /data/otp_src_19.3.tar.gz

cd otp_src_19.3

编译安装

./configure--prefix=/usr/local/erlang --with-ssl -enable-threads -enable-smmp-support-enable-kernel-poll --enable-hipe --without-javac && make &&make install