微服务网关 APISIX 群集配置指南

Posted 2022-11-12

1.前言

APISIX 通过 ETCD 存储数据，利用 ETCD 本身的功能实现群集管理和控制。有关 APISIX 的安装配置详情参考“​​APISIX 网关 CentOS 7 环境安装配置​​”。

前面文章描述的是单个 ETCD 节点的配置操作，本文通过配置 ETCD 3节点群集的方式实现 APISIX 的群集操作。

1.1.环境

操作系统：CentOS7，APISIX：2.10.1 ，Dashboard：2.9，ETCD：3.5

序号	IP	主机名称
1	192.168.112.98	etcd1
2	192.168.112.99	etcd2
3	192.168.112.100	etcd3

1.2.补充内容

ApiSix 的前置相关组件安装参考“​​nstall-dependencies.md​​”

curl https://raw.githubusercontent.com/apache/apisix/master/utils/install-dependencies.sh -sL | bash -

更详细的安装配置可以参考“​​how-to-build.md​​”

2.环境配置

2.1.配置主机

在三台机器上分别执行以下操作完成主机名称设置

cat >> /etc/hosts <<EOF 192.168.112.98 etcd1 192.168.112.99 etcd2 192.168.112.100 etcd3 EOF

2.2.安全配置

关闭系统的 Selinux 功能和操作系统防火墙

systemctl stop firewalld systemctl disable firewalld setenforce 0 vi /etc/selinux/config SELINUX=disabled

3.TLS 密钥和证书

使用TLS证书对通信进行加密，并开启即与CA根证书签名的双向认证

3.1.建立证书目录和数据目录

mkdir -p /etc/ssl/etcd/ssl/ mkdir -p /var/lib/etcd

3.2.创建根 CA 证书

可以通过 openssl 和 cfssl 来创建证书，这里通过 openssl 的方式创建

cd /etc/ssl/etcd/ssl openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -subj "/CN=apisix" -days 3650 -out ca.crt

3.3.创建 ETCD 证书和私钥

1、创建配置文件

cat > ./etcd-ca.conf <<EOF [ req ] default_bits = 2048 prompt = no default_md = rsa req_extensions = req_ext distinguished_name = dn [ dn ] C = CN ST = GuangDong L = DongGuan O = etcd OU = LingFang CN = etcd [ req_ext ] subjectAltName = @alt_names [ alt_names ] DNS.1 = localhost DNS.2 = etcd1 DNS.3 = etcd2 DNS.4 = etcd3 IP.1 = 127.0.0.1 IP.2 = 192.168.112.98 IP.3 = 192.168.112.99 IP.4 = 192.168.112.100 [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=keyEncipherment,dataEncipherment extendedKeyUsage=serverAuth,clientAuth subjectAltName=@alt_names EOF

2、生成密钥

openssl genrsa -out etcd.key 2048

3、生成证书签发请求(certificate signing request)

openssl req -new -key etcd.key -out etcd.csr -config etcd-ca.conf

4、生成证书

openssl x509 -req -in etcd.csr -CA /etc/ssl/etcd/ca.crt -CAkey /etc/ssl/etcd/ca.key -CAcreateserial -out etcd.crt -days 3650 -extensions v3_ext -extfile etcd-ca.conf

5、验证证书

openssl verify -CAfile ca.crt etcd.crt openssl x509 -text -noout -in /etc/ssl/etcd/ssl/etcd.crt

4.安装 ETCD

4.1.安装证书

将CA证书ca.crt，etcd证书etcd.crt和秘钥etcd.key，拷贝到各节点的/etc/ssl/etcd/ssl/目录中

4.2.下载并安装 ETCD

将其中的 etcd 和 etcdctl 两个可执行文件复制到各节点的/usr/local/bin目录

tar zxvf etcd-v3.5.1-linux-amd64.tar.gz cp ./etcd-v3.5.1-linux-amd64/etcd,etcdctl /usr/local/bin/

4.3.以 systemd 形式管理 etcd 服务

1、创建 ETCD 配置文件

mkdir -p /etc/etcd cat > /etc/etcd/etcd.conf <<EOF # [Member Flags] # ETCD_ELECTION_TIMEOUT=1000 # ETCD_HEARTBEAT_INTERVAL=100 # 指定etcd的数据目录 ETCD_NAME=etcd1 ETCD_DATA_DIR=/var/lib/etcd/ # [Cluster Flags] # ETCD_AUTO_COMPACTION_RETENTIO:N=0 ETCD_INITIAL_CLUSTER_STATE=new ETCD_ADVERTISE_CLIENT_URLS=https://192.168.112.98:2379 ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.112.98:2380 ETCD_LISTEN_CLIENT_URLS=https://192.168.112.98:2379,https://127.0.0.1:2379 ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster ETCD_LISTEN_PEER_URLS=https://192.168.112.98:2380 ETCD_INITIAL_CLUSTER=etcd1=https://192.168.112.98:2380,etcd2=https://192.168.112.99:2380,etcd3=https://192.168.112.100:2380 # [Proxy Flags] ETCD_PROXY=off # [Security flags] # ETCD_CLIENT_CERT_AUTH= # ETCD_PEER_CLIENT_CERT_AUTH= # 指定etcd的公钥证书和私钥 ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt ETCD_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt ETCD_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key # 指定etcd的Peers通信的公钥证书和私钥 ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key #ETCD_PEER_CLIENT_CERT_AUTH=true #ETCD_CLIENT_CERT_AUTH=true # [Profiling flags] # ETCD_METRICS= etcd_metrics EOF

2、创建 systemd 服务启用文件

cat > /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=etcd server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/usr/local/bin/etcd NotifyAccess=all Restart=always RestartSec=5s LimitNOFILE=40000 [Install] WantedBy=multi-user.target EOF

其中 ETCD2 和 ETCD3 主机配置文件中的内容需要对应修改

4.4.启动 ETCD

只有一个节点启动etcd服务后，etcd会去访问其他两个节点的2380端口，如果一定时间内无法访问，则会服务启动失败。所以应当迅速启动至少两个节点，服务才会成功启动。

systemctl daemon-reload systemctl enable etcd systemctl start etcd systemctl status etcd

4.5.检查集群是否健康

etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt \\ --key=/etc/ssl/etcd/ssl/etcd.key \\ --endpoints="https://192.168.112.98:2379,https://192.168.112.99:2379,https://192.168.112.100:2379" \\ endpoint health

curl --cacert /etc/ssl/etcd/ssl/ca.crt -i https://192.168.112.98:2379/version \\ --cert /etc/ssl/etcd/ssl/etcd.crt --key /etc/ssl/etcd/ssl/etcd.key

etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt \\ --key=/etc/ssl/etcd/ssl/etcd.key --endpoints="https://192.168.112.99:2379" \\ get / --prefix --keys-only openssl s_client -connect 192.168.112.98:2379 -key /etc/ssl/etcd/etcd.key \\ -cert /etc/ssl/etcd/etcd.crt -CAfile /etc/ssl/etcd/ssl/ca.crt

4.6.ETCD 参考指南

​​https://jimmysong.io/kubernetes-handbook/practice/etcd-cluster-installation.html​​

5.APISix 配置

主要涉及配置文件和 ETCD 连接部分的内容

5.1.config.yaml

apisix: admin_key: - name: admin key: 3cea9e6d29e70e98c09d29ca98bd452a43d057bb3e6a748f3d835492 role: admin port_admin: 9180 node_listen: - 80 allow_admin: - 127.0.0.0/24 ssl: enable: true listen: - 443 ssl_trusted_certificate: /etc/ssl/etcd/ssl/ca.crt #ssl_protocols: SSLv3 TLSv1.2 TLSv1.3 etcd: host: - "https://127.0.0.1:2379" - "https://192.168.112.98:2379" - "https://192.168.112.99:2379" - "https://192.168.112.100:2379" prefix: /apisix timeout: 30 resync_delay: 5 health_check_timeout: 10 tls: cert: /etc/ssl/etcd/ssl/etcd.crt key: /etc/ssl/etcd/ssl/etcd.key verify: true

5.2.conf.yaml

conf: listen: # host: 127.0.0.1 port: 9000 # ssl: # host: 127.0.0.1 # the address on which the `Manager API` should listen for HTTPS. # The default value is 0.0.0.0, if want to specify, please enable it. # port: 9001 # The port on which the `Manager API` should listen for HTTPS. # cert: "/tmp/cert/example.crt" # Path of your SSL cert. # key: "/tmp/cert/example.key" # Path of your SSL key. allow_list: # If we dont set any IP list, then any IP access is allowed by default. - 127.0.0.1 # The rules are checked in sequence until the first match is found. - ::1 - 0.0.0.0/0 etcd: endpoints: - 127.0.0.1:2379 - 192.168.112.98:2379 - 192.168.112.99:2379 - 192.168.112.100:2379 mtls: key_file: "/etc/ssl/etcd/ssl/etcd.key" # Path of your self-signed client side key cert_file: "/etc/ssl/etcd/ssl/etcd.crt" # Path of your self-signed client side cert ca_file: "/etc/ssl/etcd/ssl/ca.crt" # Path of your self-signed ca cert, the CA is used to sign callers certificates # prefix: /apisix # apisix configs prefix in etcd, /apisix by default log: error_log: level: warn # supports levels, lower to higher: debug, info, warn, error, panic, fatal file_path: logs/error.log # supports relative path, absolute path, standard output access_log: file_path: logs/access.log # supports relative path, absolute path, standard output max_cpu: 0 authentication: secret: secret # secret for jwt token generation. expire_time: 3600 # jwt token expire time, in second users: # yamllint enable rule:comments-indentation - username: admin # username and password for login `manager api` password: admin - username: user password: user plugins: # plugin list (sorted in alphabetical order) - api-breaker - authz-keycloak - basic-auth - batch-requests - consumer-restriction - cors # - dubbo-proxy - echo # - error-log-logger # - example-plugin - fault-injection - grpc-transcode - hmac-auth - http-logger - ip-restriction - jwt-auth - kafka-logger - key-auth - limit-conn - limit-count - limit-req # - log-rotate # - node-status - openid-connect - prometheus - proxy-cache - proxy-mirror - proxy-rewrite - redirect - referer-restriction - request-id - request-validation - response-rewrite - serverless-post-function - serverless-pre-function # - skywalking - sls-logger - syslog - tcp-logger - udp-logger - uri-blocker - wolf-rbac - zipkin - server-info - traffic-split

6.防火墙配置

受 ApiSix 在匹配 ETCD 群集 mtls 的限制，ETCD 群集不采用证书配置模式，考虑安全通过系统自带的防火墙功能实现对源 IP 地址对目标端口的访问控制。

systemctl start firewalld firewall-cmd --add-port=80/tcp --permanent firewall-cmd --add-port=443/tcp --permanent firewall-cmd --add-port=9000/tcp --permanent firewall-cmd --add-port=22/tcp --permanent firewall-cmd --list-rich-rules firewall-cmd --permanent --add-rich-rule=rule family="ipv4" source address="192.168.112.100" port port="2379" protocol=tcp accept firewall-cmd --permanent --add-rich-rule=rule family="ipv4" source address="192.168.112.100" port port="2380" protocol=tcp accept firewall-cmd --permanent --add-rich-rule=rule family="ipv4" source address="192.168.112.98" port port="2379" protocol=tcp accept firewall-cmd --permanent --add-rich-rule=rule family="ipv4" source address="192.168.112.98" port port="2380" protocol=tcp accept