k8s ingress实现http/https7层和tcp四层代理

Posted 2021-02-17

ingress做为k8s集群的入口非常重要，能实现ingress功能的软件很多，可根据自身需求选择。本篇博客主要使用nginx官方提供的nginx-ingress完成了http/https7层代理和tcp四层代理的环境配置。

系统环境

1，k8s的版本为1.8.2
2，docker ce的版本为19.03.8-3
3，五台主机操作系统版本为centos7，kernel版本3.10.0-957
4，使用五台主机部署，3台master节点+2台work节点
5，Ingress测试域名t.myk8s.com

部署nginx和apache应用

使用Deployment部署nginx

nginx服务的manifest：

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.17
        livenessProbe: # 设置存活探针，如果检测失败则重启pod
          httpGet:
            path: /
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
          timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod
        readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求
          httpGet:
            path: /
            port: 80
            scheme: HTTP
        ports:
        - containerPort: 80 # 设置为pod中应用监听的端口

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  ports:
  - name: http
    port: 80 # service服务访问的端口
    protocol: TCP
    targetPort: 80 # 设置为pod中应用监听的端口
  selector:
    app: nginx

apache服务的manifests：

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: httpd
spec:
  selector:
    matchLabels:
      app: httpd
  replicas: 1 # tells deployment to run 2 pods
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:2.4
        livenessProbe: # 设置存活探针，如果检测失败则重启pod
          httpGet:
            path: /
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
          timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod
        readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求
          httpGet:
            path: /
            port: 80
            scheme: HTTP
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: httpd
  namespace: default
  labels:
    app: httpd
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpd

通过manifest配置k8s，使用--record选项记录命令和版本信息方便后续的升级或者回滚等操作：

# kubectl apply -f nginx-dp.yaml --record
# kubectl apply -f httpd-dp.yaml --record

查看应用状态：

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          3m15s   10.107.199.77    work2   <none>           <none>
nginx-5577575ddd-499wr   1/1     Running   0          46m     10.109.125.196   work3   <none>           <none>
nginx-5577575ddd-7zwmv   1/1     Running   0          48m     10.99.1.85       work1   <none>           <none>

查看services状态，通过services暴露出来的固定ip可以实现对应应用的访问：

# kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
httpd        ClusterIP   10.104.235.200   <none>        80/TCP    45m
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   6d5h
nginx        ClusterIP   10.111.5.31      <none>        80/TCP    57m

Ingress配置http访问应用

Ingress控制器的部署方式使用daemon-set+hostNetwork方式，且使用nginx官方提供的nginx-ingress控制器（https://blog.51cto.com/leejia/2495558）。
生成配置Ingress的manifests：

# vim http-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-ingress
spec:
  rules:
  - host: t.myk8s.com
    http:
      paths:
      - path: /index.html
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: httpd
          servicePort: 80
# kubectl apply -f http-ingress.yaml

查看状态：

# kubectl get ingress -o wide
NAME           CLASS    HOSTS         ADDRESS   PORTS   AGE
http-ingress   <none>   t.myk8s.com             80      102m

master集群中其中一台master的ip为172.18.2.175，在本地pc测试访问:

~ curl -H ‘Host: t.myk8s.com‘ 172.18.2.175
<html><body><h1>It works!</h1></body></html>

?  ~ curl -H ‘Host: t.myk8s.com‘ 172.18.2.175/index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Ingress配置https访问应用

我们使用自签名证书来实现https的访问：

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ingress.key -out ingress.crt -subj "/CN=t.myk8s.com/O=t.myk8s.com"

通过secret存储证书:

# kubectl create secret tls ingress-secret --key ingress.key --cert ingress.crt

# kubectl get secrets
NAME                  TYPE                                  DATA   AGE
ingress-secret        kubernetes.io/tls                     2      2s

注意如下ingress资源要和上面的secret资源在同一个namespace下面：

# vim nginx-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-ingress
spec:
  tls:
  - hosts:
    - t.myk8s.com
    secretName: ingress-secret
  rules:
  - host: t.myk8s.com
    http:
      paths:
      - path: /index.html
        backend:
          serviceName: nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: httpd
          servicePort: 80

使用本地pc测试访问，默认http访问也会强制转https：

?  ~ curl https://t.myk8s.com  -k
<html><body><h1>It works!</h1></body></html>
?  ~ curl http://t.myk8s.com  -I -k
HTTP/1.1 301 Moved Permanently
Server: nginx/1.17.10
Date: Thu, 21 May 2020 03:53:06 GMT
Content-Type: text/html
Content-Length: 170
Connection: keep-alive
Location: https://t.myk8s.com:443/

?  ~ curl https://t.myk8s.com/index.html  -k
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

可以通过配置nginx-ingress控制器实现http和https同时支持访问，config map的name以及namespace要和nginx -ingress控制器使用的相同：

# vim nginx-cf.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
data:
  ssl-redirect: "false"

# kubectl apply -f nginx-cf.yaml -n nginx-ingress
# kubectl get configmap -n nginx-ingress
NAME           DATA   AGE
nginx-config   1      6d19h

默认nginx-ingress控制器不提供tcp和udp代理功能，但是nginx是支持的，故也可以通过调整nginx-ingress控制器来提供tcp和udp代理服务

配置nginx-ingress做4层tcp代理

打开nginx官方的nginx-ingress仓库，然后修改mainifests，使nginx加载tcp和udp代理的相关配置（即在nginx-ingress.yaml的args下面添加global-configuration配置）：

# git clone https://github.com/nginxinc/kubernetes-ingress/
# cd kubernetes-ingress/deployments
# git checkout v1.7.0
# vim daemon-set/nginx-ingress.yaml
          - -global-configuration=$(POD_NAMESPACE)/nginx-configuration
# kubectl apply -f daemon-set/nginx-ingress.yaml

生成tcp代理相关的manifests：

# vim global-configuration.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
  name: nginx-configuration
  namespace: nginx-ingress
spec:
  listeners:
  - name: nginx-tcp
    port: 8000
    protocol: TCP

# vim global-configuration.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
  name: nginx-configuration
  namespace: nginx-ingress
spec:
  listeners:
  - name: nginx-tcp
    port: 8000
    protocol: TCP
[root@master1 manifests]# cat transport-server-passthrough.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: nginx-tcp
spec:
  listener:
    name: nginx-tcp
    protocol: TCP
  upstreams:
  - name: nginx-app
    service: nginx
    port: 80
  action:
    pass: nginx-app

# kubectl apply -f global-configuration.yaml
# kubectl apply -f transport-server-passthrough.yaml

在本地pc测试访问

?  ~ curl http://172.18.2.175:8000/index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

手动对nginx-ingress的后端nginx服务扩缩容和升级回滚

由2台水平扩容为3台，扩容期间服务无影响：

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   <none>           <none>
nginx-5577575ddd-pz59n   1/1     Running   0          18s     10.99.1.91       work1   <none>           <none>
nginx-5577575ddd-qsrb2   1/1     Running   0          18s     10.109.125.207   work3   <none>           <none>

# kubectl scale deployment nginx --replicas 3

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   <none>           <none>
nginx-5577575ddd-j4nq4   1/1     Running   0          13s     10.107.199.84    work2   <none>           <none>
nginx-5577575ddd-pz59n   1/1     Running   0          68s     10.99.1.91       work1   <none>           <none>
nginx-5577575ddd-qsrb2   1/1     Running   0          68s     10.109.125.207   work3   <none>           <none>

由3台水平缩容为1台，缩容期间服务无影响：

# kubectl scale deployment nginx --replicas 1

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP              NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77   work2   <none>           <none>
nginx-5577575ddd-pz59n   1/1     Running   0          102s    10.99.1.91      work1   <none>           <none>

平滑升级，先启动一个新版本的pod，然后流量切到新pod之后，再关闭老版本的pod

# kubectl set image deployment nginx nginx=nginx:1.18 --record

# kubectl rollout status deployment nginx
Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...

Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...
deployment "nginx" successfully rolled out

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   <none>           <none>
nginx-648f9b4559-m2b8m   1/1     Running   0          19s     10.109.125.208   work3   <none>           <none>

# kubectl get deployment nginx -o yaml|grep  ‘- image:‘
      - image: nginx:1.18

查看历史操作deployment的命令

# kubectl rollout history deployment nginx
deployment.apps/nginx
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-dp.yaml --record=true
2         kubectl set image deployment nginx nginx=nginx:1.18 --record=true

可以通过两种方式回滚：
第一种：回滚最近一次操作

# kubectl rollout undo deployment nginx
# kubectl get deployment nginx -o yaml|grep  ‘- image:‘
      - image: nginx:1.17

第二种：回滚到指定的命令版本，即执行对应的命令

# kubectl rollout undo deployment nginx --to-revision=1

参考文档

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentspec-v1-apps
https://kubernetes.github.io/ingress-nginx/user-guide/tls/
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/
https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/