kubernetes基于Ingress实现灰度发布

Posted 2022-01-13 不爱吃米饭。

Ingress的Canary注解

之前写了篇基于Service实现灰度发布的文章 链接，使用Service实现的话只能保证新旧版本负载均衡时的权重，功能比较单一。
使用Ingress除了可以使用权重控制负载均衡外，还可以实现基于http协议的header、cookie指定转发到指定版本的功能。这些功能是使用Ingress的Canary注解实现的。
详细信息可以参考官方文档： https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary

具体实现

准备一个域名

在主机上配置hosts，保证有一个可以访问到ingress-nginx-controller所在node的域名，这里我将www.test.com绑定到我的node上面。

[root@master ingress]# cat /etc/hosts |grep www.test.com
192.168.1.15 www.test.com

创建版本v1的服务

这里创建了一次部署，使用的镜像是nginx，同时创建来一个Service可以访问这次部署。

[root@master ingress]# cat v1.yaml
apiVersion: v1
kind: Service
metadata:
  name: grayscale-svc
  namespace: test
spec:
  selector:
    app: grayscale-v1
  type: ClusterIP  
  ports:
  - name: grayscale-test
    port: 80   
    targetPort: 80  
    protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grayscale-deploy-v1
  namespace: test
spec:
  selector:
    matchLabels:
      app: grayscale-v1
  replicas: 1
  template:
    metadata:
      labels:
        app: grayscale-v1
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

[root@master ingress]# kubectl apply -f v1.yaml
service/grayscale-svc created
deployment.apps/grayscale-deploy-v1 created

创建服务v1的Ingress资源

这里使用前缀路由，将域名www.test.com的请求转发给刚才创建的v1服务的Service。

[root@master ingress]# cat ingress-v1.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-v1
  namespace: test
spec:
  rules:
  - host: www.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grayscale-svc
            port:
              number: 80

[root@master ingress]# kubectl apply -f ingress-v1.yaml
ingress.networking.k8s.io/ingress-v1 created              

创建版本v2的服务

使用nginx镜像进行部署，并修改了默认页的内容，同时创建了一个Service将流量转发到这次部署的Pod上。

[root@master ingress]# cat v2.yaml 
apiVersion: v1
kind: Service
metadata:
  name: grayscale-svc-v2
  namespace: test
spec:
  selector:
    app: grayscale-v2
  type: ClusterIP
  ports:
  - name: grayscale-test
    port: 80
    targetPort: 80
    protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grayscale-deploy-v2
  namespace: test
spec:
  selector:
    matchLabels:
     app: grayscale-v2
  replicas: 1
  template:
    metadata:
      labels:
        app: grayscale-v2
    spec:
      volumes:
       - name: for-input
         emptyDir: 
      initContainers:
       - name: init
         image: alpine
         volumeMounts:
          - name: for-input
            mountPath: /test
         command: ["/bin/sh","-c","echo this is version 2 > /test/index.html;"]
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
          - name: for-input
            mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80

[root@master ingress]# kubectl apply -f v2.yaml 
service/grayscale-svc-v2 created
deployment.apps/grayscale-deploy-v2 created

创建服务v2的Ingress资源

使用canary注解实现灰度发布的功能

• nginx.ingress.kubernetes.io/canary：是否开启灰度发布的标识，设置为"true"标识开启
• nginx.ingress.kubernetes.io/canary-by-header：用于通知Ingress的请求头，当请求中这个请求头的值是always的时候会被Ingress转发到灰度版本，值是never的话保证不会路由到灰度版本。
• nginx.ingress.kubernetes.io/canary-weight：本次灰度所占的权重，优先级低于header和cookie方式的设置，也就是说在不指定header和cookie的时候才会生效。

[root@master ingress]# cat ingress-v2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-v2
  namespace: test
  annotations: 
   nginx.ingress.kubernetes.io/canary: "true"
   nginx.ingress.kubernetes.io/canary-by-header: "grayscale"
   nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  rules:
  - host: www.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grayscale-svc-v2
            port:
              number: 80
[root@master ingress]# kubectl apply -f ingress-v2.yaml 
ingress.networking.k8s.io/ingress-v2 created

最终效果测试

默认不指定header访问，可以看到会随机根据权重将流量打到两个版本的服务上

[root@master ingress]# curl www.test.com
this is version 2
[root@master ingress]# curl www.test.com
this is version 2
[root@master ingress]# curl www.test.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html  color-scheme: light dark; 
body  width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; 
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

指定header访问，会只把流量打给新版本服务

[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@kmaster ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2
[root@master ingress]# curl www.test.com -H "grayscale:always"
this is version 2