Kebernetes学习总结 Ingress

Posted 2021-01-09

虽然kubernetes集群内部署的pod、server都有自己的IP，但是却无法提供外网访问，以前可以通过监听NodePort的方式暴露服务，但是这种方式并不灵活，生产环境也不建议使用。Ingresss是kubernetes集群中的一个API资源对象，扮演边缘路由器(edge router)的角色，也可以理解为集群防火墙、集群网关，可以自定义路由规则来转发、管理、暴露服务(一组pod)，非常灵活，生产环境建议使用这种方式。另外LoadBlancer也可以暴露服务，不过这种方式需要向云平台申请负债均衡器；虽然目前很多云平台都支持，但是这种方式深度耦合了云平台。
Ingress相关组件：
1) Ingress controller
kubernetes中的controller有很多，比如CronJob、DeamonSet、Deployment、ReplicationSet、StatefulSet等，它们的作用就是监控集群的变化，使集群始终保持期望的最终状态(yml文件)。同理Ingress controller的作用就是实时感知Ingress路由规则集合的变化，再与Api Server交互，获取Service、Pod在集群中的IP等信息，然后发送给反向代理web服务器(ingress pods)，刷新其路由配置信息，这就是它的服务发现机制。
2) 反向代理web服务器(ingress pods) 负责转发外部请求到后端service，比如nginx、Apache、traefik等等
3) Ingress 定义路由规则集合
4) NodePort类型的service 负责将外部流量引入kubernetes cluster，然后由反向代理web服务器处理。

本篇总结中的示例大致如下：首先部署ingress controller及 ingress pods；然后利用NodePort类型的service将外部流最引入kubernetes cluster中；然后再定义后端deployment及后端service；然后定义ingress 转发规则集合；最后访问测试。
1) 部署ingress pods
[[email protected] ingress]# mkdir example
[[email protected] ingress]# cd example/
[[email protected] example]# for file in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml default-backend.yaml udp-services-configmap.yaml ; do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file ; done
[[email protected] example]# kubectl apply -f .
configmap/nginx-configuration created
deployment.extensions/default-http-backend created
service/default-http-backend created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
namespace/ingress-nginx configured
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
configmap/tcp-services created
configmap/udp-services created
deployment.extensions/nginx-ingress-controller created
[[email protected] example]#
[[email protected] example]# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-6586bc58b6-8rxtp 1/1 Running 0 59m
nginx-ingress-controller-6bd7c597cb-xdrcm 1/1 Running 0 59m
[[email protected] example]#

2) 在ingress-nginx 名称空间定义nodePort类型的service 引入外部流量，过程如下所示：

[[email protected] ingress]# cat nginx-frontend-ingress-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-frontend-ingress-svc
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    nodePort: 30080
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
    nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
[[email protected] ingress]#
[[email protected] ingress]# kubectl apply -f nginx-frontend-ingress-svc.yaml
service/nginx-frontend-ingress-svc created
[[email protected] ingress]# kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
default-http-backend   ClusterIP   10.102.71.36   <none>        80/TCP                       1h
nginx-frontend-ingress-svc     NodePort    10.103.52.44   <none>        80:30080/TCP,443:30443/TCP   11s
[[email protected] ingress]#

3) 下例中部署两个后端nginx-backend-deploy、tomcat-backend-deploy。
首先部署nginx-backend-deploy及nginx-backend-svc，过程如下：

[[email protected] ingress]# vim backend-nginx.yaml
[[email protected] ingress]# cat backend-nginx.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-backend-svc
  namespace: default
spec:
  selector:
    app: nginx-backend
    release: canary
  ports:
  - name: http
    targetPort: 80
    port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-backend-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-backend
      release: canary
  template:
    metadata:
      labels:
        app: nginx-backend
        release: canary
    spec:
      containers:
      - name: nginx-container
        image: nginx:1.15-alpine
        ports:
        - name: http
          containerPort: 80
[[email protected] ingress]#
[[email protected] ingress]# kubectl apply -f backend-nginx.yaml
service/nginx-backend-svc created
deployment.apps/nginx-backend-deploy created
[[email protected] ingress]# kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
nginx-backend-deploy-9c9b85bf4-6587j   1/1       Running   0          8s
nginx-backend-deploy-9c9b85bf4-k64tv   1/1       Running   0          8s
[[email protected] ingress]#
[[email protected] ingress]# kubectl get svc
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes          ClusterIP   10.96.0.1        <none>        443/TCP   3d
nginx-backend-svc   ClusterIP   10.110.150.187   <none>        80/TCP    1m
[[email protected] ingress]#

然后部署tomcat-backend-deploy及tomcat-backend-svc，过程如下：

[[email protected] ingress]# vim backend-tomcat.yaml
[[email protected] ingress]# cat backend-tomcat.yaml
apiVersion: v1
kind: Service
metadata:
  name: tomcat-backend-svc
  namespace: default
spec:
  selector:
    app: tomcat-backend
    release: canary
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: ajp
    port: 8009
    targetPort: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deploy
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat-backend
      release: canary
  template:
    metadata:
      labels:
        app: tomcat-backend
        release: canary
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.32-jre8-alpine
        ports:
        - name: http
          containerPort: 8080
        - name: ajp
          containerPort: 8009
[[email protected] ingress]# kubectl apply -f backend-tomcat.yaml
service/tomcat-backend-svc created
deployment.apps/tomcat-deploy created
[[email protected] ingress]# kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
nginx-backend-deploy-9c9b85bf4-6587j   1/1       Running   0          24m
nginx-backend-deploy-9c9b85bf4-k64tv   1/1       Running   0          24m
tomcat-deploy-7b4c97ff9f-fv8rx         1/1       Running   0          2m
tomcat-deploy-7b4c97ff9f-rj94r         1/1       Running   0          2m
[[email protected] ingress]# kubectl get svc
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
kubernetes           ClusterIP   10.96.0.1        <none>        443/TCP             3d
nginx-backend-svc    ClusterIP   10.110.150.187   <none>        80/TCP              24m
tomcat-backend-svc   ClusterIP   10.105.189.191   <none>        8080/TCP,8009/TCP   2m
[[email protected] ingress]#

4) 部署 ingress 转发规则集合
首先部署nginx-backend-svc的ingress 转发规则集合，过程如下所示:

[[email protected] ingress]# vim ingress-rules-ngx.yaml
[[email protected] ingress]# cat ingress-rules-ngx.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-rule-ngx
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: nginx.inspiry.com
    http:
      paths:
      - path:
        backend:
          serviceName: nginx-backend-svc
          servicePort: 80
[[email protected] ingress]#
[[email protected] ingress]# kubectl apply -f ingress-rules.yaml
ingress.extensions/ingress-rule-ngx created
[[email protected] ingress]# kubectl get ingress
NAME               HOSTS               ADDRESS   PORTS     AGE
ingress-rule-ngx   nginx.inspiry.com             80        9s
[[email protected] ingress]#
[[email protected] ingress]# kubectl describe ingress  ingress-rule-ngx
Name:             ingress-rule-ngx
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  nginx.inspiry.com
                        nginx-backend-svc:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-rule-ngx","namespace":"default"},"spec":{"rules":[{"host":"nginx.inspiry.com","http":{"paths":[{"backend":{"serviceName":"nginx-backend-svc","servicePort":80},"path":null}]}}]}}

  kubernetes.io/ingress.class:  nginx
Events:                         <none>
[[email protected] ingress]#

然后再部署tomcat-backend-svc的ingress 转发规则集合，过程如下所示：

[[email protected] ingress]# vim ingress-rules-tomcat.yaml
[[email protected] ingress]# cat ingress-rules-tomcat.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-rule-tomcat
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: tomcat.inspiry.cn
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat-backend-svc
          servicePort: 8080
[[email protected] ingress]# kubectl apply -f ingress-rules-tomcat.yaml
ingress.extensions/ingress-rule-tomcat created
[[email protected] ingress]# kubectl get ingress
NAME                  HOSTS               ADDRESS   PORTS     AGE
ingress-rule-ngx      nginx.inspiry.com             80        11m
ingress-rule-tomcat   tomcat.inspiry.cn             80        6s
[[email protected] ingress]#
[[email protected] ingress]# kubectl describe ingress ingress-rule-tomcat
Name:             ingress-rule-tomcat
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  tomcat.inspiry.cn
                        tomcat-backend-svc:8080 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-rule-tomcat","namespace":"default"},"spec":{"rules":[{"host":"tomcat.inspiry.cn","http":{"paths":[{"backend":{"serviceName":"tomcat-backend-svc","servicePort":8080},"path":null}]}}]}}

  kubernetes.io/ingress.class:  nginx
Events:                         <none>
[[email protected] ingress]#

5) 测试
在kubernetes cluster 外的任何一台主机上使用 curl http://nginx.inspiry.com:30080/ 或 curl http://tomcat.inspiry.cn:30080/ 即可访问相应nginx-backend 、tomcat-backend 的pods。
生产环境中还可以在k8s cluster外再创建一层 LB，将接收到的 nginx.inspiry.com、tomcat.inspiry.cn 域名分别转发到192.168.20.79、192.168.20.78、192.168.20.77为后端的upstream上。

观察ingress controller的内容如下：

[[email protected] ~]# kubectl get pods -n ingress-nginx
NAME                                        READY     STATUS    RESTARTS   AGE
default-http-backend-6586bc58b6-8rxtp       1/1       Running   0          1h
nginx-ingress-controller-6bd7c597cb-xdrcm   1/1       Running   0          1h
[[email protected] ~]# kubectl exec -n ingress-nginx -it nginx-ingress-controller-6bd7c597cb-xdrcm -- /bin/sh
$ ls
fastcgi.conf        geoip    mime.types      nginx.conf         scgi_params      uwsgi_params.default
fastcgi.conf.default    koi-utf  mime.types.default  nginx.conf.default     scgi_params.default  win-utf
fastcgi_params      koi-win  modsecurity         opentracing.json       template
fastcgi_params.default  lua  modules         owasp-modsecurity-crs  uwsgi_params
$ cat nginx.conf
...... (可仔细观察该文件的内容，发现相应的转发规则是随着 ingress rules的变化而变化的)