kubernetes高级调度方式

Posted 2022-06-19 alamisu

调度方式：

• 节点选择器： nodeSelector，nodeName
• 节点亲和调度： nodeAffinity
• pod亲和调度： podAffinity

♦ kubectl get nodes --show-labels  可以查看节点存在哪些标签

 

一、节点选择器

♦ nodeSelector选择的标签需要在node上存在，node才会被选中为pod创建的节点

下图中pod使用了nodeSelector选项过滤包含了gup=9527标签的node，然后pod就创建到了ks-node2上：

 

二、节点亲和调度

♦ pod.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution   软亲和性，尽量找满足需求的

apiVersion: v1
kind: Pod
metadata:
  name: pod-node-preferaffinity
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-node-preferaffinity
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: gpu
            operator: In
            values:
            - "952"
        weight: 10

 

♦ pod.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution　 硬亲和性，必须满足需求

root@k8s-master:/home/ubuntu/manifests/k8s/scheduler# cat nodeaffinity_require.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-node-requiredaffinity
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-node-requiredaffinity
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: In
            values:
            - "952"

 

 使用同样的matchExpressions来匹配label，required必须满足标签要求，否则会一直处于pending状态：

 

三、Pod亲和调度

♦ 使用podAffinity我们需要有一个标准来判断哪些节点属于亲和；使用topologyKey判定是否属于同一个位置

♦ pod亲和性与节点亲和性一样有prefered和required两种类型

♦ pod.spec.affinity.podAffinity.requiredDuringSchedulingIgnoredDuringExecution

root@k8s-master:/home/ubuntu/manifests/k8s/scheduler# cat pod_affinity_require.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-podrequiredaffinity-second    # 该pod会根据第一个部署的pod标签找到亲和的机器来部署pod
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: tomcat
    imagePullPolicy: IfNotPresent
    name: tomcat-podrequiredaffinity
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:　　# 这里匹配的是第一个pod的标签，所以key/values应该是想要pod部署的在一起的label
          - key: app
            operator: In
            values:
            - "ngx-demo"
        topologyKey: kubernetes.io/hostname      #使用的hostname来判断是否在同一位置，也可以使用其它自定义的标签

 

♦ pod.spec.affinity.podAffinity.preferredDuringSchedulingIgnoredDuringExecution

root@k8s-master:/home/ubuntu/manifests/k8s/scheduler# cat pod_affinity_prefer.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-podrequiredaffinity-second
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: tomcat
    imagePullPolicy: IfNotPresent
    name: tomcat-podrequiredaffinity
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - "ngx-demo"
          topologyKey: kubernetes.io/hostname      #可以使用自定义label
        weight: 60

♦ pod.spec.affinity.podAntiAffinity  反亲和性，创建的pod不会在一起，反亲和性也有prefered和required两种类型

apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-podrequired_anti_affinity-second
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: tomcat
    imagePullPolicy: IfNotPresent
    name: tomcat-podrequired_anti_affinity
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - "ngx-demo"
        topologyKey: kubernetes.io/hostname 

 

四、污点调度

taint --- 污点就是定义在节点上的键值属性

taintToleration --- 容忍度是定义在pod上的键值属性，是一个列表

• 标签   可定义在所有资源
• 注解   可定义在所有资源
• 污点   只能定义在节点上

 taint的effect定义对pod的排斥效果：

• NoSchedule：仅影响调度过程，对现存的pod不影响
• NoExecute： 既影响调度，也影响现存pod；不满足容忍的pod对象将被驱逐
• PreferNoSchedule：尽量不调度，实在没地方运行也可以

♦  如果一个节点上定义了污点，一个pod能否调上来，我们先去检查能够匹配的容忍度和污点；剩余不能被容忍的污点就去检查污点的effect，

　　如果是PreferNoSchedule，还是能调度；

　　如果是NoSchedule，还没调度的就不能被调度，已经调度完的没影响；

　　如果是NoExecute，完全不能被调度，如果此前有运行的pod，后来改了污点，pod将被驱逐

♦  kubectl taint --help  打污点标签

　　kubectl taint node k8s-node2 node-type=prod:NoSchedule　　给k8s-node2打上prod环境标签，effect为NoSchedule

　　kubectl taint node k8s-node2 node-type- 　　移除所有key为node-type的污点

　　kubectl taint node k8s-node2 node-type:NoSchedule-　　移除所有key为node-type中effect为NoSchedule的污点

 

 示例：我们首先使用反亲和性特点使得nginx和tomcat的pod不能在同一个node上被创建，然后再tomcat中设置容忍度满足k8s-master节点的污点（默认情况下，master节点是不能被创建普通pod的）

root@k8s-master:/home/ubuntu/manifests/k8s/scheduler# cat pod_toleration_master_taint.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-toleration-master-taint-second
  #namespace: develop
  labels:
    app: ngx-demo
    rel: stable
spec:
  containers:
  - image: tomcat
    imagePullPolicy: IfNotPresent
    name: tomcat-toleration-master-taint
  tolerations:　　# 设置容忍度 可以容忍master节点NoSchedule
  - effect: NoSchedule　　# 如果effect为空，则表示容忍所有的
    key: node-role.kubernetes.io/master
    operator: Exists
  affinity:
    podAntiAffinity:　　#反亲和性使得tomat的pod不能和nginx在同一个pod上
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - "ngx-demo"
        topologyKey: kubernetes.io/hostname 

 

最终结果使得tomcat的pod可以被调度到master节点：