Kubernetes没有在可用节点上传播pod
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Kubernetes没有在可用节点上传播pod相关的知识，希望对你有一定的参考价值。
我有一个GKE集群，其中包含一个大小为2的单节点池。当我添加第三个节点时，没有任何pod被分发到该第三个节点。
这是原始的2节点节点池：
$ kubectl get node
NAME STATUS ROLES AGE VERSION
gke-cluster0-pool-d59e9506-b7nb Ready <none> 13m v1.8.3-gke.0
gke-cluster0-pool-d59e9506-vp6t Ready <none> 18m v1.8.3-gke.0
以下是在原始节点池上运行的pod：
$ kubectl get po -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default attachment-proxy-659bdc84d-ckdq9 1/1 Running 0 10m 10.0.38.3 gke-cluster0-pool-d59e9506-vp6t
default elasticsearch-0 1/1 Running 0 4m 10.0.39.11 gke-cluster0-pool-d59e9506-b7nb
default front-webapp-646bc49675-86jj6 1/1 Running 0 10m 10.0.38.10 gke-cluster0-pool-d59e9506-vp6t
default kafka-0 1/1 Running 3 4m 10.0.39.9 gke-cluster0-pool-d59e9506-b7nb
default mailgun-http-98f8d997c-hhfdc 1/1 Running 0 4m 10.0.38.17 gke-cluster0-pool-d59e9506-vp6t
default stamps-5b6fc489bc-6xtqz 2/2 Running 3 10m 10.0.38.13 gke-cluster0-pool-d59e9506-vp6t
default user-elasticsearch-6b6dd7fc8-b55xx 1/1 Running 0 10m 10.0.38.4 gke-cluster0-pool-d59e9506-vp6t
default user-http-analytics-6bdd49bd98-p5pd5 1/1 Running 0 4m 10.0.39.8 gke-cluster0-pool-d59e9506-b7nb
default user-http-graphql-67884c678c-7dcdq 1/1 Running 0 4m 10.0.39.7 gke-cluster0-pool-d59e9506-b7nb
default user-service-5cbb8cfb4f-t6zhv 1/1 Running 0 4m 10.0.38.15 gke-cluster0-pool-d59e9506-vp6t
default user-streams-0 1/1 Running 0 4m 10.0.39.10 gke-cluster0-pool-d59e9506-b7nb
default user-streams-elasticsearch-c64b64d6f-2nrtl 1/1 Running 3 10m 10.0.38.6 gke-cluster0-pool-d59e9506-vp6t
default zookeeper-0 1/1 Running 0 4m 10.0.39.12 gke-cluster0-pool-d59e9506-b7nb
kube-lego kube-lego-7799f6b457-skkrc 1/1 Running 0 10m 10.0.38.5 gke-cluster0-pool-d59e9506-vp6t
kube-system event-exporter-v0.1.7-7cb7c5d4bf-vr52v 2/2 Running 0 10m 10.0.38.7 gke-cluster0-pool-d59e9506-vp6t
kube-system fluentd-gcp-v2.0.9-648rh 2/2 Running 0 14m 10.0.38.2 gke-cluster0-pool-d59e9506-vp6t
kube-system fluentd-gcp-v2.0.9-fqjz6 2/2 Running 0 9m 10.0.39.2 gke-cluster0-pool-d59e9506-b7nb
kube-system heapster-v1.4.3-6fc45b6cc4-8cl72 3/3 Running 0 4m 10.0.39.6 gke-cluster0-pool-d59e9506-b7nb
kube-system k8s-snapshots-5699c68696-h8r75 1/1 Running 0 4m 10.0.38.16 gke-cluster0-pool-d59e9506-vp6t
kube-system kube-dns-778977457c-b48w5 3/3 Running 0 4m 10.0.39.5 gke-cluster0-pool-d59e9506-b7nb
kube-system kube-dns-778977457c-sw672 3/3 Running 0 10m 10.0.38.9 gke-cluster0-pool-d59e9506-vp6t
kube-system kube-dns-autoscaler-7db47cb9b7-tjt4l 1/1 Running 0 10m 10.0.38.11 gke-cluster0-pool-d59e9506-vp6t
kube-system kube-proxy-gke-cluster0-pool-d59e9506-b7nb 1/1 Running 0 9m 10.128.0.4 gke-cluster0-pool-d59e9506-b7nb
kube-system kube-proxy-gke-cluster0-pool-d59e9506-vp6t 1/1 Running 0 14m 10.128.0.2 gke-cluster0-pool-d59e9506-vp6t
kube-system kubernetes-dashboard-76c679977c-mwqlv 1/1 Running 0 10m 10.0.38.8 gke-cluster0-pool-d59e9506-vp6t
kube-system l7-default-backend-6497bcdb4d-wkx28 1/1 Running 0 10m 10.0.38.12 gke-cluster0-pool-d59e9506-vp6t
kube-system nginx-ingress-controller-78d546664f-gf6mx 1/1 Running 0 4m 10.0.39.3 gke-cluster0-pool-d59e9506-b7nb
kube-system tiller-deploy-5458cb4cc-26x26 1/1 Running 0 4m 10.0.39.4 gke-cluster0-pool-d59e9506-b7nb
然后我将另一个节点添加到节点池：
gcloud container clusters resize cluster0 --node-pool pool --size 3
第三个是添加并准备好的：
NAME STATUS ROLES AGE VERSION
gke-cluster0-pool-d59e9506-1rzm Ready <none> 3m v1.8.3-gke.0
gke-cluster0-pool-d59e9506-b7nb Ready <none> 14m v1.8.3-gke.0
gke-cluster0-pool-d59e9506-vp6t Ready <none> 19m v1.8.3-gke.0
但是，除了那些属于DaemonSet的pod之外，没有任何pod被安排到添加的节点上：
$ kubectl get po -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default attachment-proxy-659bdc84d-ckdq9 1/1 Running 0 17m 10.0.38.3 gke-cluster0-pool-d59e9506-vp6t
default elasticsearch-0 1/1 Running 0 10m 10.0.39.11 gke-cluster0-pool-d59e9506-b7nb
default front-webapp-646bc49675-86jj6 1/1 Running 0 17m 10.0.38.10 gke-cluster0-pool-d59e9506-vp6t
default kafka-0 1/1 Running 3 11m 10.0.39.9 gke-cluster0-pool-d59e9506-b7nb
default mailgun-http-98f8d997c-hhfdc 1/1 Running 0 10m 10.0.38.17 gke-cluster0-pool-d59e9506-vp6t
default stamps-5b6fc489bc-6xtqz 2/2 Running 3 16m 10.0.38.13 gke-cluster0-pool-d59e9506-vp6t
default user-elasticsearch-6b6dd7fc8-b55xx 1/1 Running 0 17m 10.0.38.4 gke-cluster0-pool-d59e9506-vp6t
default user-http-analytics-6bdd49bd98-p5pd5 1/1 Running 0 10m 10.0.39.8 gke-cluster0-pool-d59e9506-b7nb
default user-http-graphql-67884c678c-7dcdq 1/1 Running 0 10m 10.0.39.7 gke-cluster0-pool-d59e9506-b7nb
default user-service-5cbb8cfb4f-t6zhv 1/1 Running 0 10m 10.0.38.15 gke-cluster0-pool-d59e9506-vp6t
default user-streams-0 1/1 Running 0 10m 10.0.39.10 gke-cluster0-pool-d59e9506-b7nb
default user-streams-elasticsearch-c64b64d6f-2nrtl 1/1 Running 3 17m 10.0.38.6 gke-cluster0-pool-d59e9506-vp6t
default zookeeper-0 1/1 Running 0 10m 10.0.39.12 gke-cluster0-pool-d59e9506-b7nb
kube-lego kube-lego-7799f6b457-skkrc 1/1 Running 0 17m 10.0.38.5 gke-cluster0-pool-d59e9506-vp6t
kube-system event-exporter-v0.1.7-7cb7c5d4bf-vr52v 2/2 Running 0 17m 10.0.38.7 gke-cluster0-pool-d59e9506-vp6t
kube-system fluentd-gcp-v2.0.9-648rh 2/2 Running 0 20m 10.0.38.2 gke-cluster0-pool-d59e9506-vp6t
kube-system fluentd-gcp-v2.0.9-8tb4n 2/2 Running 0 4m 10.0.40.2 gke-cluster0-pool-d59e9506-1rzm
kube-system fluentd-gcp-v2.0.9-fqjz6 2/2 Running 0 15m 10.0.39.2 gke-cluster0-pool-d59e9506-b7nb
kube-system heapster-v1.4.3-6fc45b6cc4-8cl72 3/3 Running 0 11m 10.0.39.6 gke-cluster0-pool-d59e9506-b7nb
kube-system k8s-snapshots-5699c68696-h8r75 1/1 Running 0 10m 10.0.38.16 gke-cluster0-pool-d59e9506-vp6t
kube-system kube-dns-778977457c-b48w5 3/3 Running 0 11m 10.0.39.5 gke-cluster0-pool-d59e9506-b7nb
kube-system kube-dns-778977457c-sw672 3/3 Running 0 17m 10.0.38.9 gke-cluster0-pool-d59e9506-vp6t
kube-system kube-dns-autoscaler-7db47cb9b7-tjt4l 1/1 Running 0 17m 10.0.38.11 gke-cluster0-pool-d59e9506-vp6t
kube-system kube-proxy-gke-cluster0-pool-d59e9506-1rzm 1/1 Running 0 4m 10.128.0.3 gke-cluster0-pool-d59e9506-1rzm
kube-system kube-proxy-gke-cluster0-pool-d59e9506-b7nb 1/1 Running 0 15m 10.128.0.4 gke-cluster0-pool-d59e9506-b7nb
kube-system kube-proxy-gke-cluster0-pool-d59e9506-vp6t 1/1 Running 0 20m 10.128.0.2 gke-cluster0-pool-d59e9506-vp6t
kube-system kubernetes-dashboard-76c679977c-mwqlv 1/1 Running 0 17m 10.0.38.8 gke-cluster0-pool-d59e9506-vp6t
kube-system l7-default-backend-6497bcdb4d-wkx28 1/1 Running 0 17m 10.0.38.12 gke-cluster0-pool-d59e9506-vp6t
kube-system nginx-ingress-controller-78d546664f-gf6mx 1/1 Running 0 11m 10.0.39.3 gke-cluster0-pool-d59e9506-b7nb
kube-system tiller-deploy-5458cb4cc-26x26 1/1 Running 0 11m 10.0.39.4 gke-cluster0-pool-d59e9506-b7nb
到底是怎么回事？为什么pod没有扩散到添加的节点上？我原以为pod会被分发到第三个节点。如何将工作负载扩展到第三个节点？
从技术上讲，就清单资源请求而言，我的整个应用程序适合一个节点。但是，当添加第二个节点时，应用程序将分发到第二个节点。所以我认为当我添加第三个节点时，pod也会被安排到该节点上。但这不是我所看到的。只有DaemonSets被安排到第三个节点上。我试过增长和缩小节点池无济于事。
更新
两个可抢占节点重新启动，现在所有pod都在一个节点上。这是怎么回事？增加资源请求是使它们分散的唯一方法吗？
答案
这是预期的行为。新的pod将被安排到空节点上，但不会自动移动运行的pod。 kubernetes调度程序通常对重新安排pod有保守，所以它没有理由不这样做。 Pod可以是有状态的（如数据库），因此kubernetes不想杀死并重新安排pod。
有一个正在开发的项目可以满足您的需求：https://github.com/kubernetes-incubator/descheduler我没有使用它，但它正在从社区积极开发。
另一答案
我在这里是一个完整的n00b，我正在学习Docker / Kubernetes，在阅读了你的问题后，听起来你的法定人数有问题。您是否尝试过最多5个节点？ （n / 2 + 1）Kubernetes和Docker Swarmkit都使用Raft一致性算法。您可能还想查看Raft。如果确实符合您的困境，此视频可能会对您有所帮助。它谈到了Raft和Quorum。 https://youtu.be/Qsv-q8WbIZY?t=2m58s
以上是关于Kubernetes没有在可用节点上传播pod的主要内容，如果未能解决你的问题，请参考以下文章
Kubernetes 集群部署 之 多Master节点 实现高可用
Kubernetes云原生实战01 Kubernetes高可用部署架构
Kubernetes云原生实战01 Kubernetes高可用部署架构