由于CPU不足，Pod处于挂起状态

Posted 2021-05-04

在我的GCE Kubernetes集群上，我无法再创建pod。

Warning FailedScheduling    pod (www.caveconditions.com-f1be467e31c7b00bc983fbe5efdbb8eb-438ef) failed to fit in any node
fit failure on node (gke-prod-cluster-default-pool-b39c7f0c-c0ug): Insufficient CPU

查看该节点的已分配统计信息

Non-terminated Pods:        (8 in total)
  Namespace         Name                                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                                        ------------    ----------  --------------- -------------
  default           dev.caveconditions.com-n80z8                            100m (10%)  0 (0%)      0 (0%)      0 (0%)
  default           lamp-cnmrc                                  100m (10%)  0 (0%)      0 (0%)      0 (0%)
  default           mongo-2-h59ly                                   200m (20%)  0 (0%)      0 (0%)      0 (0%)
  default           www.caveconditions.com-tl7pa                            100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           fluentd-cloud-logging-gke-prod-cluster-default-pool-b39c7f0c-c0ug       100m (10%)  0 (0%)      200Mi (5%)  200Mi (5%)
  kube-system           kube-dns-v17-qp5la                              110m (11%)  110m (11%)  120Mi (3%)  220Mi (5%)
  kube-system           kube-proxy-gke-prod-cluster-default-pool-b39c7f0c-c0ug              100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kubernetes-dashboard-v1.1.0-orphh                       100m (10%)  100m (10%)  50Mi (1%)   50Mi (1%)
Allocated resources:
  (Total limits may be over 100%, i.e., overcommitted. More info: http://releases.k8s.io/HEAD/docs/user-guide/compute-resources.md)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  910m (91%)    210m (21%)  370Mi (9%)  470Mi (12%)

当然，我有91％的分配，不能再适合10％。但是不可能过度提交资源吗？

服务器的使用率约为CPU平均值的10％

如果我不能使用更多资源，那将是一种耻辱。

答案

是的，目前不支持过度使用。这是有计划的改进http://kubernetes.io/docs/user-guide/compute-resources。关于github的相关问题：https://github.com/kubernetes/kubernetes/issues/168

ps：理论上你可以定义自定义节点容量，但我不确定。

另一答案

我最近有同样的问题，经过一些研究，我发现GKE有一个默认的LimitRange，CPU请求限制设置为100m，这可以通过运行kubectl get limitrange -o=yaml来检查。它将显示如下内容：

apiVersion: v1
items:
- apiVersion: v1
  kind: LimitRange
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"v1","kind":"LimitRange","metadata":{"annotations":{},"name":"limits","namespace":"default"},"spec":{"limits":[{"defaultRequest":{"cpu":"100m"},"type":"Container"}]}}
    creationTimestamp: 2017-11-16T12:15:40Z
    name: limits
    namespace: default
    resourceVersion: "18741722"
    selfLink: /api/v1/namespaces/default/limitranges/limits
    uid: dcb25a24-cac7-11e7-a3d5-42010a8001b6
  spec:
    limits:
    - defaultRequest:
        cpu: 100m
      type: Container
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

此限制适用于每个容器。因此，例如，如果您有一个4核心节点，并假设您的每个POD将要创建2个容器，那么它将仅允许创建大约20个容器。

这里的“修复”是更改默认的LimitRange设置您自己的限制，然后删除旧的pod以便使用更新的值重新创建它们，或者在创建它们时直接设置pod限制。

一些阅读材料：

https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/#create-a-limitrange-and-a-pod

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run

https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits