kops 'protectKernelDefaults' 标志和 'EventRateLimit' 准入插件不起作用

Posted 2023-03-27

技术标签:

【中文标题】kops \'protectKernelDefaults\' 标志和 \'EventRateLimit\' 准入插件不起作用【英文标题】：kops 'protectKernelDefaults' flag and 'EventRateLimit' admission plugin not workingkops 'protectKernelDefaults' 标志和 'EventRateLimit' 准入插件不起作用 【发布时间】：2021-10-23 13:38:29 【问题描述】：

我正在尝试通过 kOps(1.21.0) 为 aws 上的自托管 Kubernetes 实施 Kubernetes 版本 1.21.4 的一些 CIS 安全基准建议。

但是，当我在 kubelet 配置和 EventRateLimit adminssion 插件 kube api 服务器配置中尝试 protectKernelDefaults:true 时，k8s 集群无法启动。我正在尝试使用这些设置创建一个新集群，而不是尝试更新任何现有的。

我尝试使用的 kops 集群 yaml 是

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: k8s.sample.com
spec:
  cloudLabels:
    team_number: "0"
    environment: "dev"
  api:
    loadBalancer:
      type: Internal
      additionalSecurityGroups:
        - sg-id
    crossZoneLoadBalancing: false
    dns:  
  authorization:
    rbac:  
  channel: stable
  cloudProvider: aws
  configBase: s3://state-data/k8s.sample.com
  etcdClusters:
    - cpuRequest: 200m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-eu-west-3a
          name: a
      memoryRequest: 100Mi
      name: main
      env:
        - name: ETCD_MANAGER_HOURLY_BACKUPS_RETENTION
          value: 2d
        - name: ETCD_MANAGER_DAILY_BACKUPS_RETENTION
          value: 1m
        - name: ETCD_LISTEN_METRICS_URLS
          value: http://0.0.0.0:8081
        - name: ETCD_METRICS
          value: basic
    - cpuRequest: 100m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-eu-west-3a
          name: a
      memoryRequest: 100Mi
      name: events
      env:
        - name: ETCD_MANAGER_HOURLY_BACKUPS_RETENTION
          value: 2d
        - name: ETCD_MANAGER_DAILY_BACKUPS_RETENTION
          value: 1m
        - name: ETCD_LISTEN_METRICS_URLS
          value: http://0.0.0.0:8081
        - name: ETCD_METRICS
          value: basic
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeControllerManager:
    enableProfiling: false
    logFormat: json
  kubeScheduler:
    logFormat: json
    enableProfiling: false
  kubelet:
    anonymousAuth: false
    logFormat: json
    protectKernelDefaults: true
    tlsCipherSuites: [ TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 ]
  kubeAPIServer:
    auditLogMaxAge: 7
    auditLogMaxBackups: 1
    auditLogMaxSize: 25
    auditLogPath: /var/log/kube-apiserver-audit.log
    auditPolicyFile: /srv/kubernetes/audit/policy-config.yaml
    enableProfiling: false
    logFormat: json
    enableAdmissionPlugins:
      - NamespaceLifecycle
      - LimitRanger
      - ServiceAccount
      - PersistentVolumeLabel
      - DefaultStorageClass
      - DefaultTolerationSeconds
      - MutatingAdmissionWebhook
      - ValidatingAdmissionWebhook
      - NodeRestriction
      - ResourceQuota
      - AlwaysPullImages
      - EventRateLimit
      - SecurityContextDeny
  fileAssets:
    - name: audit-policy-config
      path: /srv/kubernetes/audit/policy-config.yaml
      roles:
        - Master
      content: |
        apiVersion: audit.k8s.io/v1
        kind: Policy
        rules:
        - level: Metadata
  kubernetesVersion: 1.21.4
  masterPublicName: api.k8s.sample.com
  networkID: vpc-id
  sshKeyName: node_key
  networking:
    calico:
      crossSubnet: true
  nonMasqueradeCIDR: 100.64.0.0/10
  subnets:
    - id: subnet-id1
      name: sn_nodes_1
      type: Private
      zone: eu-west-3a
    - id: subnet-id2
      name: sn_nodes_2
      type: Private
      zone: eu-west-3a
    - id: subnet-id3
      name: sn_utility_1
      type: Utility
      zone: eu-west-3a
    - id: subnet-id4
      name: sn_utility_2
      type: Utility
      zone: eu-west-3a
  topology:
    dns:
      type: Private
    masters: private
    nodes: private
  additionalPolicies:
    node: |
      [
        
          "Effect": "Allow",
          "Action": [
            "kms:CreateGrant",
            "kms:Decrypt",
            "kms:DescribeKey",
            "kms:Encrypt",
            "kms:GenerateDataKey*",
            "kms:ReEncrypt*"
          ],
          "Resource": [
            "arn:aws:kms:region:xxxx:key/s3access"
          ]
        
      ]
    master: |
      [
        
          "Effect": "Allow",
          "Action": [
            "kms:CreateGrant",
            "kms:Decrypt",
            "kms:DescribeKey",
            "kms:Encrypt",
            "kms:GenerateDataKey*",
            "kms:ReEncrypt*"
          ],
          "Resource": [
            "arn:aws:kms:region:xxxx:key/s3access"
          ]
        
      ]

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.sample.com
  name: master-eu-west-3a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210720
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-3a
  role: Master
  subnets:
    - sn_nodes_1
    - sn_nodes_2
  detailedInstanceMonitoring: false
  additionalSecurityGroups:
    - sg-id

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.sample.com
  name: nodes-eu-west-3a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210720
  machineType: t3.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-3a
  role: Node
  subnets:
    - sn_nodes_1
    - sn_nodes_2
  detailedInstanceMonitoring: false
  additionalSecurityGroups:
    - sg-id

** 注意：我对上面的值进行了一些更改，以删除一些特定的细节**

我已经分别尝试了这些protectKernelDefaults & EventRateLimit 设置并尝试启动集群。在这些情况下它也不起作用。

当我尝试使用protectKernelDefaults 和ssh 到主节点并检查/var/log 目录kube-scheduler.log, kube-proxy.log, kube-controller-manager.log and kube-apiserver.log 是否为空时。

当它尝试 EventRateLimit 和 ssh 到主节点并检查 /var/log 目录时，api 服务器无法启动，并且所有其他日志文件都出现故障，表明无法连接到 api 服务器。 kube-apiserver.log 包含以下内容

Log file created at: 2021/08/23 05:35:51
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:35:54
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:36:11
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:36:32
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0823 05:36:32.654990       1 flags.go:59] FLAG: --add-dir-header="false"
Log file created at: 2021/08/23 05:37:15
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:38:44
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:41:35
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:46:47
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:51:57
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2021/08/23 05:56:59
Running on machine: ip-10-100-120-9
Binary: Built with gc go1.16.7 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg

任何指向正在发生的事情的指针都会有所帮助。提前致谢。

【问题讨论】：

对于最后一个，值得查看/var/log/containers中的日志。很可能日志没有正确刷新到/var/log/kube-apiserver.log。这通常发生在 apiserver 无法验证您传递的标志时。 【参考方案1】：

默认内核设置的问题是 kOps 中的一个错误。安装的没有设置 kubelet 期望的 sysctl 设置。

准入控制器的问题只是缺少准入控制器配置文件。

【讨论】：

以上是关于kops 'protectKernelDefaults' 标志和 'EventRateLimit' 准入插件不起作用的主要内容，如果未能解决你的问题，请参考以下文章

AWS 上的 Kops 集群超时

kops 'protectKernelDefaults' 标志和 'EventRateLimit' 准入插件不起作用

使用 kops 在 gcp 上验证集群时出现 TLS 握手超时错误

StreamNative 开源项目 KoP 入选全球技术雷达趋势报告

sh kops.sh

KoP 姝ｅ紡寮€婧愶細鍦?Apache Pulsar 涓婃敮鎸佸師鐢?Kafka 鍗忚