部署k8s ssl集群实践13:work节点配置kubelet

Posted 2021-01-03

部署 kubelet 组件

这个是全新部署，四个节点都要部署

kublet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，
执行交互式命令，如 exec、run、logs 等。
kublet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的
资源使用情况。
为确保安全，本文档只开启接收 https 请求的安全端口，对请求进行认证和授权，拒绝
未授权的访问(如 apiserver、heapster)。

先把前面下载好的二进制执行文件分发到所有work节点

[[email protected] bin]# scp kubelet kubeadm kube-proxy? [email protected]:/opt/k8s/bin/
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 146MB?? 5.4MB/s?? 00:27? ?
kubeadm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 149MB?? 4.3MB/s?? 00:35? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%?? 49MB?? 3.3MB/s?? 00:15? ?
[[email protected] bin]# scp kubelet kubeadm kube-proxy? [email protected]:/opt/k8s/bin/
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 146MB? 48.6MB/s?? 00:03? ?
kubeadm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 149MB?? 7.1MB/s?? 00:21? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%?? 49MB? 24.5MB/s?? 00:02? ?
[[email protected] bin]# scp kubelet kubeadm kube-proxy? [email protected]:/opt/k8s/bin/
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 146MB?? 6.6MB/s?? 00:22? ?
kubeadm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 149MB?? 7.5MB/s?? 00:20? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%?? 49MB? 12.3MB/s?? 00:04? ?
[[email protected] bin]# scp kubelet kubeadm kube-proxy? [email protected]:/opt/k8s/bin/

创建 kubelet bootstrap kubeconfig 文件
分别按顺序跑:
k8s-master1
k8smaster2
k8smaster3

[[email protected] kubelet]# export BOOTSTRAP_TOKEN=$(kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:k8s-master1 --kubeconfig ~/.kube/config)

[[email protected] kubelet]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.211.127:8443 --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
Cluster "kubernetes" set.

[[email protected] kubelet]# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
User "kubelet-bootstrap" set.

[[email protected] kubelet]# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
Context "default" created.

[[email protected] kubelet]# kubectl config use-context default --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
Switched to context "default".

分发 bootstrap kubeconfig 文件到 worker 节点

[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
[email protected]‘s password:
Permission denied, please try again.
[email protected]‘s password:
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
[email protected]‘s password:
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
[email protected]‘s password:
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[[email protected] kubelet]#

创建和分发 kubelet 参数配置文件

[[email protected] kubelet]# cat kubelet.config.json.template
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/cert/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"address": "##NODE_IP##",
"port": 10250,
"readOnlyPort": 0,
"cgroupDriver": "cgroupfs",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"featureGates": {
"RotateKubeletClientCertificate": true,
"RotateKubeletServerCertificate": true
},
"clusterDomain": "${CLUSTER_DNS_DOMAIN}",
"clusterDNS": ["${CLUSTER_DNS_SVC_IP}"]
}
[[email protected] kubelet]#

address：API 监听地址，不能为 127.0.0.1，否则 kube-apiserver、heapster 等不
能调用 kubelet 的 API；
readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定；
authentication.anonymous.enabled：设置为 false，不允许匿名访问 10250 端口；
authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证
书认证；
authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；
对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将
被拒绝，提示 Unauthorized；
authroization.mode=Webhook：kubelet 使用 SubjectAcce***eview API 查询
kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)；
featureGates.RotateKubeletClientCertificate、
featureGates.RotateKubeletServerCertificate：自动 rotate 证书，证书的有效期取
决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数；
需要 root 账户运行；

分发并在各节点修改

[[email protected] kubelet]# scp kubelet.config.json.template [email protected]:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 704? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet.config.json.template [email protected]:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 704? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet.config.json.template [email protected]:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 704? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet.config.json.template [email protected]:/etc/kubernetes/kubelet.config.json

修好
##NODE_IP## 改成节点真实IP

${CLUSTER_DNS_DOMAIN}
${CLUSTER_DNS_SVC_IP}

这两个改成真实的参数，参考见下

[[email protected] kubelet]# echo ${CLUSTER_DNS_DOMAIN}
cluster.local.
[[email protected] kubelet]# echo ${CLUSTER_DNS_SVC_IP}
10.254.0.2
[[email protected] kubelet]#

创建和分发 kubelet systemd unit 文件

[[email protected] kubelet]# cat kubelet.service.template
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet ? --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig ? --cert-dir=/etc/kubernetes/cert ? --kubeconfig=/etc/kubernetes/kubelet.kubeconfig ? --config=/etc/kubernetes/kubelet.config.json ? --hostname-override=##nodename## ? --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest ? --allow-privileged=true ? --alsologtostderr=true ? --logtostderr=false ? --log-dir=/var/log/kubernetes ? --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
[[email protected] kubelet]#

注意的地方：

WorkingDirectory=/var/lib/kubelet ? ?##目录默认没有，手动去创建
--hostname-override=##nodename## ? ##nodename修改成在节点的名字

分发

[[email protected] kubelet]# scp kubelet.service.template [email protected]:/etc/systemd/system/kubelet.service
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet.service.template [email protected]:/etc/systemd/system/kubelet.service
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet.service.template [email protected]:/etc/systemd/system/kubelet.service
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]# scp kubelet.service.template [email protected]:/etc/systemd/system/kubelet.service
[email protected]‘s password:
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[[email protected] kubelet]#

分别去修改

--hostname-override=##nodename##

创建目录

mkdir -p /var/lib/kubelet && chown -R k8s /var/lib/kubelet

Bootstrap Token Auth 和授予权限
kublet 启动时查找配置的 --kubeletconfig 文件是否存在，如果不存在则使用 --bootstrapkubeconfig
向 kube-apiserver 发送证书签名请求 (CSR)。
kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证（事先使用 kubeadm 创建
的 token），认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为
system:bootstrappers，这一过程称为 Bootstrap Token Auth。

[[email protected] kubernetes]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" created

启动起来了

[[email protected] kubelet]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
?? Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
?? Active: active (running) since Thu 2018-08-30 04:46:51 EDT; 6s ago
? ?? Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 22228 (kubelet)
?? Memory: 10.3M
?? CGroup: /system.slice/kubelet.service
? ? ? ? ?? └─22228 /opt/k8s/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kub...

Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.374637?? 22228 feature_gate.go:226] feature gates: &{{} map[Ro...true]}
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.390859?? 22228 mount_linux.go:211] Detected OS with systemd
Aug 30 04:46:51 k8s-master1 kubelet[22228]: W0830 04:46:51.396470?? 22228 cni.go:171] Unable to update cni config: No net.../net.d
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406764?? 22228 server.go:376] Version: v1.10.4
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406831?? 22228 feature_gate.go:226] feature gates: &{{} map[Ro...true]}
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406960?? 22228 plugins.go:89] No cloud provider specified.
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406977?? 22228 server.go:492] No cloud provider specified: "" ...le: ""
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.407001?? 22228 bootstrap.go:58] Using bootstrap kubeconfig to ...g file
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.498673?? 22228 csr.go:105] csr for this node already exists, reusing
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.507675?? 22228 csr.go:113] csr for this node is still valid
Hint: Some lines were ellipsized, use -l to show in full.

kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个
CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥
和 --kubeletconfig 文件。
注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --
cluster-signing-key-file 参数，才会为 TLS Bootstrap 创建证书和私钥。

[[email protected] kubelet]# kubectl get csr
NAME? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? AGE? ? ?? REQUESTOR? ? ? ? ? ? ? ?? CONDITION
node-csr-4lLI6VjKHHWjZg4je3Ht3mgkyc0kSDALWgqyE6hJGLY?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-BX_rIIl3T80GWXCZqCQISgB2BWKXd_-QuD04IfXyvBU?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-BhI2aoEZzt8UlcSevQr8RQ9tY4ATbawpr3GklGbkdYI?? 54s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-CYy34cOnA7RStasf8ieh9ZF5crDLmTFbvDOZV7UaulI?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-G4fpzkI_gkD9R7LUh1fOHMBllMCTnIzfcWYUhcjbNLQ?? 28s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-GtBzeHjXzw0FThw7SwAQRq7_uWO_LmJutmAKOU19lpM?? 5m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-IMzMrDG99ht6FRazQyfq4XFmG0MU0iN7rFj87dJ_LO0?? 6m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-Ne6k_9kYNM5xZPzlMIMOiew6KYbCccgEoGEsD-A2mDI?? 59s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-S3MvbCy6G8vyMmZxPxHtSj7yXWsMKiTFhiEolNhbOcc?? 2m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-TobXYGLVUitHRfAJD3cy1uwLbD9xeLRqfVKRWcaqzG8?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-XCHccj91PEcvcgtoYIlUTVwjPntZ1QJ3x0FwaiKiaBQ?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-XWCrqdKkPfKiG20VpU8cn9N8ZRcOWlbfhPr8LMaW_PU?? 2m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-_Sp69LiFaATOGVn9fmAnOLHweAWwoVzeP9U0AxtsLPE?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-b71vB9tiCT7Ru5q6LQco_nb_hbIABmcDPmNi7fH7Vn8?? 1m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-czzY0kNjKg_6OAcU8m2dRzVt2KR9zY3FQ31t1QE3tXk?? 5m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-oA3SifuLsmgSMkZyIN9dJhE66iuMXCzciaLDWH3pl8E?? 57s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-pWeUuvcTZCGqq1sh0KufCNzziyCYfhh-KUB_WAC2lpw?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-uWlqsUKKcVd_HQIMYBHusZS8hJc9yAntfE7qpGNJnSg?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-wfcltVjp2D_nzjRu7PdnB74L4JlXTFWfaumRnMAEmDg?? 5m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-zaniEi7eNGTuzIherUJbNIdPAic1EnB1tKAAGvuzoAc?? 2m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-zggzAUVrryNXFp49lytoSZYe0qBYOd4Jz5Fa4WODeKQ?? 1m? ? ? ? system:bootstrap:m435c8?? Pending

approve kubelet CSR 请求
可以手动或自动 approve CSR 请求。推荐使用自动的方式，因为从 v1.8 版本开始，可
以自动轮转approve csr 后生成的证书。

[[email protected] kubelet]# kubectl certificate approve node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk
certificatesigningrequest.certificates.k8s.io "node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk" approved

[[email protected] kubelet]# kubectl describe csr node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk
Name:? ? ? ? ? ? ?? node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk
Labels:? ? ? ? ? ?? <none>
Annotations:? ? ? ? <none>
CreationTimestamp:? Thu, 30 Aug 2018 04:51:10 -0400
Requesting User:? ? system:bootstrap:m435c8
Status:? ? ? ? ? ?? Approved,Issued
Subject:
? ? ? ?? Common Name:? ? system:node:k8s-node3
? ? ? ?? Serial Number:?
? ? ? ?? Organization:?? system:nodes
Events:? <none>
[[email protected] kubelet]#

Requesting User ：请求 CSR 的用户，kube-apiserver 对它进行认证和授权；
Subject ：请求签名的证书信息；
证书的 CN 是 system:node:kube-node2， Organization 是 system:nodes，kubeapiserver
的 Node 授权模式会授予该证书的相关权限

自动 approve CSR 请求
创建三个 ClusterRoleBinding，分别用于自动 approve client、renew client、renew
server 证书：

[[email protected] kubelet]# cat csr-crb.yaml
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: auto-approve-csrs-for-group
subjects:
- kind: Group
? name: system:bootstrappers
? apiGroup: rbac.authorization.k8s.io
roleRef:
? kind: ClusterRole
? name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
? apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: node-client-cert-renewal
subjects:
- kind: Group
? name: system:nodes
? apiGroup: rbac.authorization.k8s.io
roleRef:
? kind: ClusterRole
? name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
? apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
? resources: ["certificatesigningrequests/selfnodeserver"]
? verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: node-server-cert-renewal
subjects:
- kind: Group
? name: system:nodes
? apiGroup: rbac.authorization.k8s.io
roleRef:
? kind: ClusterRole
? name: approve-node-server-renewal-csr
? apiGroup: rbac.authorization.k8s.io
[[email protected] kubelet]#

[[email protected] kubelet]# kubectl apply -f csr-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io "auto-approve-csrs-for-group" created
clusterrolebinding.rbac.authorization.k8s.io "node-client-cert-renewal" created
clusterrole.rbac.authorization.k8s.io "approve-node-server-renewal-csr" created
clusterrolebinding.rbac.authorization.k8s.io "node-server-cert-renewal" created

等待一段时间(1-10 分钟)，节点的 CSR 都被自动 approve：

[[email protected] kubelet]# kubectl get csr
NAME? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? AGE? ? ?? REQUESTOR? ? ? ? ? ? ? ?? CONDITION
csr-7685f? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 53s? ? ?? system:node:k8s-master2?? Approved,Issued
csr-8qkxl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1m? ? ? ? system:node:k8s-node3? ?? Approved,Issued
csr-n56tk? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 44s? ? ?? system:node:k8s-master1?? Approved,Issued
csr-p8h92? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28s? ? ?? system:node:k8s-master3?? Pending

nodes起来了

[[email protected] kubernetes]# kubectl get nodes
NAME? ? ? ? ? STATUS? ? ROLES? ?? AGE? ? ?? VERSION
k8s-master1?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
k8s-master2?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
k8s-master3?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
k8s-node3? ?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
[[email protected] kubernetes]#
[[email protected] kubernetes]# netstat -lnpt|grep kubelet
tcp? ? ? ? 0? ? ? 0 192.168.211.129:10250?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 20752/kubelet? ? ??
tcp? ? ? ? 0? ? ? 0 192.168.211.129:4194? ? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 20752/kubelet? ? ??
tcp? ? ? ? 0? ? ? 0 127.0.0.1:10248? ? ? ?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 20752/kubelet? ? ??
[[email protected] kubernetes]#

cadvisor 和 metrics
cadvisor 统计所在节点各容器的资源(CPU、内存、磁盘、网卡)使用情况，分别在自己
的 http web 页面(4194 端口)和 10250 以 promehteus metrics 的形式输出。
浏览器访问 http://192.168.211.128:4194/containers/ 可以查看到 cadvisor 的监控页面：

启动服务报错

[[email protected] kubernetes]# systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet?

F0830 04:05:24.413219?? 10947 server.go:233] failed to run Kubelet: cannot create certificate signing request: Post http://192.168.211.127/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.211.127:80: getsockopt: connection refused
goroutine 1 [running]:

这个报错是因为
kubelet-bootstrap.kubeconfig 文件的配置不对导致