部署k8s ssl集群实践14:work节点部署kube-proxy

Posted 2021-01-03

二进制文件前面已经下载分发好。

6.1
创建kube-proxy证书

创建证书签名请求

[[email protected] kube-proxy]# cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "SZ",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
[[email protected] kube-proxy]#

?CN：指定该证书的 User 为 system:kube-proxy ；
预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与
Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver
Proxy 相关 API 的权限；
该证书只会被 kube-proxy 当做 client 证书使用，所以 hosts 字段为空；

生成证书和私钥

[[email protected] kube-proxy]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2018/08/30 21:58:31 [INFO] generate received request
2018/08/30 21:58:31 [INFO] received CSR
2018/08/30 21:58:31 [INFO] generating key: rsa-2048
2018/08/30 21:58:31 [INFO] encoded CSR
2018/08/30 21:58:31 [INFO] signed certificate with serial number 62542245638277052495817543993296923487092361674
2018/08/30 21:58:31 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected] kube-proxy]#

6.2
创建和分发kubeconfig文件

[[email protected] kube-proxy]# source /opt/k8s/bin/environment.sh
[[email protected] kube-proxy]# echo ${KUBE_APISERVER}
https://192.168.211.127:8443
[[email protected] kube-proxy]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.211.127:8443 --kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes" set.
[[email protected] kube-proxy]# kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.
[[email protected] kube-proxy]# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
Context "default" created.
[[email protected] kube-proxy]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".
[[email protected] kube-proxy]# ls
kube-proxy.csr? kube-proxy-csr.json? kube-proxy-key.pem? kube-proxy.kubeconfig? kube-proxy.pem
[[email protected] kube-proxy]#

分发

[[email protected] kube-proxy]# cp kube-proxy.kubeconfig /etc/kubernetes/
[[email protected] kube-proxy]# scp kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
kube-proxy.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100% 6219? ?? 6.1KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
kube-proxy.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100% 6219? ?? 6.1KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
[email protected]‘s password:
kube-proxy.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100% 6219? ?? 6.1KB/s?? 00:00? ?
[[email protected] kube-proxy]#

6.3
创建kube-proxy配置文件

创建 kube-proxy config 文件模

[[email protected] kube-proxy]# echo ${CLUSTER_CIDR}
172.30.0.0/16
[[email protected] kube-proxy]# cat kube-proxy.config.yaml.template
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: ##NODE_IP##
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: ##NODE_IP##:10256
hostnameOverride: ##NODE_NAME##
kind: KubeProxyConfiguration
metricsBindAddress: ##NODE_IP##:10249
mode: "ipvs"
[[email protected] kube-proxy]#

bindAddress : 监听地址；
clientConnection.kubeconfig : 连接 apiserver 的 kubeconfig 文件；
clusterCIDR : kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，
指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问
Service IP 的请求做 SNAT；
hostnameOverride : 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会
找不到该 Node，从而不会创建任何 ipvs 规则；
mode : 使用 ipvs 模式；

分发

[[email protected] kube-proxy]# cp kube-proxy.config.yaml.template /etc/kubernetes/kube-proxy.config.yaml
[[email protected] kube-proxy]# scp kube-proxy.config.yaml.template [email protected]:/etc/kubernetes/kube-proxy.config.yaml
kube-proxy.config.yaml.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 315? ?? 0.3KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.config.yaml.template [email protected]:/etc/kubernetes/kube-proxy.config.yaml
kube-proxy.config.yaml.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 315? ?? 0.3KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.config.yaml.template [email protected]:/etc/kubernetes/kube-proxy.config.yaml
[email protected]‘s password:
kube-proxy.config.yaml.template ? ? ??

修改NODE_IP和NODE_NAME
所有节点的都根据节点的ip和hostname修改
参考下面的

[[email protected] kube-proxy]# cat /etc/kubernetes/kube-proxy.config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.211.128
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.211.128:10256
hostnameOverride: k8s-master1
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.211.128:10249
mode: "ipvs"
[[email protected] kube-proxy]#

6.4
创建和分发kube-proxy systemd unit 文件

[[email protected] kube-proxy]# cat kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy ? --config=/etc/kubernetes/kube-proxy.config.yaml ? --alsologtostderr=true ? --logtostderr=false ? --log-dir=/var/log/kubernetes ? --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
[[email protected] kube-proxy]#

注意
WorkingDirectory=/var/lib/kube-proxy
这个目录手动去创建

分发到所有节点

[[email protected] kube-proxy]# mkdir -p /var/lib/kube-proxy
[[email protected] kube-proxy]# ls
kube-proxy.config.yaml.template? kube-proxy-csr.json? kube-proxy.kubeconfig? kube-proxy.service
kube-proxy.csr? ? ? ? ? ? ? ? ?? kube-proxy-key.pem?? kube-proxy.pem
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
[email protected]‘s password:
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]#

6.5
启动服务

systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy

启动失败报错:

[[email protected] kubernetes]# cat kube-proxy.ERROR
Log file created at: 2018/08/30 22:26:09
Running on machine: k8s-master1
Binary: Built with gc go1.9.3 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0830 22:26:09.387614? ? 4255 helpers.go:119] error: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
goroutine 1 [running]:

文件格式问题，注意参考格式见下

[[email protected] kubernetes]# cat /etc/kubernetes/kube-proxy.config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.211.128
clientConnection:
? kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig ?
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.211.128:10256
hostnameOverride: k8s-master1
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.211.128:10249
mode: "ipvs"
[[email protected] kubernetes]#

kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig ? ? ## 注意这个前面的空格，没有就会报上面的错误

检查端口

[[email protected] kubernetes]# netstat -lnpt|grep kube-prox
tcp? ? ? ? 0? ? ? 0 192.168.211.128:10256?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 5349/kube-proxy? ??
tcp? ? ? ? 0? ? ? 0 192.168.211.128:10249?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 5349/kube-proxy? ??
[[email protected] kubernetes]#

查看ip路由规则

[[email protected] kubernetes]# /usr/sbin/ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
? -> RemoteAddress:Port? ? ? ? ?? Forward Weight ActiveConn InActConn
TCP? 10.254.0.1:443 rr persistent 10800
? -> 192.168.211.128:6443? ? ? ?? Masq? ? 1? ? ? 0? ? ? ? ? 0? ? ? ??
? -> 192.168.211.129:6443? ? ? ?? Masq? ? 1? ? ? 0? ? ? ? ? 0? ? ? ??
? -> 192.168.211.130:6443? ? ? ?? Masq? ? 1? ? ? 0? ? ? ? ? 0? ? ? ??
[[email protected] kubernetes]#