部署k8s ssl集群实践14:work节点部署kube-proxy

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了部署k8s ssl集群实践14:work节点部署kube-proxy相关的知识,希望对你有一定的参考价值。

二进制文件前面已经下载分发好。

6.1
创建kube-proxy证书

创建证书签名请求

[[email protected] kube-proxy]# cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "SZ",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
[[email protected] kube-proxy]#

?CN:指定该证书的 User 为 system:kube-proxy ;
预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与
Role system:node-proxier 绑定,该 Role 授予了调用 kube-apiserver
Proxy 相关 API 的权限;
该证书只会被 kube-proxy 当做 client 证书使用,所以 hosts 字段为空;

生成证书和私钥

[[email protected] kube-proxy]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2018/08/30 21:58:31 [INFO] generate received request
2018/08/30 21:58:31 [INFO] received CSR
2018/08/30 21:58:31 [INFO] generating key: rsa-2048
2018/08/30 21:58:31 [INFO] encoded CSR
2018/08/30 21:58:31 [INFO] signed certificate with serial number 62542245638277052495817543993296923487092361674
2018/08/30 21:58:31 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected] kube-proxy]#

6.2
创建和分发kubeconfig文件

[[email protected] kube-proxy]# source /opt/k8s/bin/environment.sh
[[email protected] kube-proxy]# echo ${KUBE_APISERVER}
https://192.168.211.127:8443
[[email protected] kube-proxy]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.211.127:8443 --kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes" set.
[[email protected] kube-proxy]# kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.
[[email protected] kube-proxy]# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
Context "default" created.
[[email protected] kube-proxy]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".
[[email protected] kube-proxy]# ls
kube-proxy.csr? kube-proxy-csr.json? kube-proxy-key.pem? kube-proxy.kubeconfig? kube-proxy.pem
[[email protected] kube-proxy]#

分发

[[email protected] kube-proxy]# cp kube-proxy.kubeconfig /etc/kubernetes/
[[email protected] kube-proxy]# scp kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
kube-proxy.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100% 6219? ?? 6.1KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
kube-proxy.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100% 6219? ?? 6.1KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.kubeconfig [email protected]:/etc/kubernetes/
[email protected]‘s password:
kube-proxy.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100% 6219? ?? 6.1KB/s?? 00:00? ?
[[email protected] kube-proxy]#

6.3
创建kube-proxy配置文件

创建 kube-proxy config 文件模

[[email protected] kube-proxy]# echo ${CLUSTER_CIDR}
172.30.0.0/16
[[email protected] kube-proxy]# cat kube-proxy.config.yaml.template
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: ##NODE_IP##
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: ##NODE_IP##:10256
hostnameOverride: ##NODE_NAME##
kind: KubeProxyConfiguration
metricsBindAddress: ##NODE_IP##:10249
mode: "ipvs"
[[email protected] kube-proxy]#

bindAddress : 监听地址;
clientConnection.kubeconfig : 连接 apiserver 的 kubeconfig 文件;
clusterCIDR : kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量,
指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问
Service IP 的请求做 SNAT;
hostnameOverride : 参数值必须与 kubelet 的值一致,否则 kube-proxy 启动后会
找不到该 Node,从而不会创建任何 ipvs 规则;
mode : 使用 ipvs 模式;

分发

[[email protected] kube-proxy]# cp kube-proxy.config.yaml.template /etc/kubernetes/kube-proxy.config.yaml
[[email protected] kube-proxy]# scp kube-proxy.config.yaml.template [email protected]:/etc/kubernetes/kube-proxy.config.yaml
kube-proxy.config.yaml.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 315? ?? 0.3KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.config.yaml.template [email protected]:/etc/kubernetes/kube-proxy.config.yaml
kube-proxy.config.yaml.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 315? ?? 0.3KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.config.yaml.template [email protected]:/etc/kubernetes/kube-proxy.config.yaml
[email protected]‘s password:
kube-proxy.config.yaml.template ? ? ??

修改NODE_IP和NODE_NAME
所有节点的都根据节点的ip和hostname修改
参考下面的

[[email protected] kube-proxy]# cat /etc/kubernetes/kube-proxy.config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.211.128
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.211.128:10256
hostnameOverride: k8s-master1
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.211.128:10249
mode: "ipvs"
[[email protected] kube-proxy]#

6.4
创建和分发kube-proxy systemd unit 文件

[[email protected] kube-proxy]# cat kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy ? --config=/etc/kubernetes/kube-proxy.config.yaml ? --alsologtostderr=true ? --logtostderr=false ? --log-dir=/var/log/kubernetes ? --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
[[email protected] kube-proxy]#

注意
WorkingDirectory=/var/lib/kube-proxy
这个目录手动去创建

分发到所有节点

[[email protected] kube-proxy]# mkdir -p /var/lib/kube-proxy
[[email protected] kube-proxy]# ls
kube-proxy.config.yaml.template? kube-proxy-csr.json? kube-proxy.kubeconfig? kube-proxy.service
kube-proxy.csr? ? ? ? ? ? ? ? ?? kube-proxy-key.pem?? kube-proxy.pem
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]# scp kube-proxy.service [email protected]:/etc/systemd/system
[email protected]‘s password:
kube-proxy.service? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 450? ?? 0.4KB/s?? 00:00? ?
[[email protected] kube-proxy]#

6.5
启动服务

systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy

启动失败报错:

[[email protected] kubernetes]# cat kube-proxy.ERROR
Log file created at: 2018/08/30 22:26:09
Running on machine: k8s-master1
Binary: Built with gc go1.9.3 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0830 22:26:09.387614? ? 4255 helpers.go:119] error: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
goroutine 1 [running]:

文件格式问题,注意参考格式见下

[[email protected] kubernetes]# cat /etc/kubernetes/kube-proxy.config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.211.128
clientConnection:
? kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig ?
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.211.128:10256
hostnameOverride: k8s-master1
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.211.128:10249
mode: "ipvs"
[[email protected] kubernetes]#

kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig ? ? ## 注意这个前面的空格,没有就会报上面的错误

检查端口

[[email protected] kubernetes]# netstat -lnpt|grep kube-prox
tcp? ? ? ? 0? ? ? 0 192.168.211.128:10256?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 5349/kube-proxy? ??
tcp? ? ? ? 0? ? ? 0 192.168.211.128:10249?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 5349/kube-proxy? ??
[[email protected] kubernetes]#

查看ip路由规则

[[email protected] kubernetes]# /usr/sbin/ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
? -> RemoteAddress:Port? ? ? ? ?? Forward Weight ActiveConn InActConn
TCP? 10.254.0.1:443 rr persistent 10800
? -> 192.168.211.128:6443? ? ? ?? Masq? ? 1? ? ? 0? ? ? ? ? 0? ? ? ??
? -> 192.168.211.129:6443? ? ? ?? Masq? ? 1? ? ? 0? ? ? ? ? 0? ? ? ??
? -> 192.168.211.130:6443? ? ? ?? Masq? ? 1? ? ? 0? ? ? ? ? 0? ? ? ??
[[email protected] kubernetes]#

以上是关于部署k8s ssl集群实践14:work节点部署kube-proxy的主要内容,如果未能解决你的问题,请参考以下文章

部署k8s ssl集群实践10:work节点基础环境

部署k8s ssl集群实践13:work节点配置kubelet

部署k8s ssl集群实践6:配置高可用kube-apiserver组件ha+keepalived

部署k8s ssl集群实践4:部署etcd集群

部署k8s ssl集群实践3:部署kubectl命令工具行

部署k8s ssl集群实践1:基础环境准备