Kubernetes 集群主/工作节点
Posted
技术标签:
【中文标题】Kubernetes 集群主/工作节点【英文标题】：Kubernetes Cluster master/ Worker Nodes 【发布时间】：2021-06-22 02:16:04 【问题描述】：我正在尝试创建一个 Kubernetes 集群，这个集群将包含 3 个节点 主节点，我在其中安装和配置 kubeadm 、 kubelete，并在那里安装了我的系统（这是 laravel 开发的 Web 应用程序）， 工作节点毫无问题地加入主节点， 我将我的系统部署到 php-fpm pod 并创建了服务和水平 Pods Autoscaling 这是我的服务：
PHP LoadBalancer 10.108.218.232 <pending> 9000:30026/TCP 15h app=php
这是我的豆荚
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
qsinavphp-5b67996888-9clxp 1/1 Running 0 40m 10.244.0.4 taishan <none> <none>
qsinavphp-5b67996888-fnv7c 1/1 Running 0 43m 10.244.0.12 kubernetes-master <none> <none>
qsinavphp-5b67996888-gbtdw 1/1 Running 0 40m 10.244.0.3 taishan <none> <none>
qsinavphp-5b67996888-l6ghh 1/1 Running 0 33m 10.244.0.2 taishan <none> <none>
qsinavphp-5b67996888-ndbc8 1/1 Running 0 43m 10.244.0.11 kubernetes-master <none> <none>
qsinavphp-5b67996888-qgdbc 1/1 Running 0 43m 10.244.0.10 kubernetes-master <none> <none>
qsinavphp-5b67996888-t97qm 1/1 Running 0 43m 10.244.0.13 kubernetes-master <none> <none>
qsinavphp-5b67996888-wgrzb 1/1 Running 0 43m 10.244.0.14 kubernetes-master <none> <none>
worker nondes 是 taishan，master 是 Kubernetes-master。 这是我的 nginx 配置，它正在向 php 服务发送请求
server
listen 80;
listen 443 ssl;
server_name k8s.example.com;
root /var/www/html/Test/project-starter/public;
ssl_certificate "/var/www/cert/example.cer";
ssl_certificate_key "/var/www/cert/example.key";
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";
index index.php;
charset utf-8;
# if ($scheme = http)
# return 301 https://$server_name$request_uri;
#
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES25>
ssl_prefer_server_ciphers on;
location /
try_files $uri $uri/ /index.php?$query_string;
location = /favicon.ico access_log off; log_not_found off;
location = /robots.txt access_log off; log_not_found off;
error_page 404 /index.php;
location ~ [^/]\.php(/|$)
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_index index.php;
fastcgi_pass 10.108.218.232:9000;
include fastcgi_params;
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
location ~ /\.(?!well-known).*
deny all;
问题是我在工作节点上有 3 个 Pod，在主节点上有 5 个 Pod，但是没有请求进入工作节点，所有请求都发给主节点， 我的两个节点都处于就绪状态
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubernetes-master Ready control-plane,master 15h v1.20.4 10.14.0.58 <none> Ubuntu 20.04.1 LTS 5.4.0-70-generic docker://19.3.8
taishan Ready <none> 79m v1.20.5 10.14.2.66 <none> Ubuntu 20.04.1 LTS 5.4.0-42-generic docker://19.3.8
这是我的 kubectl describe nodes php 结果
Name: php
Namespace: default
Labels: tier=backend
Annotations: <none>
Selector: app=php
Type: LoadBalancer
IP Families: <none>
IP: 10.108.218.232
IPs: 10.108.218.232
Port: <unset> 9000/TCP
TargetPort: 9000/TCP
NodePort: <unset> 30026/TCP
Endpoints: 10.244.0.10:9000,10.244.0.11:9000,10.244.0.12:9000 + 7 more...
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Type 48m service-controller ClusterIP -> LoadBalancer
这是我用来创建部署的 yaml 文件
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: php
name: qsinavphp
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: php
strategy:
type: Recreate
template:
metadata:
labels:
app: php
spec:
containers:
- name: taishan-php-fpm
image: starking8b/taishanphp:last
imagePullPolicy: Never
ports:
- containerPort: 9000
volumeMounts:
- name: qsinav-nginx-config-volume
mountPath: /usr/local/etc/php-fpm.d/www.conf
subPath: www.conf
- name: qsinav-nginx-config-volume
mountPath: /usr/local/etc/php/conf.d/docker-php-memlimit.ini
subPath: php-memory
- name: qsinav-php-config-volume
mountPath: /usr/local/etc/php/php.ini-production
subPath: php.ini
- name: qsinav-php-config-volume
mountPath: /usr/local/etc/php/php.ini-development
subPath: php.ini
- name: qsinav-php-config-volume
mountPath: /usr/local/etc/php-fpm.conf
subPath: php-fpm.conf
- name: qsinav-www-storage
mountPath: /var/www/html/Test/qSinav-starter
resources:
limits:
cpu: 4048m
requests:
cpu: 4048m
restartPolicy: Always
serviceAccountName: ""
volumes:
- name: qsinav-www-storage
persistentVolumeClaim:
claimName: qsinav-pv-www-claim
- name: qsinav-nginx-config-volume
configMap:
name: qsinav-nginx-config
- name: qsinav-php-config-volume
configMap:
name: qsinav-php-config
这是我的服务 yaml 文件
apiVersion: v1
kind: Service
metadata:
name: php
labels:
tier: backend
spec:
selector:
app: php
ports:
- protocol: TCP
port: 9000
type: LoadBalancer
我不确定我的错误在哪里，所以请帮助解决这个问题
【问题讨论】：
因为你的服务没有正常启动。 > LoadBalancer 10.108.218.232实际上问题出在法兰绒网络上，它无法在节点之间建立连接，所以我通过安装现在工作正常的 weave 插件解决了这个问题 通过应用此命令
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
【讨论】：
【参考方案2】：这里我是从基本的baremetal k8安装中添加的
##### Creating ssh keys
From master node
`ssh-keygen`
Copy content in `~/.ssh/id_rsa.pub`
Login to other servers and paste this copied part into `~/.ssh/authorized_keys`
Follow these steps in all servers. Master and worker.
`sudo apt-get install python`
`sudo apt install python3-pip`
Adding Ansible
`sudo apt-add-repository ppa:ansible/ansible`
`sudo apt update`
`sudo apt-get install ansible -y`
[Reference](https://www.techrepublic.com/article/how-to-install-ansible-on-ubuntu-server-18-04/)
### Install Kubernetes
`sudo apt-get update`
`sudo apt-get install docker.io`
`sudo systemctl enable docker`
`curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add`
`sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"`
`sudo apt-get install kubeadm kubelet kubectl`
`sudo apt-mark hold kubeadm kubelet kubectl`
For more details please [refer](https://phoenixnap.com/kb/install-kubernetes-on-ubuntu)
### Installing Kubespray
`git clone https://github.com/kubernetes-incubator/kubespray.git`
`cd kubespray`
`sudo pip3 install -r requirements.txt`
`cp -rfp inventory/sample inventory/mycluster`
`declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)`
Please put your IP addresses here separated with a space.
`CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py $IPS[@]`
`ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml`
For none root user access
`ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml --extra-vars "ansible_sudo_pass=password"`
This will take around 15mins to run successfully. If `root` user ssh is not working properly, this will fail. Please check key sharing step again.
[10 Simple stepms](https://dzone.com/articles/kubespray-10-simple-steps-for-installing-a-product)
[Add a node to existing cluster](https://www.serverlab.ca/tutorials/containers/kubernetes/how-to-add-workers-to-kubernetes-clusters/)
[kubelet debug](https://***.com/questions/56463783/how-to-start-kubelet-service)
### Possible Errors
`kubectl get nodes`
> The connection to the server localhost:8080 was refused - did you specify the right host or port?
Perform followings as normal user (none root user)
`mkdir -p $HOME/.kube`
`sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config`
`sudo chown $(id -u):$(id -g) $HOME/.kube/config`
If you are in worker node, you will have to use `scp` to get `/etc/kubernetes/admin.conf` from master node. Master node may have this problem, if so please do these steps locally using normal user.
[Refer](https://www.edureka.co/community/18633/error-saying-connection-server-localhost-refused-specify)
## Installing MetalLB
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
[Official Installation guide](https://metallb.universe.tf/installation/)
### Configuring L2 config
sachith@master:~$ cat << EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.1.19-192.168.1.29 # Preferred IP range.
EOF
使用kubectl describe configmap config -n metallb-system验证安装成功
这将安装两个组件。
控制器：负责分配 IP。 演讲者：促进服务在 LB 中导航。【讨论】：
不，我没有使用 minikube，所以，我该怎么办？因为我没有使用它，所以我按照这个文档安装了 kubernetes kubernetes.io/docs/setup/production-environment/tools/kubeadm/… 实际上在公司有 3 台台山服务器，它们现在在本地，我正在尝试在这些服务器之间创建 Kubernetes 集群，所以我按照我发送给你的链接，为了创建其中一台服务器上的主节点，然后我在另一台服务器上应用了 join 命令 我们可以聊聊吗？所以也许你会对我想做的事情有很好的看法 您有什么建议吗？ 嗨，兄弟，感谢您的帮助和询问，是的，实际上问题出在法兰绒网络上，它无法在节点之间建立连接，所以我通过安装 weave 插件解决了它现在工作正常以上是关于Kubernetes 集群主/工作节点的主要内容，如果未能解决你的问题，请参考以下文章
如何用 Kubernetes 管理超过 2500 个节点的集群