7pod应用监控及创建service移除不就绪后端pod
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了7pod应用监控及创建service移除不就绪后端pod相关的知识，希望对你有一定的参考价值。
关于pod
什么是Pod？
◼一个或多个容器的集合，因而也可称为容器集，但却是Kubernetes调度、部署和运行应用的原子单元
◼ 另外封装的内容：可被该组容器共享的存储资源、网络协议栈及容器的运行控制策略等
◼ 依赖于pause容器事先创建出可被各应用容器共享的基础环境，它默认共享Network、IPC和UTS名称空间给各
容器，PID名称空间也可以共享，但需要用户显式定义
Pod的组成形式有两种
◼ 单容器Pod：仅含有单个容器
◼ 多容器Pod：含有多个具有“超亲密”关系的容器
◆同一Pod内的所有容器都将运行于由Scheduler选定的同一个节点上
pod的phase（相位）和重启策略
pod的相位
容器的状态
Pod的重启策略：决定了容器终止后是否应该重启
◼ Always：无论何种exit code，都要重启容器
◼ OnFailure：仅在exit code为非0值（即错误退出）时才重启容器 默认是OnFailure
◼ Never：无论何种exit code，都不重启容器
应用监控
pod的健康状态监测机制
Pod支持的监测类型
◼ startup Probe #初始化时的监测机制，成功进入下一阶段的存活、就绪状态监测机制，不成功重启
◼ liveness Probe #存活状态监控 （表示当前进程状态健康与否） 失败容器会重启
◼ readiness Probe #就绪状态监控 （表示当前进程还能否提供服务） 被service从上游可用端点中移除
监测机制
◼ Exec Action：在容器内根据指定命令的结果状态码判定
◼ TcpSocket Action：根据相应TCP套接字连接建立状态判定 如：80端口
◼ HTTPGet Action：根据指定https/http服务URL的响应结果判定 如：返回200响应码
配置参数
◼ initialDelaySeconds #延迟启动
◼ periodSeconds #周期性监测，每隔多长时间检测一次
◼ timeoutSeconds #监测到没成功，超时时间多长，超时时间不能大过周期监测事件
◼ successThreshold #本来是失败的，连续成功几次，才认为是成功的
◼ failureThreshold #本来是成功的，连续失败几次，才认为是失败的
健康探针配置示例一：
同时定义了三种探针
◼ startup使用了Exec Action
◼ liveness和readiness使用HTTPGet Action
apiVersion: v1
kind: Pod
metadata:
name: pod-probe-demo
namespace: default
spec:
containers:
- name: demo
image: ikubernetes/demoapp:v1.0
imagePullPolicy: IfNotPresent
startupProbe:
exec:
command: [/bin/sh,-c,test,"$(curl -s 127.0.0.1/livez)"=="OK"]
initialDelaySeconds: 0 #容器起来立马开始探测
failureThreshold: 3 #失败预值是三次，成功预值没指，默认一次
periodSeconds: 2 #每次监测时长间隔是两秒钟一次
livenessProbe:
httpGet:
path: /livez
port: 80
scheme: HTTP #响应码200正常
initialDelaySeconds: 3 #容器起来延迟三秒钟进行探测
timeoutSeconds: 2 #每次探测超时时长2秒钟
readinessProbe: #其他没定义的使用默认值
httpGet:
path: /readyz
port: 80
scheme: HTTP
initialDelaySeconds: 15 #容器起来延迟15s进行探测
timeoutSeconds: 2 #每次探测超时时长2秒钟
restartPolicy: Always #如果startupProbe和livenessProbe探测失败，表示失败退出，通过重启来解决
测试效果
liveness
URL “/livez” 支持以POST方法为livez参数设定不同值，非OK值都以5xx响应码响应；
readiness
URL “/readyz” 支持以POST方法为readyz参数设定不同值，非OK值都以5xx响应码响应；
健康探针配置示例二：存活探针根据指定命令的结果状态码判定
vim liveness-exec-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: liveness-exec-demo
namespace: default
spec:
containers:
- name: demo
image: ikubernetes/demoapp:v1.0
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command: [/bin/sh, -c, [ "$(curl -s 127.0.0.1/livez)" == "OK" ]]
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 5
启动
[root@K8s-master01 chapter4]#kubectl apply -f liveness-exec-demo.yaml
pod/liveness-exec-demo created
查看
[root@K8s-master01 chapter4]#kubectl get pods
liveness-exec-demo 1/1 Running 0 28s
查看详细事件状态
[root@K8s-master01 chapter4]#kubectl describe pods liveness-exec-demo
查看Events
手动请求livez，返回结果OK
[root@K8s-master01 chapter4]#curl 10.244.4.15/livez
OK
人为的变成失败（让livez变成FAIL）
[root@K8s-master01 chapter4]#curl -XPOST -d livez=FAIL 10.244.4.15/livez
[root@K8s-master01 chapter4]#curl 10.244.4.15/livez
FAIL
再次查看详细事件状态
[root@K8s-master01 chapter4]#kubectl describe pods liveness-exec-demo
查看Events
Warning Unhealthy 2m51s (x3 over 3m1s) Liveness probe failed: 三次失败
Normal Killing Container demo failed liveness probe, will be restarted podkill掉重启
重启之后，刚才修改的值被重置成OK，再次访问
[root@K8s-master01 chapter4]#curl 10.244.4.15/livez
OK
查看pods，重启次数变为1
[root@K8s-master01 chapter4]#kubectl get pods
liveness-exec-demo 1/1 Running 1 (6m41s ago) 14m
健康探针配置示例三：存活探针根据相应TCP套接字连接建立状态判定
TCP套接字的监听端口状态的探测
[root@K8s-master01 chapter4]#cat liveness-tcpsocket-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: liveness-tcpsocket-demo
namespace: default
spec:
containers:
- name: demo
image: ikubernetes/demoapp:v1.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
securityContext:
capabilities:
add:
- NET_ADMIN
livenessProbe:
tcpSocket:
port: http
periodSeconds: 5
initialDelaySeconds: 5
运行
[root@K8s-master01 chapter4]#kubectl apply -f liveness-tcpsocket-demo.yaml
pod/liveness-tcpsocket-demo created
查看
[root@K8s-master01 chapter4]#kubectl get pods
NAME READY STATUS RESTARTS AGE
liveness-tcpsocket-demo 1/1 Running 0 19s
查看详细事件状态
[root@K8s-master01 chapter4]#kubectl describe pods liveness-tcpsocket-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 116s default-scheduler Successfully assigned default/liveness-tcpsocket-demo to k8s-node01
Normal Pulling 116s kubelet Pulling image "ikubernetes/demoapp:v1.0"
Normal Pulled 105s kubelet Successfully pulled image "ikubernetes/demoapp:v1.0" in 10.541507469s
Normal Created 105s kubelet Created container demo
Normal Started 105s kubelet Started container demo
状态正常
手动改错（交互式）
[root@K8s-master01 chapter4]#kubectl exec -it liveness-tcpsocket-demo -- /bin/sh
[root@liveness-tcpsocket-demo /]# iptables -vnL
Chain INPUT (policy ACCEPT 4 packets, 216 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 2 packets, 112 bytes)
pkts bytes target prot opt in out source destination
阻断端口80的访问
[root@liveness-tcpsocket-demo /]# iptables -A INPUT -p tcp --dport 80 -j REJECT
查看防火墙规则
[root@liveness-tcpsocket-demo /]# iptables -vnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3 180 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3 packets, 264 bytes)
pkts bytes target prot opt in out source destination
再次查看详细事件状态
[root@K8s-master01 ~]#kubectl describe pods liveness-tcpsocket-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned default/liveness-tcpsocket-demo to k8s-node01
Normal Pulling 20m kubelet Pulling image "ikubernetes/demoapp:v1.0"
Normal Pulled 20m kubelet Successfully pulled image "ikubernetes/demoapp:v1.0" in 10.541507469s
Warning Unhealthy 96s kubelet Liveness probe failed: dial tcp 10.244.3.19:80: connect: connection refused
Normal Created 61s (x4 over 20m) kubelet Created container demo
Normal Started 61s (x4 over 20m) kubelet Started container demo
Normal Pulled 61s (x3 over 2m41s) kubelet Container image "ikubernetes/demoapp:v1.0" already present on machine
Warning Unhealthy 41s (x11 over 3m21s) kubelet Liveness probe failed: dial tcp 10.244.3.19:80: i/o timeout
Normal Killing 41s (x4 over 3m11s) kubelet Container demo failed liveness probe, will be restarted
直接被重启，但规则不会被冲掉，需要手动清除iptables规则
[root@K8s-master01 chapter4]#kubectl exec -it liveness-tcpsocket-demo -- iptables -F
健康探针配置示例四：存活探针根据指定https/http服务URL的响应结果判定
[root@K8s-master01 chapter4]#cat liveness-httpget-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: liveness-tcpsocket-demo
namespace: default
spec:
containers:
- name: demo
image: ikubernetes/demoapp:v1.0
imagePullPolicy: IfNotPresent
livenessProbo:
httpGet"
path: /livez
port: 80
scheme: HTTP
initialDelaySeconds: 5
[root@K8s-master01 chapter4]#kubectl apply -f liveness-httpget-demo.yaml
[root@K8s-master01 chapter4]#kubectl get pods
手动出错
查看IP
[root@K8s-master01 chapter4]#kubectl get pods -o wide
[root@K8s-master01 chapter4]#curl -XPOST -d livez=FAIL 10.244.5.15/livez
健康探针配置示例五：就绪状态根据指定https/http服务URL的响应结果判定
[root@K8s-master01 chapter4]#cat readiness-httpget-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: readiness-httpget-demo
namespace: default
labels: #可用service进行关联
name: readiness-httpget-demo
spec:
containers:
- name: demo
image: ikubernetes/demoapp:v1.0
imagePullPolicy: IfNotPresent
readinessProbe: #livenessProbe失败会导致容器重启，
httpGet: #readinessProbe失败并不会导致容器被重启，只会导致就绪的状态不能转成就绪
path: /readyz 对应的service也不会把它添加成客户端
port: 80
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 2
periodSeconds: 5
failureThreshold: 3
restartPolicy: Always
创建pod
[root@K8s-master01 chapter4]# kubectl apply -f readiness-httpget-demo.yaml
pod/readiness-httpget-demo created
查看pod
[root@K8s-master01 chapter4]#kubectl get pods
NAME READY STATUS RESTARTS AGE
readiness-httpget-demo 0/1 Running 0 31s
持续为0，直到第一次监测并成功以后，才变成1。容器起来默认15s之后开始监测，直到第一次监测成功才就绪
0表示总有一个容器处于未就绪状态，1表示该pod有多少个容器
访问测试
[root@K8s-master01 chapter4]#curl -I 10.244.5.15/readyz
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 2
Server: Werkzeug/1.0.0 Python/3.8.2
Date: Thu, 10 Nov 2022 11:49:27 GMT
[root@K8s-master01 chapter4]#curl 10.244.5.15/readyz
OK
人为改成不就绪
[root@K8s-master01 chapter4]#curl -XPOST -d readyz=FAIL 10.244.5.15/readyz
[root@K8s-master01 chapter4]#curl -I 10.244.5.15/readyz
HTTP/1.0 507 INSUFFICIENT STORAGE
Content-Type: text/html; charset=utf-8
Content-Length: 4
Server: Werkzeug/1.0.0 Python/3.8.2
Date: Thu, 10 Nov 2022 11:51:32 GMT
[root@K8s-master01 chapter4]#curl 10.244.5.15/readyz
FAIL
查看pods
[root@K8s-master01 chapter4]#kubectl get pods
NAME READY STATUS RESTARTS AGE
readiness-httpget-demo 0/1 Running 0 9m9s
当前pod变成不就绪状态了，不会重启pod，只会把它从所属的service可用后端列表中移除掉
创建service并测试service移除不就绪状态的pod
先恢复就绪状态
[root@K8s-master01 chapter4]#curl -XPOST -d readyz=OK 10.244.5.15/readyz
创建service，把pod当作后端来使用
[root@K8s-master01 chapter4]#kubectl create service clusterip readiness-probe --tcp=80:80 --dry-run=client -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: readiness-probe
name: readiness-probe
spec:
ports:
- name: 80-80
port: 80
protocol: TCP
targetPort: 80
selector:
app: readiness-probe
type: ClusterIP
status:
loadBalancer:
把资源定义保存下来叫做readiness-probe-service.yaml并对其进行编辑
[root@K8s-master01 chapter4]#kubectl create service clusterip readiness-probe --tcp=80:80 --dry-run=client -o yaml > readiness-probe-service.yaml
编辑readiness-probe-service.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: readiness-probe
name: readiness-probe
spec:
ports:
- name: 80-80
port: 80
protocol: TCP
targetPort: 80
selector:
name: readiness-httpget-demo #定义标签选择器
type: ClusterIP
status:
loadBalancer:
把改好的readiness-probe创建出来
[root@K8s-master01 chapter4]#kubectl apply -f readiness-probe-service.yaml
service/readiness-probe created
查看service
[root@K8s-master01 chapter4]#kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
readiness-probe ClusterIP 10.102.54.32 <none> 80/TCP 25s
查看某servise的详细状态
[root@K8s-master01 chapter4]#kubectl describe service readiness-probe
Name: readiness-probe
Namespace: default
Labels: app=readiness-probe
Annotations: <none>
Selector: name=readiness-httpget-demo
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.102.54.32
IPs: 10.102.54.32
Port: 80-80 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.5.15:80 #其就是readiness-httpget-demo pod的ip地址
Session Affinity: None
Events: <none>
持续请求service的地址
[root@K8s-master01 chapter4]#while true; do curl 10.102.54.32; sleep .5; done
iKubernetes demoapp v1.0 !! ClientIP: 10.244.0.0, ServerName: readiness-httpget-demo, ServerIP: 10.244.5.15!
iKubernetes demoapp v1.0 !! ClientIP: 10.244.0.0, ServerName: readiness-httpget-demo, ServerIP: 10.244.5.15!
把pod手动改为不就绪
[root@K8s-master01 chapter4]#curl -XPOST -d readyz=FAIL 10.102.54.32/readyz
查看就绪状态变为0，pod就从可用后端列表中删除了，连接被拒绝
readiness-httpget-demo 0/1 Running 0 30m
iKubernetes demoapp v1.0 !! ClientIP: 10.244.0.0, ServerName: readiness-httpget-demo, ServerIP: 10.244.5.16!
iKubernetes demoapp v1.0 !! ClientIP: 10.244.0.0, ServerName: readiness-httpget-demo, ServerIP: 10.244.5.16!
curl: (7) Failed to connect to 10.101.85.48 port 80: 拒绝连接
curl: (7) Failed to connect to 10.101.85.48 port 80: 拒绝连接
修复好了，就变回来了
[root@K8s-master01 chapter4]#curl -XPOST -d readyz=OK 10.244.5.16/readyz
以上是关于7pod应用监控及创建service移除不就绪后端pod的主要内容，如果未能解决你的问题，请参考以下文章
sourceInsight工具移除不掉项目 source Insight Add and Remove Project Files
从服务器物理移除不相关的硬盘驱动器后，ZFS 报告健康状况下降