混沌工程之ChaosBlade-Operator 使用之模拟 POD 丢包场景

Posted zuozewei

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了混沌工程之ChaosBlade-Operator 使用之模拟 POD 丢包场景相关的知识，希望对你有一定的参考价值。

文章目录

实现说明

ChaosBlade-Operator 是 ChaosBlade 的 K8s 平台场景的实现。将混沌实验通过 Kubernetes 标准的 CRD 方式定义，用户可以像定义 Deployment 或 StatefulSet 那样定义 ChaosBlade 实验，只要对 kubectl 和 Kubernetes 对象有所了解，就可以轻松的创建、更新和删除实验场景；同时也可以通过 chaosblade cli 工具来操作实验场景。

关于 ChaosBlade 你可以参考此文：

安装

使用 Helm 3 安装：

# 下载安装包
[root@s5 k8s]# wget -qO chaosblade-operator-1.2.0-v3.tgz https://chaosblade.oss-cn-hangzhou.aliyuncs.com/agent/github/1.2.0/chaosblade-operator-1.2.0-v3.tgz

# 为 chaosblade 创建一个 namespace
[root@s5 k8s]#  kubectl create namespace chaosblade
namespace/chaosblade created

# 安装 ChaosBlade-Operator
[root@s5 k8s]#  ./helm-darwin-amd64/helm install chaos chaosblade-operator-1.2.0-v3.tgz --set webhook.enable=true --namespace=chaosblade
W0621 14:39:16.362347   42437 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0621 14:39:16.375507   42437 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0621 14:39:18.394761   42437 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0621 14:39:20.669546   42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0621 14:39:20.674105   42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0621 14:39:20.687832   42437 warnings.go:70] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W0621 14:39:20.734308   42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0621 14:39:20.742406   42437 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0621 14:39:20.804103   42437 warnings.go:70] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
NAME: chaos
LAST DEPLOYED: Mon Jun 21 14:39:20 2021
NAMESPACE: chaosblade
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for using chaosblade.

# 查看安装结果
[root@s5 k8s]# kubectl get pod -n chaosblade | grep chaosblade
chaosblade-operator-67779995db-cs2lv   1/1     Running   0          4m49s
chaosblade-tool-58ch2                  1/1     Running   0          3m27s
chaosblade-tool-qdwb6                  1/1     Running   0          2m57s
chaosblade-tool-z8jds                  1/1     Running   0          2m57s
[root@s5 k8s]#

ChaosBlade-Operator 启动后将会在每个节点部署一个 chaosblade-tool Pod 和一个 chaosblade-operator Pod，如果都运行正常，则安装成功。上面设置 --set webhook.enable=true 是为了 Pod 文件系统 I/O 故障实验，如果不需要进行该实验，则无需添加该设置。

示例应用准备

配置实验对象，这里使用 guestbook 应用，如果你已经有应用了，就不需要安装这个小的示例应用。

示例应用安装

## 添加helm仓库
[root@s5 k8s]# helm repo add apphub-incubator https://apphub.aliyuncs.com/incubator/
"apphub-incubator" has been added to your repositories

## 安装应用示例
[root@s5 k8s]# helm install guestbook apphub-incubator/guestbook --set service.type=NodePort --namespace=chaosblade
NAME: guestbook
LAST DEPLOYED: Mon Jun 21 22:42:41 2021
NAMESPACE: chaosblade
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace chaosblade -o jsonpath=".spec.ports[0].nodePort" services guestbook)
  export NODE_IP=$(kubectl get nodes --namespace chaosblade -o jsonpath=".items[0].status.addresses[0].address")
  echo http://$NODE_IP:$NODE_PORT
[root@s5 k8s]# echo http://$NODE_IP:$NODE_PORT
http://172.31.184.225:32310

默认的 Service 类型为 LoadBalancer，这里为了方便访问设置为了 NodePort。

示例应用安装验证

访问 http://nodeip:nodeport。

成功后可以看到这个界面，输入任何字符点击submit都会显示在上面，如此而已的一个小应用。

模拟Pod 网络丢包场景

目标

对 redis-master-b96c9795b-4ghxq Pod 注入丢包率 50% 的故障，持续10分钟，只针对 IP 为 10.100.53.195 的 pod 生效，也就是除 10.100.53.195 以外的 pod 都能正常访问 redis-master-b96c9795b-4ghxq。

配置

当前网络信息如下：

[root@s5 chaosblade_scenarios]# kubectl get pods -n chaosblade -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP               NODE   NOMINATED NODE   READINESS GATES
chaosblade-operator-67779995db-ns4qg    1/1     Running   0          100m   10.100.53.193    s7     <none>           <none>
chaosblade-tool-bhgbk                   1/1     Running   0          100m   172.31.184.224   s7     <none>           <none>
chaosblade-tool-mqmrc                   1/1     Running   0          100m   172.31.184.226   s6     <none>           <none>
chaosblade-tool-xqgl5                   1/1     Running   0          100m   172.31.184.225   s5     <none>           <none>
guestbook-7fcc447874-q248s              1/1     Running   0          98m    10.100.53.194    s7     <none>           <none>
guestbook-7fcc447874-zpbn4              1/1     Running   0          98m    10.100.220.67    s6     <none>           <none>
mall-tiny-deployment-85bdb875cf-zl6jw   1/1     Running   0          54m    10.100.220.71    s6     <none>           <none>
redis-master-b96c9795b-4ghxq            1/1     Running   0          10m    10.100.53.196    s7     <none>           <none>
redis-slave-6b8d456947-c6h64            1/1     Running   0          98m    10.100.53.195    s7     <none>           <none>
redis-slave-6b8d456947-twgk9            1/1     Running   0          98m    10.100.220.68    s6     <none>           <none>

配置文件如下。

[root@s5 chaosblade_scenarios]# cat loss_pod_network_by_names.yaml
apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata:
  name: loss-pod-network-by-names
spec:
  experiments:
  - scope: pod
    target: network
    action: loss
    desc: "loss pod network by names"
    matchers:
    - name: names
      value:
      - "redis-master-b96c9795b-4ghxq"
    - name: namespace
      value:
      - "chaosblade"
    - name: interface
      value: ["eth0"]
    - name: percent
      value: ["50"]
    - name: timeout
      value: ["600"]
    - name: destination-ip
      value: ["10.100.53.195"]
[root@s5 chaosblade_scenarios]#

执行

[root@s5 chaosblade_scenarios]# kubectl apply -f loss_pod_network_by_names.yaml
chaosblade.chaosblade.io/loss-pod-network-by-names created
[root@s5 chaosblade_scenarios]#

验证

  1. 登录到10.100.53.195服务器，执行ping命令。
[root@s5 chaosblade_scenarios]# kubectl exec -it redis-slave-6b8d456947-c6h64 bash -n chaosblade
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[ root@redis-slave-6b8d456947-c6h64:/data ]$ ping 10.100.53.196
PING 10.100.53.196 (10.100.53.196) 56(84) bytes of data.
64 bytes from 10.100.53.196: icmp_seq=1 ttl=63 time=0.112 ms
64 bytes from 10.100.53.196: icmp_seq=2 ttl=63 time=0.096 ms
64 bytes from 10.100.53.196: icmp_seq=3 ttl=63 time=0.098 ms
64 bytes from 10.100.53.196: icmp_seq=4 ttl=63 time=0.091 ms
64 bytes from 10.100.53.196: icmp_seq=7 ttl=63 time=0.092 ms
64 bytes from 10.100.53.196: icmp_seq=8 ttl=63 time=0.084 ms
64 bytes from 10.100.53.196: icmp_seq=13 ttl=63 time=0.085 ms
64 bytes from 10.100.53.196: icmp_seq=14 ttl=63 time=0.088 ms
64 bytes from 10.100.53.196: icmp_seq=17 ttl=63 time=0.086 ms
^C
--- 10.100.53.196 ping statistics ---
17 packets transmitted, 9 received, 47% packet loss, time 15999ms
rtt min/avg/max/mdev = 0.084/0.092/0.112/0.012 ms
[ root@redis-slave-6b8d456947-c6h64:/data ]$

确实产生近50%的丢包。

  1. 进入master机器。检查网络队列。
[root@s5 chaosblade_scenarios]# kubectl exec -it redis-master-b96c9795b-4ghxq bash -n chaosblade
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@redis-master-b96c9795b-4ghxq:/data# tc qdisc ls dev eth0
qdisc prio 1: root refcnt 2 bands 4 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc netem 40: parent 1:4 limit 1000 loss 50%
root@redis-master-b96c9795b-4ghxq:/data#

确实产生队列规则。

可见这个模拟是通过操作 qdisc来实现的。如果你有兴趣的话，可以去查一下 linux上的 traffic control 的逻辑。

恢复

[root@s5 chaosblade_scenarios]# kubectl delete -f loss_pod_network_by_names.yaml
chaosblade.chaosblade.io "loss-pod-network-by-names" deleted

留个思考的空间：

  • 通过 tc 命令模拟的丢包和应用问题导致的丢包有什么不同？

相关系列：

以上是关于混沌工程之ChaosBlade-Operator 使用之模拟 POD 丢包场景的主要内容，如果未能解决你的问题，请参考以下文章

混沌工程之ChaosMesh使用之模拟POD网络丢包

混沌工程之ChaosMesh使用之模拟POD网络丢包

云上混沌工程实践之对照实验设计篇-收集

云上混沌工程实践之可行性评估篇-收集

混沌工程之ChaosMesh使用之模拟CPU使用率

混沌工程之ChaosMesh使用之模拟CPU使用率