[k8s]docker calico网络

Posted 2020-10-18 毛台

docker cluster-store选项

etcd-calico(bgp)实现docker夸主机通信

配置calico网络

-  启动etcd
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls=http://192.168.2.11:2379 --debug

- 启动docker
iptables -P FORWARD ACCEPT
systemctl stop docker
dockerd --cluster-store=etcd://192.168.2.11:2379


- 设置calico网络配置
mkdir -p /etc/calico/
cat > /etc/calico/calicoctl.cfg <<EOF
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.2.11:2379"
EOF

- 启动calico
calicoctl node run
calicoctl node status

- n1创建calico驱动类型的global网卡
docker network rm cal_net1
docker network create --driver calico --ipam-driver calico-ipam cal_net1
docker network ls

- n1创建的网络会自动同步到n2

docker network create --driver calico --ipam-driver calico-ipam cal_net1

--driver calico           指定使用 calico 的 libnetwork CNM driver。
--ipam-driver calico-ipam 指定使用 calico 的 IPAM driver 管理 IP。

测试calico网络-2台node的容器互通

docker run --net cal_net1 --name b1 -itd busybox
docker exec -it b1 ip a

docker run --net cal_net1 --name b2 -itd busybox
docker exec -it b2 ip a


[root@n1 ~]# docker exec -it b1 ping 192.168.158.64
PING 192.168.158.64 (192.168.158.64): 56 data bytes
64 bytes from 192.168.158.64: seq=0 ttl=62 time=0.774 ms

calico网络结构探究

遇到的问题,docker global类型的网络不自动同步: 原因 etcd advertise-client-urls被错误指定为0.0.0.0了,需要指明具体的ip地址

- 正确姿势
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls=http://192.168.2.11:2379 --debug

- 错误姿势
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls=http://0.0.0.0:2379

calicoctl node run干了什么事

1,开启ip_forward
2,下载calico-node镜像,并启动

docker run --net=host --privileged \\
    --name=calico-node -d \\
    --restart=always -e NODENAME=n1.ma.com \\
    -e CALICO_NETWORKING_BACKEND=bird \\
    -e CALICO_LIBNETWORK_ENABLED=true \\
    -e ETCD_ENDPOINTS=http://127.0.0.1:2379 \\
    -v /var/log/calico:/var/log/calico \\
    -v /var/run/calico:/var/run/calico \\
    -v /lib/modules:/lib/modules \\
    -v /run:/run -v /run/docker/plugins:/run/docker/plugins \\
    -v /var/run/docker.sock:/var/run/docker.sock \\
quay.io/calico/node:latest


3,写入etcd信息

定制 Calico 的 IP 池

- 创建自定义ipam网络
cat << EOF | calicoctl create -f -
- apiVersion: v1
  kind: ipPool
  metadata:
    cidr: 17.2.0.0/16
EOF

docker network create --driver calico --ipam-driver calico-ipam --subnet=17.2.0.0/16 my_net

- 启动容器(固定ip地址方式)
docker run --net my_net --ip 17.2.3.11 -it busybox

- 查看网段
calicoctl node status
calicoctl get ipPool

定制 Calico 的 IP 池

- 创建自定义ipam网络
cat << EOF | calicoctl create -f -
- apiVersion: v1
  kind: ipPool
  metadata:
    cidr: 17.2.0.0/16
EOF

docker network create --driver calico --ipam-driver calico-ipam --subnet=17.2.0.0/16 my_net

- 启动容器(固定ip地址方式)
docker run --net my_net --ip 17.2.3.11 -it busybox

- 查看网段
calicoctl node status
calicoctl get ipPool

cat << EOF | calicoctl create -f -

• apiVersion: v1
  kind: ipPool
  metadata:
  cidr: 17.2.0.0/16
  EOF

calico默认网络策略

- calico默认网络策略
Calico 默认的 policy 规则是：容器只能与同一个 calico 网络中的容器通信。

- 查看calico默认策略
calicoctl get profile cal_net1 -o yaml

① 命名为 cal_net1，这就是 calico 网络 cal_net1 的 profile。
② 为 profile 添加一个 tag cal_net1。注意，这个 tag 虽然也叫 cal_net1，其实可以随便设置，这跟上面的 name: cal_net1 没有任何关系。此 tag 后面会用到。
③ egress  对从容器发出的数据包进行控制，当前没有任何限制。
④ ingress 对进入容器的数据包进行限制，当前设置是接收来自 tag cal_net1 的容器，根据第 ① 步设置我们知道，实际上就是只接收本网络的数据包，这也进一步解释了前面的实验结果。

定制calico默认网络策略

Calico 能够让用户定义灵活的 policy 规则，精细化控制进出容器的流量，下面我们就来实践一个场景：
    创建一个新的 calico 网络 cal_web 并部署一个 httpd 容器 web1。
    定义 policy 允许 cal_net2 中的容器访问 web1 的 80 端口。


- 首先创建 cal_web。
docker network create --driver calico --ipam-driver calico-ipam cal_web
calicoctl get profile cal_web -o yaml

- 在 host1 中运行容器 web1，连接到 cal_web：
docker run --net cal_web --name web1 -d httpd
docker exec -it web1 ip a

- 创建net1
docker network create --driver calico --ipam-driver calico-ipam cal_net1
docker run --net cal_net1 --name b2 -itd busybox
docker exec -it b2 ip a
访问 web1的80
docker exec -it b2 wget x.x.x.x  #不通

- 创建 policy 文件 web.yml，内容为:
cat > web.yaml<<EOF
- apiVersion: v1
  kind: profile
  metadata:
    name: cal_web
  spec:
    ingress:
    - action: allow
      protocol: tcp
      source:
        tag: cal_net1
      destination:
        ports:
          - 80
EOF
calicoctl apply -f web.yaml
calicoctl get profile cal_web -o yaml

- 访问 web1的80
docker exec -it b2 wget x.x.x.x  #通了

- 在两个节点分别查看策略(一致)
calicoctl get profile cal_web -o yaml

calico最佳实战

calico数据转发流程

发数据
查路由
arp网关(网关有代理arp功能)
数据发到主机
根据主机路由表转发

实现网络

参考: http://cizixs.com/2017/10/19/docker-calico-network

calico数据库探究

使用etcd brower图形化查看etcd

docker run --name etcd-browser -p 0.0.0.0:8000:8000 --env ETCD_HOST=192.168.2.11 --env ETCD_PORT=2379 --env AUTH_PASS=doe -itd buddho/etcd-browser

注意端口
注意etcdip

[root@n1 ~]# etcdctl ls /calico
/calico/bgp
/calico/ipam
/calico/v1

[root@n1 ~]# etcdctl ls /calico/bgp/v1
/calico/bgp/v1/host
/calico/bgp/v1/global


[root@n1 ~]# etcdctl ls /calico/bgp/v1/global
/calico/bgp/v1/global/node_mesh
/calico/bgp/v1/global/as_num
/calico/bgp/v1/global/loglevel
/calico/bgp/v1/global/custom_filters


[root@n1 ~]# etcdctl get /calico/bgp/v1/global/as_num
64512

[root@n1 ~]# etcdctl ls /calico/bgp/v1/host
/calico/bgp/v1/host/n1.ma.com
/calico/bgp/v1/host/n2.ma.com


[root@n1 ~]# etcdctl ls /calico/bgp/v1/host/n1.ma.com
/calico/bgp/v1/host/n1.ma.com/ip_addr_v6
/calico/bgp/v1/host/n1.ma.com/network_v4
/calico/bgp/v1/host/n1.ma.com/ip_addr_v4

[root@n1 ~]# etcdctl get /calico/bgp/v1/host/n1.ma.com/ip_addr_v4
192.168.2.11

[root@n1 ~]# etcdctl get /calico/bgp/v1/host/n1.ma.com/network_v4
192.168.2.0/24

calico组件担任的角色

参考: https://docs.projectcalico.org/v1.6/reference/without-docker-networking/docker-container-lifecycle

http://www.youruncloud.com/blog/131.html

bird   实现了bgp协议
flex   调度执行者
confd  服务发现,通过flex修改配置

libnetwork-管理ip/接口


etcd
    ipam  节点地址记录
    bgp   bgp相关
    v2    policy

以前总结的calico零散知识

#!/usr/bin/env bash


docker stats

etcd --advertise-client-urls=http://0.0.0.0:2379 --listen-client-urls=http://0.0.0.0:2379 --enable-v2 --debug

vim /usr/lib/systemd/system/docker.service
# /etc/systemd/system/docker.service
--cluster-store=etcd://192.168.14.132:2379

systemctl daemon-reload
systemctl restart docker.service

[root@node1 ~]# ps -ef|grep docker
root       8122      1  0 Nov07 ?        00:01:01 /usr/bin/dockerd --cluster-store=etcd://192.168.14.132:2379

etcdctl ls
/docker


cd /usr/local/bin
wget https://github.com/projectcalico/calicoctl/releases/download/v1.6.1/calicoctl
chmod +x calicoctl

[root@node1 ~]# rpm -qa|grep etcd
etcd-3.2.5-1.el7.x86_64

mkdir /etc/calico
cat >> /etc/calico/calicoctl.cfg <<EOF
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.14.132:2379"
EOF

calicoctl node run
calicoctl node run --ip=192.168.14.132

1,开启ip_forward
2,下载calico-node镜像,并启动
3,写入etcd信息

iptables -P FORWARD ACCEPT
etcdctl rm --recursive /calico
etcdctl rm --recursive /docker

# 可以看到bgp邻居已经建立起来了(14.132 14.133)
calicoctl node status

# 任意一台机器创建网络,另一台机器会同步过去的
docker network rm cal_net1
docker network create --driver calico --ipam-driver calico-ipam cal_net1

#+++++++++++++++++++++++++++
#  测试
#+++++++++++++++++++++++++++
# 14.132
docker container run --net cal_net1 --name bbox1 -tid busybox
docker exec bbox1 ip address
docker exec bbox1 route -n

# 14.133
docker container run --net cal_net1 --name bbox2 -tid busybox


docker exec bbox2 ip address
docker exec bbox2 ping  192.168.108.128


#+++++++++++++++++++++++++++
#  参考
#+++++++++++++++++++++++++++
https://mp.weixin.qq.com/s/VL72aVjU4KB3c2UTihl-DA
http://blog.csdn.net/felix_yujing/article/details/55213239


#+++++++++++++++++++++++++++
#  创建网段
#+++++++++++++++++++++++++++
calicoctl node status
calicoctl get ipPool
- apiVersion: v1
  kind: ipPool
  metadata:
    cidr: 10.20.0.0/24
  spec:
    ipip:
      enabled: true
    nat-outgoing: true


另外一个测试
docker network create --driver calico --ipam-driver calico-ipam  --subnet 10.30.0.0/24 net1

docker network create --driver calico --ipam-driver calico-ipam  --subnet 10.30.0.0/24 net1
docker network create --driver calico --ipam-driver calico-ipam  --subnet 10.30.0.0/24 net2
docker network create --driver calico --ipam-driver calico-ipam  --subnet 10.30.0.0/24 net3

#node1
docker run --net net1 --name workload-A -tid busybox
docker run --net net2 --name workload-B -tid busybox
docker run --net net1 --name workload-C -tid busybox
#node2
docker run --net net3 --name workload-D -tid busybox
docker run --net net1 --name workload-E -tid busybox



#同一网络内的容器（即使不在同一节点主机上）可以使用容器名来访问
docker exec workload-A ping -c 4 workload-C.net1
docker exec workload-A ping -c 4 workload-E.net1
#不同网络内的容器需要使用容器ip来访问（使用容器名会报：bad address）
docker exec workload-A ping -c 2  `docker inspect --format "{{ .NetworkSettings.Networks.net2.IPAddress }}" workload-B`


#calico默认策略,同一网络内的容器是能相互通信的；不同网络内的容器相互是不通的。不同节点上属于同一网络的容器也是能相互通信的，这样就实现了容器的跨主机互连。



#+++++++++++++++++++++++++++
#  修改默认策略
#+++++++++++++++++++++++++++

cat << EOF | calicoctl apply -f -
- apiVersion: v1
  kind: profile
  metadata:
    name: cal_net12icmp
    labels:
      role: database
  spec:
    ingress:
    - action: allow
      protocol: icmp
      source:
        tag: net1
      destination:
        tag: net2
EOF





https://docs.projectcalico.org/v2.2/reference/public-cloud/aws
$ calicoctl apply -f - << EOF
apiVersion: v1
kind: ipPool
metadata:
  cidr: 192.168.0.0/16
spec:
  ipip:
    enabled: true
    mode: cross-subnet
  nat-outgoing: true
EOF

参考:
Docker网络解决方案-Calico部署记录
https://allgo.cc/2015/04/16/centos7%E7%BD%91%E5%8D%A1%E6%A1%A5%E6%8E%A5/
yum install bridge-utils
calico原理
http://www.cnblogs.com/kevingrace/p/6864804.html

#!/usr/bin/env bash
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-udp-ingress-controller
  labels:
    k8s-app: nginx-udp-ingress-lb
  namespace: kube-system
spec:
  replicas: 1
  selector:
    k8s-app: nginx-udp-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: nginx-udp-ingress-lb
        name: nginx-udp-ingress-lb
    spec:
      hostNetwork: true
      terminationGracePeriodSeconds: 60
      containers:
      #- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.8
      - image: 192.168.1.103/k8s_public/nginx-ingress-controller:0.9.0-beta.5
        name: nginx-udp-ingress-lb
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - containerPort: 81
          hostPort: 81
        - containerPort: 443
          hostPort: 443
        - containerPort: 53
          hostPort: 53
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --udp-services-configmap=$(POD_NAMESPACE)/nginx-udp-ingress-configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: udp-configmap-example
data:
  53: "kube-system/kube-dns:53"