OpenShift DNS的机制

Posted 2021-01-31 ericnie的技术博客

为什么不直接用kube-dns?

为什么不直接用kube-dns?

为什么不直接用kube-dns?

 

感谢各位前辈的专研，在下午有限的时间里把Openshift DNS的机制理了一下。更详细的材料大家可以参考

https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/

https://www.redhat.com/en/blog/red-hat-openshift-container-platform-dns-deep-dive-dns-changes-red-hat-openshift-container-platform-36

https://www.cnblogs.com/sammyliu/p/10056035.html

本篇主要是基于3.11版本

 

1.DNS架构

 

也就是说所有容器的dns寻址都是通过外部的dnsmasq以及SkyDNS来进行的，不是走容器内部网络去找kube-dns或者类似的Pod

可以通过命令查看一下

• 在master节点

[root@master dnsmasq.d]# netstat -tunlp|grep 53
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      16864/openshift     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      14937/openshift     
tcp        0      0 10.128.0.1:53           0.0.0.0:*               LISTEN      3191/dnsmasq        
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      3191/dnsmasq        
tcp        0      0 192.168.56.113:53       0.0.0.0:*               LISTEN      3191/dnsmasq      

查看具体的进程，注意进程号

# ps -ef|grep openshift

root      14937  14925  0 15:20 ?        00:00:28 openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2
root      16864  16851  6 15:22 ?        00:16:28 openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=2
root      17582  17570  3 15:23 ?        00:09:22 openshift start master controllers --config=/etc/origin/master/master-config.yaml --listen=https://0.0.0.0:8444 --loglevel=2

查看路由信息

[root@master dnsmasq.d]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.56.1    0.0.0.0         UG    100    0        0 enp0s3
10.128.0.0      0.0.0.0         255.252.0.0     U     0      0        0 tun0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.30.0.0      0.0.0.0         255.255.0.0     U     0      0        0 tun0
192.168.56.0    0.0.0.0         255.255.255.0   U     100    0        0 enp0s3

可以看到10.128.0.0是pod网段，172.17.0.0是SVC网段，192.169.56.0是宿主机网段，每个网段都启动了一个dnsmasq,监听在53端口

 

• 在node节点

[root@node1 node]# netstat -tunlp|grep 53
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      17134/openshift     
tcp        0      0 10.131.0.1:53           0.0.0.0:*               LISTEN      3243/dnsmasq        
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      3243/dnsmasq        
tcp        0      0 192.168.56.104:53       0.0.0.0:*               LISTEN      3243/dnsmasq       

ps -ef|grep openshift

root      17134  17120  0 15:21 ?        00:00:31 openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2

 

2.DNS的配置信息

Pod中的dns配置会指向Pod所在宿主机IP，配置为

$ cat /etc/resolv.conf
nameserver 192.168.56.105
search myproject.svc.cluster.local svc.cluster.local cluster.local redhat.com example.com
options ndots:5

192.168.56.105是pod所在的宿主机

 

宿主机的resolv.conf文件

[root@node1 node]# cat /etc/resolv.conf
# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# Generated by NetworkManager
search cluster.local cluster.local example.com
nameserver 192.168.56.104

在部署环境时，会在每个节点上部署 /etc/NetworkManager/dispatcher.d/99-origin-dns.sh 文件。每当节点上的 NetworkManager 服务启动时，该文件会被运行。它的任务包括：

 

• 创建 dnsmasq 配置文件 :
  • node-dnsmasq.conf （没有）
  • origin-dns.conf  
  • origin-upstream-dns.conf(没有)
• 当 NetworkManager 服务启动时启动 dnsmasq 服务
• 设置宿主机的所有默认路由 IP 为 Dnsmasq 的侦听IP
• 修改 /etc/resolv.conf，设置搜索域，以及将宿主机的默认 IP 作为 nameserver
• 创建 /etc/origin/node/resolv.conf

origin-dns.conf的配置目录在/etc/dnsmasq.d/,内容如下

[root@node1 dnsmasq.d]# cat origin-dns.conf 
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=10000
cache-size=10000
bind-dynamic
min-port=1024
except-interface=lo
# End of config

如果有文件origin-upstream-dns.conf ，中定义了上游（upstream） DNS 名字服务器，如果没有可以手工创建.

[root@node2 dnsmasq.d]# cat origin-upstream-dns.conf 
server=10.72.17.5
server=10.68.5.26
server=202.96.134.33
server=202.96.128.86

如果需要解析外部域名，是需要在pod运行的宿主机节点上进行创建的。

node-dnsmasq.conf的内容是

server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1

根据前辈SammyTalksAboutCloud的研究，这个已经写到程序里面去了。

可以通过journalctl -u dnsmasq去查看日志

[root@node2 dnsmasq.d]# journalctl -u dnsmasq
-- Logs begin at Fri 2018-12-28 22:08:53 CST, end at Thu 2019-01-03 20:04:52 CST. --
Dec 28 22:10:48 node2.example.com systemd[1]: Started DNS caching server..
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: started, version 2.76 cachesize 150
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-D
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: DBus support enabled: connected to system bus
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: warning: no upstream servers configured
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: read /etc/hosts - 7 addresses
Dec 28 22:11:59 node2.example.com dnsmasq[3561]: setting upstream servers from DBus
Dec 28 22:11:59 node2.example.com dnsmasq[3561]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Dec 28 22:11:59 node2.example.com dnsmasq[3561]: using nameserver 127.0.0.1#53 for domain cluster.local

根据日志信息，知道dnsmasq实际把请求转发给了监听在127.0.0.1：53上的skyDNS

skyDNS并不是作为一个单独的进程启动，而是在启动网络

openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2

中启动,SkyDNS 调用 OpenShift API 服务来获取主机名、IP地址等信息，然后封装成标准 DNS 记录并返回给查询客户端。 

 

3.DNS在openshift中的配置

• master

cat /etc/origin/master/master-config.yaml

dnsConfig:
  bindAddress: 0.0.0.0:8053
  bindNetwork: tcp4

bind在每个ip的8053端口

• node

cat /etc/origin/node/node-config.yaml

dnsBindAddress: 127.0.0.1:53
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf

 

根据这种机制，service在宿主机范围内（不仅只是在容器中)能够解析

[root@node2 dnsmasq.d]# dig tomcat.myproject.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> tomcat.myproject.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 719
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tomcat.myproject.svc.cluster.local. IN    A

;; ANSWER SECTION:
tomcat.myproject.svc.cluster.local. 30 IN A    172.30.16.194

;; Query time: 0 msec
;; SERVER: 10.0.3.15#53(10.0.3.15)
;; WHEN: Thu Jan 03 20:26:06 CST 2019
;; MSG SIZE  rcvd: 68

能够访问

[root@node2 dnsmasq.d]# curl  tomcat.myproject.svc:8080



<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <title>Apache Tomcat/8.5.37</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>

 

4.查询流程图

 

查看dnsmasq更详细日志

vi /etc/dnsmasq.conf 

# For debugging purposes, log each DNS query as it passes through
# dnsmasq.
log-queries

systemctl restart dnsmasq

 

[root@node2 dnsmasq.d]# journalctl -f -u dnsmasq
-- Logs begin at Fri 2018-12-28 22:08:53 CST. --
Jan 03 20:48:12 node2.example.com dnsmasq[33966]: using nameserver 10.72.17.5#53
Jan 03 20:48:12 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 03 20:48:12 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: setting upstream servers from DBus
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 202.96.128.86#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 202.96.134.33#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 10.68.5.26#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 10.72.17.5#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: query[A] www.baidu.com from 10.0.3.15
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.128.86
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.134.33
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.68.5.26
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.72.17.5
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: query[AAAA] www.baidu.com from 10.0.3.15
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.128.86
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.134.33
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.68.5.26
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.72.17.5
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.baidu.com is <CNAME>
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.a.shifen.com is 14.215.177.38
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.a.shifen.com is 14.215.177.39
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.baidu.com is <CNAME>