Consul on Docker Swarm 与 Spring Boot 客户端
Posted
技术标签:
【中文标题】Consul on Docker Swarm 与 Spring Boot 客户端【英文标题】：Consul on Docker Swarm with Spring Boot clients 【发布时间】：2019-06-10 17:38:35 【问题描述】：我目前在 Centos 7 上的 Docker Swarm 上运行 Consul 时遇到问题（Docker 版本是 18.09.1，构建 4c52b90），或者更确切地说，从工作节点（从 Consul 连接到它）代理或尝试注册的 Spring Boot 应用程序）。
我目前只有一个管理器和一个工作节点。
我使用以下命令创建了一个覆盖网络：
docker network create -d overlay smartdeploy_evo
在管理器上，我正在使用以下名为“docker-compose.consul.master.yml”的撰写文件部署 Consul：
version: '3'
services:
consul:
image: consul:0.9.3
hostname: "consul"
volumes:
- consul_data:/consul/data
ports:
- "8300-8302:8300-8302"
- "8301-8302:8301-8302/udp"
- "8400:8400"
- "8500:8500"
- "53:8600/udp"
entrypoint:
- consul
- agent
- -ui
- -server
- -bootstrap-expect=1
- -bind= GetInterfaceIP "eth0"
- -advertise= GetInterfaceIP "eth0"
- -client=0.0.0.0
- -data-dir=/consul/data
- -disable-host-node-id
healthcheck:
test: ["CMD-SHELL", "consul info | awk '/health_score/if ($$3 >=1) exit 1; else exit 0'"]
labels:
- "evo-type=discovery"
networks:
- smartdeploy_evo
deploy:
placement:
constraints:
- node.role == manager
networks:
smartdeploy_evo:
external: true
volumes:
consul_data:
如果我使用以下命令将其部署到 Swarm：
docker stack deploy -c docker-compose.consul.master.yml consul
然后拖尾输出，我得到以下输出，显示成功启动：
==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
Version: 'v0.9.3'
Node ID: 'cb651a04-9aff-17f6-c5ab-2765fa7b0595'
Node name: 'consul'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: true)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.255.0.243 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2019/01/16 15:49:29 [INFO] raft: Initial configuration (index=1): [Suffrage:Voter ID:10.255.0.243:8300 Address:10.255.0.243:8300]
2019/01/16 15:49:29 [INFO] raft: Node at 10.255.0.243:8300 [Follower] entering Follower state (Leader: "")
2019/01/16 15:49:29 [INFO] serf: EventMemberJoin: consul.dc1 10.255.0.243
2019/01/16 15:49:29 [INFO] serf: EventMemberJoin: consul 10.255.0.243
2019/01/16 15:49:29 [INFO] consul: Adding LAN server consul (Addr: tcp/10.255.0.243:8300) (DC: dc1)
2019/01/16 15:49:29 [INFO] consul: Handled member-join event for server "consul.dc1" in area "wan"
2019/01/16 15:49:29 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2019/01/16 15:49:29 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2019/01/16 15:49:29 [INFO] agent: Started HTTP server on [::]:8500
2019/01/16 15:49:37 [ERR] agent: failed to sync remote state: No cluster leader
2019/01/16 15:49:39 [WARN] raft: Heartbeat timeout from "" reached, starting election
2019/01/16 15:49:39 [INFO] raft: Node at 10.255.0.243:8300 [Candidate] entering Candidate state in term 2
2019/01/16 15:49:39 [INFO] raft: Election won. Tally: 1
2019/01/16 15:49:39 [INFO] raft: Node at 10.255.0.243:8300 [Leader] entering Leader state
2019/01/16 15:49:39 [INFO] consul: cluster leadership acquired
2019/01/16 15:49:39 [INFO] consul: New leader elected: consul
2019/01/16 15:49:39 [INFO] consul: member 'consul' joined, marking health alive
2019/01/16 15:49:41 [INFO] agent: Synced node info
然后我尝试使用以下名为“docker-compose.services.2.yml”的撰写文件在 Worker 节点上运行我的 Spring Boot 应用程序：
version: '3'
services:
user-mgmt:
image: smartdeployevo_usermgmt:latest
ports:
- "10500:10500"
environment:
- spring.cloud.consul.hostHealth=user-mgmt
- spring.profiles.active=production
labels:
- "evo-type=service"
networks:
- smartdeploy_evo
deploy:
placement:
constraints:
- node.role == worker
networks:
smartdeploy_evo:
external: true
Spring Boot Web 应用程序尝试使用端口 8500 上的主机别名“consul”连接到 Consul 服务器。
我使用以下命令将此服务部署到堆栈：
docker stack deploy -c docker-compose.services.2.yml springboot
Spring Boot 应用程序在启动时失败并出现以下错误：
2019-01-16 15:51:38.753 ERROR [user-mgmt,,,] 1 --- [ main] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2019-01-16 15:51:38.766 ERROR [user-mgmt,,,] 1 --- [ main] o.s.boot.SpringApplication : Application run failed
com.ecwid.consul.transport.TransportException: org.apache.http.conn.HttpHostConnectException: Connect to consul:8500 [consul/10.0.2.110] failed: Connection refused (Connection refused)
请注意，工人认为 Consul 正在运行的 IP 地址是 10.0.2.110
如果我进入容器上运行的 Consul 容器并运行“ipconfig”，我会得到以下 IP 地址：
docker exec -it 714805722ace sh
ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0A:FF:00:F3
inet addr:10.255.0.243 Bcast:10.255.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth1 Link encap:Ethernet HWaddr 02:42:0A:00:02:6F
inet addr:10.0.2.111 Bcast:10.0.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth2 Link encap:Ethernet HWaddr 02:42:AC:12:00:03
inet addr:172.18.0.3 Bcast:172.18.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:698 (698.0 B) TX bytes:810 (810.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:80 errors:0 dropped:0 overruns:0 frame:0
TX packets:80 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:38312 (37.4 KiB) TX bytes:38312 (37.4 KiB)
请注意，“eth1”在 Spring Boot 应用程序尝试连接的同一网络上的 IP 地址是
10.0.2.111
而不是 Spring Boot 使用的 IP 地址 10.0.2.110。
问 - 为什么 IP 地址存在差异，如何让我的 Spring Boot 应用程序连接到正确的 IP 地址？
作为附加信息，如果我在工作节点上运行另一个容器，例如 Apache HTTPD 容器，然后进入它并运行
ping consul
PING consul (10.0.2.110): 56 data bytes
64 bytes from 10.0.2.110: seq=0 ttl=64 time=0.189 ms
64 bytes from 10.0.2.110: seq=1 ttl=64 time=0.167 ms
它还会看到 10.0.2.110 IP 地址，而不是 consul 正在侦听的 10.0.2.111 IP 地址。
任何帮助将不胜感激！
更新：
根据要求，这是在 Manager 和 Worker 节点上要求的输出：
经理：
docker network inspect smartdeploy_evo
[
"Name": "smartdeploy_evo",
"Id": "7qyimelt9rjcaukfgd22tenxr",
"Created": "2019-01-16T15:49:28.627116387Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM":
"Driver": "default",
"Options": null,
"Config": [
"Subnet": "10.0.2.0/24",
"Gateway": "10.0.2.1"
]
,
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom":
"Network": ""
,
"ConfigOnly": false,
"Containers":
"3e4ca1fd66b8d8012e7f4aca14b3a0d853761a92e83a95870dceef315de0f8dc":
"Name": "consul_consul.1.s0pbo1iuntiry2eh310wdsdqp",
"EndpointID": "ee3bed0936fd2a9a4aa79e1efc23b60f1d45402a3b63a474743af5c2f8f5a479",
"MacAddress": "02:42:0a:00:02:6f",
"IPv4Address": "10.0.2.111/24",
"IPv6Address": ""
,
"lb-smartdeploy_evo":
"Name": "smartdeploy_evo-endpoint",
"EndpointID": "2492956b7df72f4c3f3c03e12b7f18d16601a1948b2d7fe202543d91bca75ec6",
"MacAddress": "02:42:0a:00:02:70",
"IPv4Address": "10.0.2.112/24",
"IPv6Address": ""
,
"Options":
"com.docker.network.driver.overlay.vxlanid_list": "4100"
,
"Labels": ,
"Peers": [
"Name": "cdfe9dfef133",
"IP": "192.221.173.234"
,
"Name": "2a07b72b316f",
"IP": "192.221.173.235"
]
]
工人：
docker network inspect smartdeploy_evo
[
"Name": "smartdeploy_evo",
"Id": "7qyimelt9rjcaukfgd22tenxr",
"Created": "2019-01-16T15:53:39.639835468Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM":
"Driver": "default",
"Options": null,
"Config": [
"Subnet": "10.0.2.0/24",
"Gateway": "10.0.2.1"
]
,
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom":
"Network": ""
,
"ConfigOnly": false,
"Containers":
"4943e458d4e08020ba2e49e375fd9371b75f8adb3835d249a453ab3cfbac4bc1":
"Name": "springboot_user-mgmt.1.p91mu1nmybt6k8zfezpe24a2f",
"EndpointID": "2e04bfc51b50fedd2a766897832047800e55f555f91521767bd0a2647b2f3662",
"MacAddress": "02:42:0a:00:02:90",
"IPv4Address": "10.0.2.144/24",
"IPv6Address": ""
,
"b50a7205440d0e7937348fe43d0d7bb0a8e66b7371c2c46dd3c7831248bdceb9":
"Name": "h_httpd.1.wymlfscwus5rk8ajf62qscb4r",
"EndpointID": "d46331e82c16c4a4597470a7aa72685721fe063a06e11bc739623178031b24ea",
"MacAddress": "02:42:0a:00:02:88",
"IPv4Address": "10.0.2.136/24",
"IPv6Address": ""
,
"lb-smartdeploy_evo":
"Name": "smartdeploy_evo-endpoint",
"EndpointID": "9e3a507c4f3a58e0b74479c4a3878f858a816b5fd6905aa2fbfa89c2d1a8276a",
"MacAddress": "02:42:0a:00:02:81",
"IPv4Address": "10.0.2.129/24",
"IPv6Address": ""
,
"Options":
"com.docker.network.driver.overlay.vxlanid_list": "4100"
,
"Labels": ,
"Peers": [
"Name": "cdfe9dfef133",
"IP": "192.221.173.234"
,
"Name": "2a07b72b316f",
"IP": "192.221.173.235"
]
]
【问题讨论】：
只是为了清除 ip 混淆，您可以在管理器和工作节点上运行 docker network inspect your_network_name 并获取两个容器的正确 ip 按要求更新原帖，感谢支持！ 【参考方案1】：我一直在做一些研究，并从Consul github site 中遇到了类似的问题，其中用户soakes 建议使用 gliderlabs/registrator 作为领事经理和代理之间的桥梁。
我下面的版本与他的版本略有不同（主要是容器名称与我在 Spring Boot 代码中的默认值相匹配，并且没有 SSL 等），但如果没有他的多数意见，就会被难住！
我按如下方式创建了我的 Swarm 覆盖网络：
docker network create -d overlay --opt com.docker.network.swarm.name=smartdeploy_evo smartdeploy_evo
然后使用我从 Manager 节点部署的本文末尾显示的 Compose 文件，如下所示：
docker stack deploy -c consul.yml consul
文件 consul.yml 定义如下：
version: "3.4"
networks:
smartdeploy_evo:
external: true
volumes:
consul:
services:
consul:
image: consul:0.9.3
volumes:
- consul:/consul
ports:
- target: 8500
published: 8500
mode: host
networks:
smartdeploy_evo:
aliases:
- consul.cluster
environment:
- 'CONSUL_LOCAL_CONFIG= "skip_leave_on_interrupt": true,
"data_dir":"/consul/data",
"server":true '
- CONSUL_BIND_INTERFACE=eth0
command: agent -ui -data-dir /consul/data -server -client 0.0.0.0 -bootstrap-expect=1 -retry-join consul.cluster
deploy:
endpoint_mode: dnsrr
mode: global
placement:
constraints: [node.role == manager]
consul_client:
image: consul:0.9.3
volumes:
- consul:/consul
networks:
smartdeploy_evo:
aliases:
- consul.client.cluster
environment:
- 'CONSUL_LOCAL_CONFIG= "skip_leave_on_interrupt": true,
"data_dir":"/consul/data" '
- CONSUL_BIND_INTERFACE=eth0
command: agent -ui -data-dir /consul/data -client 0.0.0.0 -retry-join consul.cluster
deploy:
endpoint_mode: dnsrr
mode: global
placement:
constraints: [node.role != manager]
consul_registrator:
image: gliderlabs/registrator:master
command: -internal consul://consul.cluster:8500
volumes:
- /var/run/docker.sock:/tmp/docker.sock
networks:
- smartdeploy_evo
deploy:
mode: global
【讨论】：
以上是关于Consul on Docker Swarm 与 Spring Boot 客户端的主要内容，如果未能解决你的问题，请参考以下文章
Docker可视化界面（Consul+Shipyard+Swarm+Service Discover
Docker可视化界面（Consul+Shipyard+Swarm+Service Discover）部署记录
ubuntu-docker-consul-swarm-shipyard-portainer
Docker（十三）：OpenStack部署Docker集群实战