docker固定IP第三种方法
Posted puremans
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了docker固定IP第三种方法相关的知识，希望对你有一定的参考价值。
docker高级应用之单机持久化固定容器IP
过完年了，快1个月没有写博客，现在给大家分享一下我是如何在单机环境，实现持久化固定容器IP。
我是去年6月份开始玩docker，到现在已经9个月了，docker的最新版本也到了1.5.目前我这里对docker已经放到测试环境给研发使用，目前感觉最大的问题就是容器手动重启或者异常关闭后在启动容器，那么容器的ip就回变化（这个是使用默认的docker0做桥接），如果使用pipework手动配置ip，也得重新在来一遍，很是麻烦，所以我写了一个程序来自动化的解决这个问题。
一、程序介绍
使用到的程序有2个：
1、创建容器程序，名字是create_docker_container_use_static_ip.sh,使用shell脚本编写。
2、监控容器状态并自动给重启没有ip的容器设置ip，名字是auto_check_modify_container.py，使用python编写。
创建容器程序的逻辑：
1、运行创建脚本来创建docker container；
2、创建的信息会写入到etcd里进行保存；
3、根据配置信息来创建具体的container。
其中我程序使用openvswitch来坐桥接，并对不同vlan做不同的网段设置。
监控容器并给自动重启设置ip程序逻辑：
1、从etcd里获取容器的信息；
2、根据第一步的信息，进行nmap扫描ssh服务是否启动，未启动的转给下一步；
3、收到上一步的ip，然后根据etcd的容器信息进行重新pipework配置之前的固定ip。
下面是创建容器程序脚本内容
#!/bin/bash
#author:Deng Lei
#email: [email protected]
#this script is create new docker container and use pipework add static ip,when container is restart,it also use last static ip
container=$1
images=$2
daemon_program=$3
vlan_tag=$4
#ip_pool=‘172.16.10.1/24‘
bridge=ovs2
default_vlan=1
default_vlan_ip=‘172.16.0.1/24‘
email=‘[email protected]‘
local_ip=`/sbin/ifconfig | grep ‘inet‘| grep -Ev ‘(127|117|211|172|::1|fe)‘ |awk ‘{print $2}‘|head -n 1`
if [ -z $1 ] || [ -z $2 ]; then
echo "Usage: container_name container_image vlan_tag(default:1)"
echo "Example: I want create new docker container test1 centos6:latest supervisord vlan 100"
echo "The command is: bash `basename $0` test1 centos6:latest supervisord 100"
exit 1
fi
[ `docker ps -a|grep -v "NAMES"|awk ‘{print $NF}‘|grep "$container" &>>/dev/null && echo 0 || echo 1` -eq 0 ] && echo "The container $container is exist!" && exit 1
[ `which pipework &>/dev/null && echo 0|| echo 1` -eq 1 ] && echo "I can‘t find pipework,please install it!" && exit 1
if [ -z $3 ];then
vlan_tag=${default_vlan}
fi
#create new docker container
docker run --restart always -d --net="none" --name="$container" $images $daemon_program &>>/dev/null
#check openvswitch bridge exist
if [ `ovs-vsctl list-br |grep ${bridge} &>/dev/null && echo 0|| echo 1` -eq 1 ];then
ovs-vsctl add-br ${bridge}
fi
#set up default vlan info
check_default_vlan_exist=`etcdctl get /app/docker/${local_ip}/vlan${default_vlan}/gateway 2>/dev/null || echo 1`
if [ ${check_default_vlan_exist} = "1" ];then
ovs-vsctl add-port $bridge vlan${default_vlan} tag=${default_vlan} -- set interface vlan${default_vlan} type=internal
#ifconfig vlan${default_vlan} ${default_vlan_ip}
ip link set vlan${default_vlan} up
ip addr add $default_vlan_ip dev vlan${default_vlan}
etcdctl set /app/docker/${local_ip}/vlan${default_vlan}/nowip $default_vlan_ip &>>/dev/null
etcdctl set /app/docker/${local_ip}/vlan${default_vlan}/gateway $default_vlan_ip &>>/dev/null
etcdctl set /app/docker/now_vlan_ip $default_vlan_ip &>>/dev/null
fi
#set up new vlan info
new_vlan_gateway_ip=`etcdctl get /app/docker/${local_ip}/vlan${vlan_tag}/gateway 2>/dev/null || echo 1`
if [ ${new_vlan_gateway_ip} = "1" ];then
sleep 1
now_vlan_ip=`etcdctl get /app/docker/now_vlan_ip`
new_vlan_ip=$(awk -v new_ip=`echo $now_vlan_ip|cut -d/ -f 1|awk -F‘.‘ ‘{print $1"."$2"."$3+1"."$4}‘` -v new_ip_netmask=`echo $now_vlan_ip|cut -d/ -f2` ‘BEGIN{print new_ip"/"new_ip_netmask}‘)
ovs-vsctl add-port $bridge vlan${vlan_tag} tag=${vlan_tag} -- set interface vlan${vlan_tag} type=internal
ip link set vlan${vlan_tag} up
ip addr add ${new_vlan_ip} dev vlan${vlan_tag}
# ifconfig vlan${vlan_tag} ${new_vlan_ip}
etcdctl set /app/docker/now_vlan_ip ${new_vlan_ip} &>>/dev/null
etcdctl set /app/docker/${local_ip}/vlan${vlan_tag}/gateway ${new_vlan_ip} &>>/dev/null
etcdctl set /app/docker/${local_ip}/vlan${vlan_tag}/nowip ${new_vlan_ip} &>>/dev/null
new_vlan_gateway_ip=${new_vlan_ip}
fi
#calculate new ip
sleep 1
now_container_ip=`etcdctl get /app/docker/${local_ip}/vlan${vlan_tag}/nowip`
echo "now container ip is:$now_container_ip"
new_container_ip=$(awk -v new_ip=`echo $now_container_ip|cut -d/ -f 1|awk -F‘.‘ ‘{print $1"."$2"."$3"."$4+1}‘` -v new_ip_netmask=`echo $now_container_ip|cut -d/ -f2` ‘BEGIN{print new_ip"/"new_ip_netmask}‘)
etcdctl set /app/docker/${local_ip}/vlan${vlan_tag}/nowip $new_container_ip &>/dev/null
echo "finish calculate new ip!"
create_time=`date +"%Y-%m-%d %T"`
`which pipework` $bridge $container ${new_container_ip}@`echo ${new_vlan_gateway_ip}|awk -F ‘/‘ ‘{print $1}‘` @${vlan_tag}
etcdctl set /app/docker/${local_ip}/vlan${vlan_tag}/container/${container} "{‘Physics_ip‘:‘${local_ip}‘,‘Container_name‘:‘${container}‘,‘Container_ip‘:‘${new_container_ip}‘,‘Container_vlan‘:‘${vlan_tag}‘,‘Container_vlan_gateway‘:‘${new_vlan_gateway_ip}‘,‘Container_create‘:‘${create_time}‘,‘Container_status‘:‘running‘}"
注意 ：如果想使用本程序，客户端必须安装openvswitch、etcd与etcdctl、pipework。
监控容器并给自动重启设置ip程序脚本内容：
#!/usr/bin/env python
#author:Deng Lei
#email: [email protected]
import sys
import nmap
import multiprocessing
import time
import socket, struct, fcntl
import etcd
import re
from docker import Client
import subprocess
def get_local_ip(iface = ‘em1‘):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sockfd = sock.fileno()
SIOCGIFADDR = 0x8915
ifreq = struct.pack(‘16sH14s‘, iface, socket.AF_INET, ‘\x00‘*14)
try:
res = fcntl.ioctl(sockfd, SIOCGIFADDR, ifreq)
except:
return None
ip = struct.unpack(‘16sH2x4s8x‘, res)[2]
return socket.inet_ntoa(ip)
def get_etcd_info(key):
etcd_result=[]
r = etcd_client.read(‘%s%s‘%(docker_etcd_key,local_ip), recursive=True, sorted=True)
for child in r.children:
if child.dir is not True and "Container_name" in child.value:
etcd_result.append(child.value)
return etcd_result
def check_by_nmap(ip,port):
check_result=len(nm.scan(‘%s‘%ip,‘%s‘%port)[‘scan‘])
if check_result == 0:
msg=‘%s‘%(ip)
return msg
def docker_container_stop():
docker_container=docker_client.containers(all=True)
container_name=[]
container_stop_name=[]
for i in docker_container:
if re.match(‘Exited‘,i[‘Status‘]):
container_name.append(i[‘Names‘])
for b in container_name:
for c in b:
container_stop_name.append(c)
return container_stop_name
def docker_container_all():
docker_container=docker_client.containers(all=True)
container_name=[]
container_stop_name=[]
for i in docker_container:
container_name.append(i[‘Names‘])
for b in container_name:
for c in b:
container_stop_name.append(c)
return container_stop_name
if __name__ == "__main__":
nm = nmap.PortScanner()
etcd_client=etcd.Client(host=‘127.0.0.1‘, port=4001)
docker_client = Client(base_url=‘unix://var/run/docker.sock‘, version=‘1.15‘, timeout=10)
local_ip=get_local_ip(‘ovs1‘)
bridge=‘ovs2‘
docker_etcd_key=‘/app/docker/‘
port=‘22‘
ip_list=[]
now_etcd_result=get_etcd_info(docker_etcd_key)
docker_container_stop_name=docker_container_stop()
docker_container_all_name=docker_container_all()
for i in range(len(now_etcd_result)):
now_info=eval(now_etcd_result[i])
key=‘%s%s/vlan%s/container/%s‘%(docker_etcd_key,local_ip,now_info[‘Container_vlan‘],now_info[‘Container_name‘])
if ‘/‘+now_info[‘Container_name‘] not in docker_container_stop_name and now_info[‘Container_status‘] != ‘delete‘:
ip_list.append(eval(now_etcd_result[i])[‘Container_ip‘].split(‘/‘)[0])
if now_info[‘Container_status‘] == ‘stop‘:
now_info[‘Container_status‘]=‘running‘
etcd_client.write(key,now_info)
else:
now_info[‘Container_status‘]=‘stop‘
etcd_client.write(key,now_info)
if ‘/‘+now_info[‘Container_name‘] not in docker_container_all_name:
print ‘%s %s will delete,because docker client had deleted!‘%(time.strftime(‘%Y-%m-%d %T‘),key)
now_info[‘Container_status‘]=‘delete‘
etcd_client.write(key,now_info)
ip_list=list(set(ip_list))
if len(ip_list) == 1:
num=1
elif len(ip_list)>=4 and len(ip_list)<=8:
num=4
elif len(ip_list) >8 and len(ip_list)<=20:
num=8
else:
num=15
pool = multiprocessing.Pool(processes=num)
scan_result=[]
for i in ip_list:
pool.apply_async(check_by_nmap, (i,port, ))
scan_result.append(pool.apply_async(check_by_nmap, (i,port, )))
pool.close()
pool.join()
result=[]
for res in scan_result:
if res.get() is not None:
result.append(res.get())
result=list(set(result))
if len(result) <= 0:
print ‘%s Everything is all right!‘%time.strftime(‘%Y-%m-%d %T‘)
sys.exit(1)
else:
print ‘%s container need modify container static ip is:%s‘%(time.strftime(‘%Y-%m-%d %T‘),result)
for i in range(len(now_etcd_result)):
now_info=eval(now_etcd_result[i])
if (now_info[‘Container_ip‘]).split(‘/‘)[0] in result:
modify_container=subprocess.Popen("`which pipework` %s %s %[email protected]%s @%s &>>/dev/null && echo 0 || echo 1"%(bridge,now_info[‘Container_name‘],now_info[‘Container_ip‘],now_info[‘Container_vlan_gateway‘].split(‘/‘)[0],now_info[‘Container_vlan‘]),shell=True,stdout=subprocess.PIPE)
if (modify_container.stdout.readlines()[0]).strip(‘\n‘) == 0:
print ‘container_name:%s container_ip:%s modify static ip is fail!‘%(now_info[‘Container_name‘],now_info[‘Container_ip‘])
else:
print ‘container_name:%s container_ip:%s modify static ip is success!‘%(now_info[‘Container_name‘],now_info[‘Container_ip‘])
注意：使用本脚本必须安装nmap、etcd、docker模块。
其中python里nmap的安装方法是
easy_install python-nmap
其中python里etcd的安装方法是
安装基础库
yum install libffi libffi-devel python-devel
安装程序
git clone https://github.com/jplana/python-etcd.git
cd python-etcd
python setup.py install
其中python里docker的安装方法是
easy_install docker-py
二、效果展示
需求：
创建主机test1、test2、test3、test4，需要test1与test2属于vlan100，test3与test4属于vlan300，架构图如下
ovs2
|
|
-----------------
| |
| |
vlan100(tag100) vlan300(tag300)
| |
| |
----------- ----------------------
| | | |
| | | |
test1 test2 test3 test4
(tag100) (tag100) (tag300) (tag300)
下面是操作
1、创建ovs1、ovs2网桥，其中em1绑定到ovs1走内网，ovs2是容器的网络
ovs-vsctl add-br ovs1
ovs-vsctl add-br ovs2
ovs-vsctl add-port ovs1 em1
ip link set ovs1 up
ifconfig em1 0
ifconfig ovs1 10.10.17.3
2、创建test1、test2、test3、test4的容器
[[email protected] ~]# sh create_docker_container_use_static_ip.sh test1 docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord 100
{‘Physics_ip‘:‘10.10.17.3‘,‘Container_name‘:‘test1‘,‘Container_ip‘:‘172.16.1.2/24‘,‘Container_vlan‘:‘100‘,‘Container_vlan_gateway‘:‘172.16.1.1/24‘,‘Container_create‘:‘2015-03-02 14:59:59‘,‘Container_status‘:‘running‘}
[[email protected] ~]# sh create_docker_container_use_static_ip.sh test2 docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord 100
{‘Physics_ip‘:‘10.10.17.3‘,‘Container_name‘:‘test2‘,‘Container_ip‘:‘172.16.1.3/24‘,‘Container_vlan‘:‘100‘,‘Container_vlan_gateway‘:‘172.16.1.1/24‘,‘Container_create‘:‘2015-03-02 15:00:09‘,‘Container_status‘:‘running‘}
[[email protected] ~]# sh create_docker_container_use_static_ip.sh test3 docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord 300
{‘Physics_ip‘:‘10.10.17.3‘,‘Container_name‘:‘test3‘,‘Container_ip‘:‘172.16.2.2/24‘,‘Container_vlan‘:‘300‘,‘Container_vlan_gateway‘:‘172.16.2.1/24‘,‘Container_create‘:‘2015-03-02 15:00:22‘,‘Container_status‘:‘running‘}
[[email protected] ~]# sh create_docker_container_use_static_ip.sh test4 docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord 300
{‘Physics_ip‘:‘10.10.17.3‘,‘Container_name‘:‘test4‘,‘Container_ip‘:‘172.16.2.3/24‘,‘Container_vlan‘:‘300‘,‘Container_vlan_gateway‘:‘172.16.2.1/24‘,‘Container_create‘:‘2015-03-02 15:00:31‘,‘Container_status‘:‘running‘}
这个脚本第一个参数是创建的容器名，第二个参数是docker images，第三个参数是运行命令，第四个参数是vlan。
下面是当前docker信息
[[email protected] ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
386b4180b3d0 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor About a minute ago Up About a minute test4
92fee54ec498 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor About a minute ago Up About a minute test3
4dc84301894a docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor About a minute ago Up About a minute test2
45d4a73c6ed0 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 2 minutes ago Up 2 minutes test1
下面是openvswitch的信息
[[email protected] ~]# ovs-vsctl show
d895d78b-8c89-49bc-b429-da6a4a2dcb3a
Bridge "ovs2"
Port "vlan300"
tag: 300
Interface "vlan300"
type: internal
Port "ovs2"
Interface "ovs2"
type: internal
Port "veth1pl11491"
tag: 100
Interface "veth1pl11491"
Port "veth1pl12003"
tag: 300
Interface "veth1pl12003"
Port "vlan100"
tag: 100
Interface "vlan100"
type: internal
Port "veth1pl12251"
tag: 300
Interface "veth1pl12251"
Port "veth1pl11788"
tag: 100
Interface "veth1pl11788"
Port "vlan1"
tag: 1
Interface "vlan1"
type: internal
Bridge "ovs1"
Port "em1"
Interface "em1"
Port "ovs1"
Interface "ovs1"
type: internal
ovs_version: "2.3.1"
下面是etcd的信息
[[email protected] ~]# etcdctl ls / --recursive|grep docker|grep "10.10.17.3/"
/app/docker/ip/vlan/10.10.17.3/gate_way
/app/docker/ip/vlan/10.10.17.3/now_ip
/app/docker/10.10.17.3/vlan300
/app/docker/10.10.17.3/vlan300/container
/app/docker/10.10.17.3/vlan300/container/test3
/app/docker/10.10.17.3/vlan300/container/test4
/app/docker/10.10.17.3/vlan300/gateway
/app/docker/10.10.17.3/vlan300/nowip
/app/docker/10.10.17.3/vlan1
/app/docker/10.10.17.3/vlan1/gateway
/app/docker/10.10.17.3/vlan1/nowip
/app/docker/10.10.17.3/vlan100
/app/docker/10.10.17.3/vlan100/gateway
/app/docker/10.10.17.3/vlan100/nowip
/app/docker/10.10.17.3/vlan100/container
/app/docker/10.10.17.3/vlan100/container/test1
/app/docker/10.10.17.3/vlan100/container/test2
查看test1的信息
[[email protected] ~]# etcdctl get /app/docker/10.10.17.3/vlan100/container/test1
{‘Physics_ip‘:‘10.10.17.3‘,‘Container_name‘:‘test1‘,‘Container_ip‘:‘172.16.1.2/24‘,‘Container_vlan‘:‘100‘,‘Container_vlan_gateway‘:‘172.16.1.1/24‘,‘Container_create‘:‘2015-03-02 14:59:59‘,‘Container_status‘:‘running‘}
跟创建容器的信息一样
进入容器，然后查看设置ip情况
[[email protected] ~]# docker exec test1 ifconfig eth1|grep "inet addr"
inet addr:172.16.1.2 Bcast:0.0.0.0 Mask:255.255.255.0
[[email protected] ~]# docker exec test2 ifconfig eth1|grep "inet addr"
inet addr:172.16.1.3 Bcast:0.0.0.0 Mask:255.255.255.0
[[email protected] ~]# docker exec test3 ifconfig eth1|grep "inet addr"
inet addr:172.16.2.2 Bcast:0.0.0.0 Mask:255.255.255.0
[[email protected] ~]# docker exec test4 ifconfig eth1|grep "inet addr"
inet addr:172.16.2.3 Bcast:0.0.0.0 Mask:255.255.255.0
可以看到ip都是之前设置的
然后在test1里ping外网与test2
[[email protected] ~]# docker exec test1 ping -c1 www.baidu.com
PING www.a.shifen.com (180.149.131.205) 56(84) bytes of data.
64 bytes from 180.149.131.205: icmp_seq=1 ttl=54 time=1.94 ms
--- www.a.shifen.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 12ms
rtt min/avg/max/mdev = 1.948/1.948/1.948/0.000 ms
[[email protected] ~]# docker exec test1 ping -c1 172.16.1.3
PING 172.16.1.3 (172.16.1.3) 56(84) bytes of data.
64 bytes from 172.16.1.3: icmp_seq=1 ttl=64 time=0.431 ms
--- 172.16.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.431/0.431/0.431/0.000 ms
可以看到都是通的
在从test3里ping外网与test4
[[email protected] ~]# docker exec test3 ping -c1 www.baidu.com
PING www.a.shifen.com (180.149.131.236) 56(84) bytes of data.
64 bytes from 180.149.131.236: icmp_seq=1 ttl=54 time=2.19 ms
--- www.a.shifen.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 12ms
rtt min/avg/max/mdev = 2.197/2.197/2.197/0.000 ms
[[email protected] ~]# docker exec test3 ping -c1 172.16.2.3
PING 172.16.2.3 (172.16.2.3) 56(84) bytes of data.
64 bytes from 172.16.2.3: icmp_seq=1 ttl=64 time=0.504 ms
--- 172.16.2.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.504/0.504/0.504/0.000 ms
也是通的
3、关闭test1并测试
[[email protected] ~]# docker stop test1
test1
[[email protected] ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
386b4180b3d0 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 12 minutes ago Up 12 minutes test4
92fee54ec498 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 12 minutes ago Up 12 minutes test3
4dc84301894a docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 12 minutes ago Up 12 minutes test2
45d4a73c6ed0 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 12 minutes ago Exited (0) 2 seconds ago test1
[[email protected] ~]# ping -c2 172.16.1.2
PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data.
--- 172.16.1.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
可以看到test1关闭了，给予的ip也不能通了，下面启动容器，然后使用我自动检测与重新设置容器ip
[[email protected] ~]# docker start test1
test1
[[email protected] ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
386b4180b3d0 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 13 minutes ago Up 13 minutes test4
92fee54ec498 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 13 minutes ago Up 13 minutes test3
4dc84301894a docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 13 minutes ago Up 13 minutes test2
45d4a73c6ed0 docker.ops-chukong.com:5000/centos6-http:new "/usr/bin/supervisor 13 minutes ago Up 2 seconds test1
[[email protected] ~]# python auto_check_modify_container.py
2015-03-02 15:14:03 container need modify container static ip is:[‘172.16.1.2‘]
container_name:test1 container_ip:172.16.1.2/24 modify static ip is success!
可以看到启动容器后，程序自动搜集之前的ip信息，并重新配置了
在test1里测试
[[email protected] ~]# docker exec test1 ifconfig eth1|grep "inet addr"
inet addr:172.16.1.2 Bcast:0.0.0.0 Mask:255.255.255.0
[[email protected] ~]# docker exec test1 ping -c1 www.baidu.com
PING www.a.shifen.com (180.149.131.236) 56(84) bytes of data.
64 bytes from 180.149.131.236: icmp_seq=1 ttl=54 time=2.18 ms
--- www.a.shifen.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 12ms
rtt min/avg/max/mdev = 2.183/2.183/2.183/0.000 ms
[[email protected] ~]# docker exec test1 ping -c1 172.16.1.3
PING 172.16.1.3 (172.16.1.3) 56(84) bytes of data.
64 bytes from 172.16.1.3: icmp_seq=1 ttl=64 time=0.382 ms
--- 172.16.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.382/0.382/0.382/0.000 ms
以上是关于docker固定IP第三种方法的主要内容，如果未能解决你的问题，请参考以下文章