记一次dubbo连接zookeeper注册中心发生异常的排查经历

Posted 2021-03-15 yaofengdoit

????spring boot集成dubbo项目启动过程中报错，错误信息如下：

2020-01-19 17:42:30.466 ERROR - [main] org.springframework.boot.SpringApplication.reportFailure(822) | Application run failed
java.lang.IllegalStateException: zookeeper not connected
    at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.<init>(CuratorZookeeperClient.java:83)
    at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperTransporter.createZookeeperClient(CuratorZookeeperTransporter.java:26)
    at org.apache.dubbo.remoting.zookeeper.support.AbstractZookeeperTransporter.connect(AbstractZookeeperTransporter.java:68)
    at org.apache.dubbo.remoting.zookeeper.ZookeeperTransporter$Adaptive.connect(ZookeeperTransporter$Adaptive.java)
    at org.apache.dubbo.configcenter.support.zookeeper.ZookeeperDynamicConfiguration.<init>(ZookeeperDynamicConfiguration.java:70)
    at org.apache.dubbo.configcenter.support.zookeeper.ZookeeperDynamicConfigurationFactory.createDynamicConfiguration(ZookeeperDynamicConfigurationFactory.java:37)
    at org.apache.dubbo.common.config.configcenter.AbstractDynamicConfigurationFactory.lambda$getDynamicConfiguration$0(AbstractDynamicConfigurationFactory.java:39)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
    at org.apache.dubbo.common.config.configcenter.AbstractDynamicConfigurationFactory.getDynamicConfiguration(AbstractDynamicConfigurationFactory.java:39)
    at org.apache.dubbo.common.config.configcenter.DynamicConfiguration.getDynamicConfiguration(DynamicConfiguration.java:223)
    at org.apache.dubbo.config.bootstrap.DubboBootstrap.prepareEnvironment(DubboBootstrap.java:815)
    at org.apache.dubbo.config.bootstrap.DubboBootstrap.startConfigCenter(DubboBootstrap.java:560)
    at org.apache.dubbo.config.bootstrap.DubboBootstrap.useRegistryAsConfigCenterIfNecessary(DubboBootstrap.java:624)
    at org.apache.dubbo.config.bootstrap.DubboBootstrap.initialize(DubboBootstrap.java:508)
    at org.apache.dubbo.config.bootstrap.DubboBootstrap.start(DubboBootstrap.java:698)
    at org.apache.dubbo.config.spring.context.DubboBootstrapApplicationListener.onContextRefreshedEvent(DubboBootstrapApplicationListener.java:52)
    at org.apache.dubbo.config.spring.context.DubboBootstrapApplicationListener.onApplicationContextEvent(DubboBootstrapApplicationListener.java:45)
    at org.apache.dubbo.config.spring.context.OneTimeExecutionApplicationContextEventListener.onApplicationEvent(OneTimeExecutionApplicationContextEventListener.java:40)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:402)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:359)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:896)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:743)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:390)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1214)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1203)
    at cn.mogutrip.base.foundation.shell.Application.main(Application.java:28)
Caused by: java.lang.IllegalStateException: zookeeper not connected
    at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.<init>(CuratorZookeeperClient.java:80)
    ... 30 common frames omitted

项目中引用的是dubbo-spring-boot-starter，版本配置如下：

<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-spring-boot-starter</artifactId>
    <version>2.7.5</version>
</dependency>

????从启动结果可以看出，dubbo的注册中心zookeeper连接不上了，起初我以为apollo中配置的zookeeper地址错了，核对了一遍没毛病。然后看了下项目中的application.properties配置文件，又看了下启动日志，排除了apollo没生效的可能性。而且项目中使用的Elastic-Job的配置中心zookeeper和dubbo的一样，但是Elastic-Job却启动成功了。伴随了一头雾水，习惯性的百度了下spring boot集成dubbo所遇到的坑，大多数的情况是启动类上没有添加 @EnableDubboConfig 注解，可是我的项目添加了的。尝试了网上的各种坑之后，还是没有解决。然后我让组员在他电脑上启动项目看一下，居然一点问题都没有。那就是服务器zookeeper不稳定喽！

????可是怎么解决这个不稳定呢？从出错的问题更了下源码：

可以发现配置的信息都读取到了的，走到下面这一步：

boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS);

返回的值是false。如果timeout的值变大，是不是就可能为true了呢？上面一步看出timeout=3000。那么timeout哪里来的，我没发现我的配置里有3000。

不断跟进、跟进、跟进：

到了DubboBootstrap类的startConfigCenter方法，已经没有入参了，这里拿到的timeout也是3000，那就只有下面这行代码可以取到：

 Collection<ConfigCenterConfig> configCenters = configManager.getConfigCenters();

继续追踪：

终于看到ConfigCenterConfig类中初始化了timeout = 3000L。原来dubbo的config-center配置中心，默认的获取配置的超时时间是3秒钟。

在项目的配置文件中修改一下这个时间为10秒：

dubbo.config-center.timeout=10000

再次启动项目，完美解决！

总结：遇到问题，还是要从错误的本质出发，多跟进底层源码，加深理解的同时，也学到了其架构思想和perfect的设计。
更多config-center的配置可以参考链接：http://dubbo.apache.org/zh-cn/docs/user/references/xml/dubbo-config-center.html