spring boot 项目启动 com.netflix.hystrix.exception.HystrixRuntimeException:timed-out and fallback failed

Posted 2023-02-14 回归心灵

项目启动报错

2022-03-04 14:26:40.643  WARN 22212 --- [  restartedMain] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deviceTypeService' defined in class path resource [so/dian/type/config/DeviceTypeConfig.class]: Invocation of init method failed; nested exception is java.lang.Exception: get deviceType list failed with:XdcloudGatewayClient#getDeviceTypeList(String) timed-out and no fallback available.

​

线程dump 信息

​

"hystrix-xdcloud-gateway-1" #245 daemon prio=5 os_prio=31 tid=0x00007f972d11e800 nid=0x17903 waiting for monitor entry [0x000000030df74000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:179)
        - waiting to lock <0x00000006c0c2c5a0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:517)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:527)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:491)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:230)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:238)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:223)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:213)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchingBeans(OnBeanCondition.java:167)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:142)
        at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)
        at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:184)
        at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:144)
        at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:120)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:337)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:242)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:275)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:95)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:706)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
        - locked <0x00000007be047d80> (a java.lang.Object)
        at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:136)
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:101)
        - locked <0x00000006c1dcd7e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.CachingSpringLoadBalancerFactory.create(CachingSpringLoadBalancerFactory.java:60)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.lbClient(LoadBalancerFeignClient.java:120)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:82)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.hystrix.HystrixInvocationHandler$1.run(HystrixInvocationHandler.java:109)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OperatorSubscribeOn$SubscribeOnSubscriber.call(OperatorSubscribeOn.java:100)
        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56)
        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47)
        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69)
        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000007bdbba878> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"main" #1 prio=5 os_prio=31 tid=0x00007f974b00d800 nid=0x1903 waiting on condition [0x00000003058da000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007be160dd0> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at rx.internal.operators.BlockingOperatorToFuture$2.get(BlockingOperatorToFuture.java:101)
        at com.netflix.hystrix.HystrixCommand$4.get(HystrixCommand.java:423)
        at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
        at feign.hystrix.HystrixInvocationHandler.invoke(HystrixInvocationHandler.java:170)
        at com.sun.proxy.$Proxy204.getDeviceTypeList(Unknown Source)
        at so.dian.type.service.impl.DeviceTypeServiceImpl.refreshDeviceType(DeviceTypeServiceImpl.java:53)
        at so.dian.type.service.impl.DeviceTypeServiceImpl.afterPropertiesSet(DeviceTypeServiceImpl.java:80)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$184/1260634890.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x00000006c0c2c5a0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
        - locked <0x00000006c065a960> (a java.lang.Object)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
        at so.dian.idmon.IdmonApplication.main(IdmonApplication.java:16)

   Locked ownable synchronizers:
        - None

线程 BLOCKED

main 线程和hystrix-xdcloud-gateway-1 线程死锁，main 线程初始化 bean 时调用 getSingleton() 方法取得锁：synchronized (this.singletonObjects) ，远程调用时 Hystrix 封装 HystrixCommand 线程进入 WAITING 状态，等待结果返回，此时并未释放锁。hystrix-xdcloud-gateway-1 线程会异步执行真正的远程过程调用，此时调用时需要创建 Ribbon Context bean，创建过程需要调用 AbstractApplicationContext.refresh() 方法，也就会
调用DefaultSingletonBeanRegistry.getSingleton() , 此时需要获取锁synchronized (this.singletonObjects) ，由于 main 线程还未释放锁，所以hystrix-xdcloud-gateway-1 会一直处于 BLOCKED 状态，直到超过 hystrix 设置的超时时间，就会抛出 timed-out and no fallback available 异常。
​

注意设置 hystrix 超时时间

时间设置长点可以 dump 到线程信息

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 10000

解决方案

spring 初始化时，禁用远程调用 FeignClient Hystrix 。
​

package so.dian.type.config;

import feign.Feign;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Scope;

/**
 * @Author: yrs
 * @Date: 2022/3/4 12:02 上午
 * @Description:
 */
@Configuration
public class FeignNoHystrixConfigure 

    @Bean
    @Scope("prototype")
    public Feign.Builder feignBuilder() 
        return Feign.builder();
    

package so.dian.type.client;

import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import so.dian.type.config.FeignNoHystrixConfigure;
import so.dian.type.model.DeviceTypeResultDto;


@FeignClient(value = "xdcloud-gateway", contextId = "deviceTypeXdcloudGateway", configuration = FeignNoHystrixConfigure.class)
public interface XdcloudGatewayClient 

    @GetMapping("/sunflower/ota/deviceType/allList")
    DeviceTypeResultDto getDeviceTypeList(@RequestParam("t") String t);