Hystrix 熔断机制原理

Posted 2020-11-03 Jabnih

相关配置

circuitBreaker.enabled  是否开启熔断
circuitBreaker.requestVolumeThreshold  熔断最低触发请求数阈值
circuitBreaker.sleepWindowInMilliseconds  产生熔断后恢复窗口
circuitBreaker.errorThresholdPercentage  错误率阈值
circuitBreaker.forceOpen  强制打开熔断
circuitBreaker.forceClosed  强制关闭熔断

状态图

执行流程

命令执行前调用circuitBreaker.attemptExecution()，正常情况下会执行返回true，但是如果发生熔断，则需要通过sleepWindows来进行恢复

public boolean attemptExecution() {
    if (properties.circuitBreakerForceOpen().get()) {
        return false;
    }
    if (properties.circuitBreakerForceClosed().get()) {
        return true;
    }
    if (circuitOpened.get() == -1) {
        return true;
    } else {
        if (isAfterSleepWindow()) {
            if (status.compareAndSet(Status.OPEN, Status.HALF_OPEN)) {
                //only the first request after sleep window should execute
                return true;
            } else {
                return false;
            }
        } else {
            return false;
        }
    }
}

发生熔断流程

在新版本1.5.12中，会有一个后台线程订阅metrics流实时计算：

1. 如果没有达到RequestVolume，则直接返回，不计算是否需要熔断

2. 如果当前错误率大于设置的阈值，则触发熔断，状态由CLOSED切换到OPEN，并设置当前熔断的时间（用于后续SleepWindows判断使用）

   if (hc.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
   // we are not past the minimum volume threshold for the stat window,
   // so no change to circuit status.
   // if it was CLOSED, it stays CLOSED
   // if it was half-open, we need to wait for a successful command execution
   // if it was open, we need to wait for sleep window to elapse
   } else {
   if (hc.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
       //we are not past the minimum error threshold for the stat window,
       // so no change to circuit status.
       // if it was CLOSED, it stays CLOSED
       // if it was half-open, we need to wait for a successful command execution
       // if it was open, we need to wait for sleep window to elapse
   } else {
       // our failure rate is too high, we need to set the state to OPEN
       if (status.compareAndSet(Status.CLOSED, Status.OPEN)) {
           circuitOpened.set(System.currentTimeMillis());
       }
   }
   }

熔断恢复流程

 当发生熔断，达到SleepWindows指定时间后，则状态会由OPEN转换为HALF_OPEN，此时只会让一个请求通过，如果该请求执行成功，状态则会转换为CLOSED，否则会回到OPEN状态，并且熔断时间设置为当前时间，重新等待SleepWindows的时间

// 执行成功后调用
public void markSuccess() {
    if (status.compareAndSet(Status.HALF_OPEN, Status.CLOSED)) {
        //This thread wins the race to close the circuit - it resets the stream to start it over from 0
        metrics.resetStream();
        Subscription previousSubscription = activeSubscription.get();
        if (previousSubscription != null) {
            previousSubscription.unsubscribe();
        }
        Subscription newSubscription = subscribeToStream();
        activeSubscription.set(newSubscription);
        circuitOpened.set(-1L);
    }
}

// 执行失败后调用
public void markNonSuccess() {
    if (status.compareAndSet(Status.HALF_OPEN, Status.OPEN)) {
        //This thread wins the race to re-open the circuit - it resets the start time for the sleep window
        circuitOpened.set(System.currentTimeMillis());
    }
}

参考

1. https://github.com/Netflix/Hystrix/wiki/Configuration
2. https://github.com/Netflix/Hystrix/wiki/How-it-Works#CircuitBreaker
3. HystrixCircuitBreaker源码