Circuit Breakers in Microservices: A Complete Guide with Kotlin Implementation

In distributed systems and microservices architecture, failures are inevitable. Services can become slow, unresponsive, or completely unavailable.

This is where the Circuit Breaker pattern comes in - a crucial design pattern that helps build resilient and fault-tolerant systems. In this guide, we'll explore what circuit breakers are, why they're essential, and how to implement them in Kotlin using Resilience4j.

What is a Circuit Breaker?

The Circuit Breaker pattern is named after the electrical circuit breaker - a switching device that interrupts the flow of current when a fault is detected. In software, it works similarly by preventing an application from repeatedly trying to execute an operation that's likely to fail.

The Three States of a Circuit Breaker

1. CLOSED (Default State):
   • Requests flow through normally
   • Failures are counted
   • When failure threshold is reached, circuit moves to OPEN state
2. OPEN:
   • All requests immediately return with an error
   • After a timeout period, circuit moves to HALF-OPEN state
3. HALF-OPEN:
   • Limited number of test requests are allowed through
   • If these requests succeed, circuit moves to CLOSED state
   • If they fail, circuit returns to OPEN state

Why Use Circuit Breakers?

1. Fail Fast: Instead of waiting for timeouts, fail immediately when a service is down
2. Reduce Load: Prevent overwhelming failing services with requests
3. Quick Recovery: Allow systems to recover without cascade of failures
4. Improved User Experience: Provide immediate feedback instead of making users wait

Implementation with Kotlin and Resilience4j

Let's implement a circuit breaker in a Spring Boot application using Kotlin and Resilience4j.

Setup

First, add the necessary dependencies to your build.gradle.kts:

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("io.github.resilience4j:resilience4j-spring-boot3:2.2.0")
    implementation("org.springframework.boot:spring-boot-starter-aop")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
}

Configuration

Add circuit breaker configuration to application.yml:

resilience4j:
  circuitbreaker:
    instances:
      orderService:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 10000
        permittedNumberOfCallsInHalfOpenState: 3
        automaticTransitionFromOpenToHalfOpenEnabled: true

Let's break down each configuration property:

• slidingWindowSize: 10
  • Defines how many calls are recorded in the sliding window
  • Example: With size 10, only the last 10 requests are considered when calculating the failure rate
• failureRateThreshold: 50
  • The percentage of calls that must fail to open the circuit
  • Example: If set to 50, circuit opens when 5 out of 10 calls fail (given slidingWindowSize of 10)
• waitDurationInOpenState: 10000
  • Time (in milliseconds) the circuit stays in OPEN state before transitioning to HALF-OPEN
  • Example: 10000ms = 10 seconds before attempting recovery
• permittedNumberOfCallsInHalfOpenState: 3
  • Number of calls allowed in HALF-OPEN state to test if the service has recovered
  • Example: Allows 3 test calls to determine if circuit should close or remain open
• automaticTransitionFromOpenToHalfOpenEnabled: true
  • When true, circuit automatically switches from OPEN to HALF-OPEN after waitDuration
  • When false, circuit requires manual intervention to switch states

Additional properties you might want to consider:

resilience4j:
  circuitbreaker:
    instances:
      orderService:
        # Previously defined properties...
        
        # Minimum number of calls required before calculating failure rate
        minimumNumberOfCalls: 5
        
        # Window type: COUNT_BASED or TIME_BASED
        slidingWindowType: COUNT_BASED
        
        # Duration of the sliding window when using TIME_BASED type (in seconds)
        # slidingWindowTime: 60
        
        # Percentage of calls that must succeed to close the circuit
        successThreshold: 60

Basic Implementation

Here's a simple service that uses a circuit breaker:

@Service
class OrderService(
    private val orderClient: OrderClient
) {
    private val logger = LoggerFactory.getLogger(javaClass)

    @CircuitBreaker(name = "orderService", fallbackMethod = "fallbackGetOrder")
    fun getOrder(orderId: String): Order {
        return orderClient.fetchOrder(orderId)
    }

    private fun fallbackGetOrder(orderId: String, ex: Exception): Order {
        logger.error("Circuit breaker activated for order $orderId", ex)
        return Order(orderId, "Fallback Order", OrderStatus.UNKNOWN)
    }
}

Advanced Implementation

Let's create a more robust implementation with custom configuration and monitoring:

@Configuration
class CircuitBreakerConfig {
    @Bean
    fun circuitBreakerRegistry(): CircuitBreakerRegistry {
        val config = CircuitBreakerConfig.custom()
            .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
            .slidingWindowSize(10)
            .failureRateThreshold(50f)
            .waitDurationInOpenState(Duration.ofSeconds(10))
            .permittedNumberOfCallsInHalfOpenState(3)
            .recordExceptions(IOException::class.java, TimeoutException::class.java)
            .build()

        return CircuitBreakerRegistry.of(config)
    }
}

@Service
class OrderServiceAdvanced(
    private val orderClient: OrderClient,
    private val circuitBreakerRegistry: CircuitBreakerRegistry
) {
    private val logger = LoggerFactory.getLogger(javaClass)
    private val circuitBreaker: CircuitBreaker

    init {
        circuitBreaker = circuitBreakerRegistry.circuitBreaker("orderService")
        circuitBreaker.eventPublisher
            .onStateTransition { event ->
                logger.info("Circuit breaker state changed: ${event.stateTransition}")
            }
    }

    fun getOrder(orderId: String): Order {
        return Try.ofSupplier(
            CircuitBreaker.decorateSupplier(circuitBreaker) {
                orderClient.fetchOrder(orderId)
            }
        ).recover { throwable ->
            handleFallback(orderId, throwable)
        }.get()
    }

    private fun handleFallback(orderId: String, throwable: Throwable): Order {
        logger.error("Circuit breaker fallback for order $orderId", throwable)
        return when (throwable) {
            is IOException -> Order(orderId, "Network Error", OrderStatus.ERROR)
            is TimeoutException -> Order(orderId, "Service Timeout", OrderStatus.ERROR)
            else -> Order(orderId, "Unknown Error", OrderStatus.ERROR)
        }
    }
}

Monitoring and Metrics

Add metrics endpoints to monitor your circuit breakers:

@RestController
@RequestMapping("/metrics")
class CircuitBreakerMetricsController(
    private val circuitBreakerRegistry: CircuitBreakerRegistry
) {
    @GetMapping("/circuitbreaker/{name}")
    fun getMetrics(@PathVariable name: String): CircuitBreakerMetrics {
        val circuitBreaker = circuitBreakerRegistry.circuitBreaker(name)
        return CircuitBreakerMetrics(
            state = circuitBreaker.state,
            failureRate = circuitBreaker.metrics.failureRate,
            numberOfFailedCalls = circuitBreaker.metrics.numberOfFailedCalls,
            numberOfSuccessfulCalls = circuitBreaker.metrics.numberOfSuccessfulCalls
        )
    }
}

data class CircuitBreakerMetrics(
    val state: CircuitBreaker.State,
    val failureRate: Float,
    val numberOfFailedCalls: Int,
    val numberOfSuccessfulCalls: Int
)

Testing Circuit Breakers

Here's how to test your circuit breaker implementation:

@SpringBootTest
class OrderServiceTest {
    @Autowired
    private lateinit var orderService: OrderService

    @MockBean
    private lateinit var orderClient: OrderClient

    @Test
    fun `should trigger circuit breaker after multiple failures`() {
        // Simulate failures
        whenever(orderClient.fetchOrder(any()))
            .thenThrow(RuntimeException("Service unavailable"))

        // Make multiple calls to trigger circuit breaker
        repeat(10) {
            val order = orderService.getOrder("123")
            assertThat(order.status).isEqualTo(OrderStatus.UNKNOWN)
        }

        // Verify circuit breaker is open
        verify(orderClient, times(5)).fetchOrder(any())
    }
}

Best Practices

1. Configure Thresholds Carefully: Set appropriate failure thresholds based on your service's normal behavior
2. Implement Fallbacks: Always provide meaningful fallback behavior
3. Monitor Circuit Breakers: Use metrics to track circuit breaker state and performance
4. Test Circuit Breaker Behavior: Include tests for all circuit breaker states
5. Log State Changes: Keep track of when and why circuit breakers trip

Common Pitfalls to Avoid

1. Too Sensitive Settings: Setting failure thresholds too low
2. Insufficient Timeout: Not setting appropriate timeouts
3. Missing Fallbacks: Not implementing proper fallback mechanisms
4. Lack of Monitoring: Not tracking circuit breaker metrics
5. Improper Exception Handling: Not considering which exceptions should trigger the circuit breaker

Conclusion

Circuit breakers are essential for building resilient microservices. They help prevent cascade failures, reduce load on failing services, and improve system reliability. With Kotlin and Resilience4j, implementing circuit breakers becomes straightforward and maintainable.

Remember to:

• Configure circuit breakers appropriately for your use case
• Implement meaningful fallbacks
• Monitor circuit breaker states and metrics
• Test different failure scenarios