Design Circuit-Breaker-Aware Database Routing

Question

Design a database routing system that incorporates circuit breaker patterns to manage requests to a primary database and its replicas.

Codemia · Accepted Answer

To design a circuit-breaker-aware database routing system, we need to ensure that requests are directed to the primary database and, in case of failure, to the secondary replicas. Each database will have its own circuit breaker to monitor the health of the connection. The system should open the circuit after a predefined number of consecutive failures and close it after a successful request.

The system will consist of a routing layer that interacts with the primary and secondary databases. Each database will have a circuit breaker that tracks the status of requests. When a request fails, the circuit breaker increments a failure count. If the count exceeds a threshold, the circuit breaker opens, and subsequent requests are routed to the next available replica until the circuit is closed again.

In our implementation, we will define a `CircuitBreaker` class that maintains the state of the circuit (open, closed, half-open) and the failure count. The routing logic will check the state of the circuit breaker before sending requests. If the circuit is open, it will skip the primary database and attempt to access the secondary replicas. Here is a simplified version of the code:

```python
class CircuitBreaker:
    def __init__(self, failure_threshold=2):
        self.state = 'CLOSED'
        self.failure_count = 0
        self.failure_threshold = failure_threshold

def call(self, func):
        if self.state == 'OPEN':
            raise Exception('Circuit is open')
        try:
            result = func()
            self.reset()
            return result
        except Exception:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self.open_circuit()
            raise

def open_circuit(self):
        self.state = 'OPEN'

def reset(self):
        self.failure_count = 0
        self.state = 'CLOSED'
```

One trade-off of using circuit breakers is the potential for increased latency during failover scenarios, as requests may take longer to route to secondary databases. Additionally, the system must be designed to handle a large number of concurrent requests, which may require scaling the routing layer. Monitoring metrics such as response times and failure rates will be crucial for optimizing performance and ensuring reliability.

Design Circuit-Breaker-Aware Database Routing

Databricks

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Requirements

High-Level Design

Submit Your Answer

Databricks Software Engineer Interview Guide

Related Questions

Design a scalable Search System

Design a Video Streaming for Rippling

Design a Load Balancing Service

Design a Image Processing Service

Build a high-throughput Image Processing Pipeline