Architect a event-driven Load Balancing Engine

Question

Design a event-driven load balancing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Request Handling:** The system must process millions of incoming requests per second (QPS) efficiently.
2. **Load Balancing:** The system should intelligently distribute incoming requests across multiple backend servers based on current load and request type.
3. **Event-Driven Architecture:** The system should respond to events (such as server health checks, new server instances, etc.) to adjust load balancing dynamically.
4. **Monitoring:** Provide real-time analytics on request distribution, server health, and system performance.

### Non-Functional Requirements:
1. **Scalability:** The system should scale horizontally, adding new servers without downtime.
2. **High Availability:** The system should be available 99.99% of the time, with minimal downtime during maintenance.
3. **Performance:** Response time for load balancing decisions should be under 100 milliseconds.
4. **Security:** Implement authentication and authorization for incoming requests.

Assuming Databricks expects to handle **10 million requests per second** at peak:
- **Storage Needs:** If each request generates an average of 1 KB of data for logging, we would require:
  - 10M requests/s * 1 KB/request = 10 GB/s of storage.
  - For 24 hours of logging, that’s 864 TB of data per day.

- **Bandwidth:** Assuming a response size of 200 bytes and a 1:10 request to response ratio:
  - 10M requests/s * 200 bytes = 2 GB/s outbound bandwidth.

- **Server Capacity:** If each server can handle 100,000 requests per second:
  - Total servers needed = 10M / 100K = 100 servers at peak.

### Component Diagram:
- **API Gateway:** A cloud-based API gateway (e.g., AWS API Gateway) to handle incoming requests and direct them to the appropriate service.
- **Load Balancer:** An event-driven load balancer (e.g., AWS Elastic Load Balancer or NGINX) to distribute requests to backend services based on the load.
- **Event Bus:** A messaging system (e.g., Kafka or RabbitMQ) to handle event-driven communication among components.
- **Backend Services:** Multiple microservices (e.g., built with Spring Boot or Node.js) to process requests, managed in a Kubernetes cluster for orchestration.
- **Database:** A distributed database (e.g., Cassandra or DynamoDB) for storing persistent data, with a caching layer (e.g., Redis) for frequently accessed data.

### Schema Design:
1. **Requests Table:**
   - `request_id (UUID)` - Unique identifier for each request.
   - `timestamp (DATETIME)` - Time of the request.
   - `client_id (STRING)` - Identifier for the client making the request.
   - `status (STRING)` - Status of the request (e.g., 'success', 'failed').
   - `response_time (FLOAT)` - Time taken to process the request.

2. **Servers Table:**
   - `server_id (STRING)` - Unique identifier for each server.
   - `current_load (INTEGER)` - Current load on the server.
   - `status (STRING)` - Health status of the server (e.g., 'active', 'down').

3. **Logs Table:**
   - `log_id (UUID)` - Unique identifier for each log entry.
   - `request_id (UUID)` - Related request ID.
   - `log_message (STRING)` - Details of the log entry.

### Architectural Decisions:
1. **Consistency vs. Availability:** We prioritize availability by using an eventually consistent database like Cassandra, allowing for faster read/write operations while sacrificing immediate consistency.
2. **Load Balancing Strategy:** Implementing a round-robin load balancing strategy for simplicity and low latency, but it could lead to uneven load distribution if backend services have varying processing capabilities.
3. **Event Handling:** Using an event-driven architecture allows for better responsiveness but introduces complexity in managing event ordering and potential event loss.
4. **Caching vs. Database Calls:** The decision to use Redis as a caching layer increases performance for frequently accessed data but requires strategies for cache invalidation to ensure data accuracy.

Architect a event-driven Load Balancing Engine

Databricks

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Submit Your Answer

Databricks Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search