Design a Load Balancer

Functional

A load balancer is a network appliance that distributes incoming traffic across multiple backend servers so that no single server becomes a bottleneck. It sits between clients and your server fleet, making routing decisions on every request. Think Nginx, HAProxy, or AWS ALB.

Traffic distribution:

Traffic

• Requests: 1M requests/sec peak across the entire fleet. Each LB node handles ~100K requests/sec (limited by NIC throughput and CPU for L7 parsing). Need ~10-12 LB nodes at peak, with headroom for bursts.
• Concurrent connections: Each LB node maintains up to 100K concurrent TCP connections (tuned via kernel parameters: somaxconn, file descriptors). At 1M total connections, 10 nodes suffice. WebSocket connections are long-lived and consume more memory, plan for 50K WebSocket + 50K HTTP per node.
• Health check traffic: 10 LB nodes each probing 500 backend servers every 5 seconds = 1,000 probes/sec total. Negligible compared to production traffic.

Bandwidth

Server Pool Management

POST /api/servers: Register a new backend server. Body: address (IP:port), weight (for weighted algorithms), metadata (datacenter, version, tags). The server enters a "pending" state, receives health checks, and is added to the routing pool only after passing the health threshold. Returns server_id.

DELETE /api/servers/:server_id: Remove a server from the pool. Triggers connection draining: no new requests routed, in-flight requests complete (up to drain_timeout seconds), then full removal. Returns immediately with drain status.

Storage Architecture

A load balancer is unique among system design problems: it has almost no persistent storage needs. The critical data (routing decisions, health status, connection counts) changes on every request and must be accessed in microseconds. This means in-memory data structures and Redis, not PostgreSQL.

The sharper constraint is not which store to pick but how many of them the request path is allowed to touch. The answer here is zero. Every piece of data a routing decision needs is either in the node's own memory or carried by the client in the request itself.

Architecture Overview

SSL Termination Module: The entry point. Decrypts incoming HTTPS using the configured certificate. Outputs plain HTTP to the routing pipeline. For L4 (TCP passthrough) mode, this module is bypassed, traffic is forwarded as-is. Hardware-accelerated TLS (Intel QAT, custom ASICs) handles 10K+ handshakes/sec on high-end LBs.

Standard HTTP Request Flow

Let's trace a request from a returning user carrying an affinity cookie:

Traffic Distribution Algorithms: The Core Challenge

This is the heart of the load balancer. The algorithm must make correct routing decisions at 100K req/sec per node with microsecond overhead, while adapting to changing server health and capacity.

Hardware vs. Software Load Balancers

Hardware LBs (F5 BIG-IP, Citrix ADC): Purpose-built appliances with custom ASICs for SSL/TLS, packet processing, and routing. Handle millions of connections, sub-100μs latency. Pros: extreme performance, vendor support. Cons: $50K-$500K per unit, proprietary configuration, difficult to automate, vendor lock-in. No horizontal scaling, you buy bigger boxes.

Software LBs (Nginx, HAProxy, Envoy): Run on commodity servers. Configured via text files or APIs. Pros: free/open-source, trivially automated (Infrastructure as Code), horizontally scalable, testable in CI/CD. Cons: higher latency than hardware (1-5ms vs under 0.1ms at L4), limited by server NIC and CPU. But at L7, software LBs match or exceed hardware for HTTP routing because HTTP parsing is inherently CPU-bound regardless of hardware.

LB Node Failure: Active-Passive Failover

The load balancer is the single entry point for all traffic. If it fails, everything fails. This is why LB HA is non-negotiable in production.

Detection: The standby node sends heartbeats to the primary every 1 second. Three missed heartbeats (3 seconds) triggers failover. For faster detection: use a separate heartbeat network (out-of-band) to avoid false positives from production traffic congestion.

Adaptive Load Balancing

Current algorithms use static rules (round-robin cycle, connection count). An adaptive algorithm would measure real-time backend performance (response time P50/P99, error rate, CPU utilization) and continuously adjust weights. A server with rising P99 latency gets fewer requests before health checks detect a problem. This is more responsive than passive health checks but requires continuous metrics collection from backends and a feedback control loop (PID controller or reinforcement learning model) that's careful to avoid oscillation.

DDoS Mitigation