Requirements

Functional Requirements:

• Distribute incoming HTTP/HTTPS requests across backend servers
• Perform health checks and exclude unhealthy instances
• Support multiple load balancing strategies (round-robin, least connections, etc.)
• Provide SSL termination
• Support optional session persistence (sticky sessions)
• Allow dynamic backend registration/deregistration

Non-Functional Requirements:

• High availability
  • Load balancer itself must not be a single point of failure (replication, failover)
• Scalability
  • Must handle ~1M RPS and millions of concurrent connections
• Low latency
  • Minimal overhead added per request (routing must be fast, no heavy logic)
• Fault tolerance
  • Should continue operating despite backend failures or partial outages

API Design

• POST /backends
  • Register a new backend server
• DELETE /backends/{id}
  • Remove a backend server
• GET /backends
  • List all backend servers + health status
• PUT /config
  • Update load balancing strategy (round robin, least connections, etc.)

High-Level Design

Core components:

• Client
  • Sends HTTP/HTTPS requests
• Load Balancer (entry point)
  • Terminates SSL
  • Receives all incoming traffic
• Routing / Decision Engine
  • Selects backend using algorithm (round-robin, least connections, etc.)
  • Applies session persistence if needed
• Backend Server Pool
  • Set of web servers handling requests
• Health Check Service
  • Continuously monitors backend health
  • Updates routing decisions

Request flow (clean version)

• Client sends request → Load Balancer
• LB terminates SSL and forwards request to Routing Engine
• Routing Engine selects a healthy backend
• Request is forwarded to backend server

Detailed Component Design

Routing Engine

Something like:

• Default to round-robin for simplicity and low overhead
• Use least-connections when dealing with uneven or long-lived workloads
• Use hash-based routing when session persistence is required
• Always select from the set of healthy backends maintained by health checks

Health Check Service

• Health Check Service
  • Periodically sends requests to each backend:
    • e.g., GET /health or TCP check
  • Runs every N seconds (e.g., 5s)
• Failure detection
  • Do not fail on a single error
  • Use threshold:
    • e.g., 3 consecutive failures → mark unhealthy
  • Same for recovery:
    • e.g., 2–3 consecutive successes → mark healthy again
• State propagation
  • Maintains a shared in-memory list of healthy backends
  • Routing engine reads from this (or gets updates via push)
• On failure
  • Backend is removed from routing pool immediately
  • New requests stop going there
  • Existing connections may:
    • continue (graceful) OR
    • be terminated (depends on policy)

How to scale the load balancer

Clean design

• Run multiple load balancer instances
• All are active and serving traffic
• No leader election needed

How do clients reach them?

This is the key part you were missing clarity on.

You typically use:

• DNS-based load balancing
  • api.example.com → multiple LB IPs
  • DNS returns multiple IPs (round-robin or weighted)

OR

• Anycast (more advanced)
  • Same IP advertised from multiple locations
  • network routes to nearest LB

Stick Sessions

• Option 1: Consistent Hashing (e.g., based on client IP or session ID)
  • Same input → same backend
  • Minimal remapping when nodes change
  • Trade-off: session loss when a node goes down