System requirements

Functional:

Traffic Distribution

• Support multiple load-balancing algorithms (e.g., Round Robin, Least Connections, IP Hash).

Session Persistence

Health Monitoring

• Continuously check the health of backend servers using periodic health checks (HTTP, TCP, custom scripts).

SSL/TLS Termination

• Offload SSL decryption/encryption from backend servers.

Scalability & Auto-Scaling

High Availability

Configuration Management

Logging & Monitoring:

• Provide real-time metrics (traffic, errors, latency, active connections).

Security Features:

• Prevent DDoS amplification by rate limiting and filtering malformed requests.
• Support firewall rules and IP whitelisting/blacklisting.

Non-Functional:

Performance:

• • Must handle high throughput (millions of requests per second depending on scale).

Reliability & Availability:

• • 99.99% uptime target with redundant instances.

Scalability:

• • Scale horizontally without significant reconfiguration.

Security:

• • Strong SSL/TLS protocols with support for modern ciphers.
  • Regular patching to protect against vulnerabilities.

Capacity estimation

1. Traffic Volume: 1 million RPS.
2. Request Size: Let's assume an average size of 3KB for both the request and response.
3. Data Throughput: 1 million RPS * 3KB/request = 3GB/sec of incoming data.
4. Web Server Capacity: Assume each server can handle 5,000 RPS.
5. Minimum Server Requirement: 1 million RPS / 5,000 RPS/server = 200 servers. Add extra servers for redundancy, need around 220-250 servers.

API design

Define what APIs are expected from the system...

Database design

Server:

• Keeps track of each server's basic data, including its IP address, port, current status, and the last time it was checked for health.
• Database Type: Relational Database (SQL) - Example: MySQL, PostgreSQL
• Reasoning:
  • Normalized data structure for efficient querying and data integrity.
  • ACID properties ensure data consistency (important for server health and configuration).
  • SQL queries are well-suited for retrieving server and health check data.
• CAP Theorem Focus: Availability and Partition Tolerance (AP)

Metrics/Logs:

• Metrics Table collects data related to the performance of each server, useful for monitoring and triggering auto-scaling events.
• Logs Table archives all types of logs which are crucial for debugging, auditing, and compliance. This can include logs for traffic flow, session updates, health checks, and scaling activities.
• Database (NoSQL) - Example: CassandraDB, Elasticsearch for logs
• Reasoning:
  • Elasticsearch perfectly suited for log analytics due to its complex search capabilities and ability to handle JSON data efficiently.
  • Metrics: A Time-Series Database like InfluxDB would be optimal. They are tailored for handling data characterized by time-stamps efficiently, allowing for rapid ingestion and complex time-based queries.
• CAP Theorem Focus: Availability and Partition Tolerance (AP)

Sessions:

•  Manages user sessions to provide persistence and continuity across requests, ensuring stateful interactions.
• Database (NoSQL) - Example: Redis or Memcached
• Reasoning:
  • Fast read/write operations due to being memory-resident.
  • Ideal for ephemeral data that doesn't need durability (although persistence options exist for Redis).
• CAP Theorem Focus: Availability and Partition Tolerance (AP)

High-level design

1. Client: Initiates requests to the system; represents the end users.
2. Listener : component that handles incoming requests and directs them to the Dispatcher. It serves as the entry point and may include SSL termination. A listener is generally implemented as a network socket that binds to an IP address and a port number. It's responsible for listening for incoming TCP or UDP requests
3. Dispatcher: core routing component that directs requests to the appropriate Web Server based on the load balancing algorithm (e.g., round-robin, least connections).
4. Web Server: server responsible for handling and processing client requests.
5. Health Check: monitors the health of web servers. Feeds back into Dispatcher to assist in traffic routing decisions.
6. Session Manager: manages session persistence and routes requests based on session information to maintain user state.
7. Metric Collectors: gathers performance and usage metrics to assist in system monitoring and diagnostics.
8. Auto Scaler: automatically scales the number of servers up or down based on the metric data and traffic demand.
9. Logger: Records all relevant data, including traffic logs, session details, health checks, and scaling actions for auditing and analysis.

Request flows

Request Flow Steps:

1. Client Request:
   • A client sends a request, typically via a web browser or mobile app.
2. Load Balancer Listener:
   • The request is first received by the Load Balancer's Listener, which listens on a designated port (e.g., port 80 for HTTP, port 443 for HTTPS).
3. Dispatcher:
   • The Listener forwards the request to the Dispatcher, which uses the configured load balancing algorithm (e.g., Round Robin, Least Connections, etc.) to determine which web server should handle the request.
4. Session Management:
   • If sessions are required, the Session Manager checks if there's an existing session or creates one if needed, ensuring that session persistence is maintained across requests.
5. Web Server Processing:
   • The request is routed to the selected Web Server. The server processes the request (e.g., querying a database, executing application logic) and generates a response.
6. Response Logging:
   • As the response is prepared, the system logs the transaction details, including session data, processing time, and any errors that might have occurred for auditing and analysis.
7. Metrics Collection and Monitoring:
   • The metrics collector gathers performance data such as request processing time, server load, active connections, and logs the data for further analysis. This data can trigger scaling actions or alert administrators.
8. Response to Client:
   • Finally, the response is sent back to the client via the Load Balancer, completing the transaction.

Detailed component design

1. Listener

• Receive and Forward: Listens on IP address/port and forwards requests to the Dispatcher.
• SSL/TLS Termination: Handles SSL/TLS offloading for HTTPS traffic.
• Security Filters: Basic request filtering for security measures like rate limiting or IP blocking.
• Connection Management: Handles active connections efficiently and queues requests if needed.

2. Dispatcher

• Load Balancing Logic: Implements algorithms like Round Robin, Least Connections, or IP Hash to distribute incoming requests.
• Session Affinity: Ensures requests from the same session are directed to the appropriate server (Sticky Sessions).
• Error Handling: Manages and reroutes requests if some servers are unresponsive or in maintenance mode.
• Metrics Collection Interface: Interfaces with metrics systems to gather performance data.

3. Session Manager

• Session Tracking: Keeps track of user sessions and maintains state across requests.
• Session Storage: Persists session data using in-memory or persistent storage.
• Authentication and Security: Manages session tokens or cookies and handles session authentication.
• Expiration and Timeout Management: Automatically expires sessions based on inactivity or predefined rules.

4. Web Server

• Request Processing: Executes application logic to handle client requests.
• Data Interaction: Interfaces with databases and services for data retrieval/manipulation.
• Response Generation: Prepares the response to be sent back to the client.

5. Health Check Service

• Monitoring: Periodically checks the health and availability of web servers.
• Alerts and Rerouting: Notifies the system of any server failures and recommends rerouting actions.
• Performance Metrics: Measures and records server performance metrics for analytics.

6. Metrics Collector

• Data Collection: Collects various performance data, including traffic load, server response times, and errors.
• Storage and Analytics: Stores collected data and provides interfaces for real-time analysis and reporting.
• Alert Generation: Triggers alerts based on predefined thresholds to administrators for proactive measures.

7. Auto Scaler

• Scaling Policies: Defines rules for automatically scaling the number of active web servers based on current load.
• Resource Management: Adds/removes servers dynamically to handle varying loads efficiently.
• Integration with Metrics: Interfaces with metrics to make informed scaling decisions.

8. Logger

• Log Management: Records all significant activities, including requests, responses, errors, and security events.
• Search and Analysis: Provides tools for searching through logs and generating insights/reports.
• Compliance and Auditing: Ensures logged data meets compliance for auditing and maintains data integrity.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?