Design a URL Shortening for Splunk

Question

Design a low-latency url shortening system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **URL Shortening**: Users can submit a long URL and receive a short URL in return.
2. **Redirection**: When users access the short URL, they should be redirected to the original long URL.
3. **Analytics**: Track usage metrics for each short URL (clicks, referrer data, etc.).
4. **User Management**: Users can create accounts to manage their URLs.
5. **Expiration**: Users can set expiration dates for their short URLs.

### Non-Functional Requirements
1. **Low Latency**: The system should respond to URL shortening and redirection requests in under 100ms.
2. **Scalability**: The system should handle millions of requests per day with a focus on horizontal scaling.
3. **High Availability**: The system should be operational 99.99% of the time to ensure continuous access to short URLs.
4. **Data Durability**: Shortened URLs and their mappings should persist across server failures.

Assuming Splunk aims to process 1 million URLs per day:
- **Requests Per Second (QPS)**: 
  - 1 million URLs per day translates to approximately 11.57 requests per second (1,000,000 / (24 * 60 * 60)).
- **Storage Requirements**: 
  - If each URL mapping (short URL + long URL) takes about 100 bytes, then:
    - 1 million URLs x 100 bytes = 100 MB of storage per day.
  - Over a year, this would lead to approximately 36.5 GB (100 MB * 365 days).
- **Bandwidth**: 
  - Assuming each redirection request results in a 1 KB payload, the bandwidth required would be:
    - 1 million redirects x 1 KB = 1 GB per day.

### Component Diagram
- **Load Balancer**: Distributes incoming requests across multiple web servers.
- **Web Servers**: Handles URL shortening and redirection requests using a REST API.
  - Technology: Node.js (for low latency) and Express.js framework.
- **Database**: Stores URL mappings.
  - Technology: NoSQL (e.g., MongoDB) for flexible schema and scalability.
- **Cache**: Caches popular short URLs for fast access.
  - Technology: Redis for in-memory caching.
- **Analytics Service**: Asynchronously processes click metrics.
  - Technology: Apache Kafka for event streaming.
- **Monitoring & Alerting**: Uses a combination of Splunk's own product and Prometheus for system health and performance metrics.

### Schema Design
- **URLs Collection**:
  - `short_url`: String (unique identifier for the short URL)
  - `long_url`: String (original URL)
  - `created_at`: Timestamp (when the URL was created)
  - `expires_at`: Timestamp (optional expiration date)
  - `click_count`: Integer (number of times the short URL has been accessed)
  - `user_id`: String (optional user ID if the URL is associated with a user)

### Access Patterns
- **Read**: Fetching the long URL for a given short URL (high volume).
- **Write**: Creating new short URLs (lower volume but requires consistency).

### Consistency vs Availability
- **Consistency**: Given the nature of URL shortening, eventual consistency is acceptable. This allows for faster writes and better availability.
- **Availability**: By using NoSQL, we can achieve higher availability and partition tolerance, but at the cost of immediate consistency.

### Caching Decisions
- Using Redis for caching popular short URLs reduces load on the database. However, it introduces cache invalidation challenges, which need to be managed effectively.

### Partitioning Strategy
- Sharding the database based on the hash of the short URL ensures even distribution of data and allows the system to scale horizontally as the number of URLs increases.

### Monitoring and Alerting
- Implementing alerts for error rates and latency will help maintain the system’s reliability. Using Splunk for logging will provide insights into usage patterns and potential issues.

Design a URL Shortening for Splunk

Splunk

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Splunk Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search