System requirements

Functional:

1. Shorten a URL:
   • The system must accept a long URL from the user and generate a unique short URL alias.
   • The system should validate the long URL format before processing.
2. Redirect to Original URL:
   • When a user accesses the short URL, the system must redirect them to the original long URL.
   • The system should handle cases where the short URL does not exist with an appropriate error message.
3. Retrieve Statistics:
   • Users should be able to view analytics on their shortened URLs (e.g., number of clicks, geographical location of users).
   • The system may need to aggregate data over time to provide meaningful insights.
4. User Management (if applicable):
   • Users should be able to create accounts, log in, and manage their shortened URLs.
   • The system should support user authentication and authorization.
5. Custom Short URLs:
   • Users may have the option to specify a custom alias when creating a short URL, provided it’s unique.
6. Expiration of URLs (optional):
   • The system might allow users to set expiration dates for their shortened URLs.
7. Batch Processing:
   • Users should be able to submit multiple long URLs at once and receive corresponding short URLs in a batch.
8. Error Handling:
   • The system must handle and report errors gracefully, such as invalid URLs or server issues.
9. API Interface:
   • An API should be available for developers to programmatically shorten URLs and retrieve statistics.

Non-Functional:

Non-functional requirements are just as important as functional requirements, as they address the system’s performance, security, and other qualities that affect user experience and system sustainability.

1. Performance:
   • The system should have low latency for both generating short URLs and redirecting to the original URLs, ideally under 100 milliseconds for redirects.
   • It should be able to handle at least 10,000 requests per second (RPS) during peak times.
2. Scalability:
   • The system must be able to scale horizontally to accommodate increasing traffic and storage requirements without significant degradation of performance.
3. Reliability:
   • The system should ensure high availability, aiming for at least 99.9% uptime.
   • Error rates for redirection should be minimal, ideally less than 0.1%.
4. Security:
   • The system should protect against common vulnerabilities such as SQL injection, XSS, and CSRF.
   • Authorization and authentication measures should be in place to protect user accounts and data.
5. Maintainability:
   • The codebase should be organized and documented, allowing for easy updates and modifications.
   • Support for logging and monitoring should be in place for easy debugging and system health checks.
6. Usability:
   • The user interface (if applicable) should be intuitive and easy to navigate, allowing users to quickly shorten links and access features.
   • The API should be well-documented, providing clear instructions and examples for developers.
7. Data Integrity:
   • The system should ensure the integrity of the mappings between short URLs and long URLs, safeguarding against data loss or corruption.
8. Backup and Recovery:
   • Regular backups of the database should be implemented, along with a recovery strategy to restore data in case of failures.

Capacity estimation

1. Traffic Estimation:
   • Estimate the number of users accessing the service daily. Let's say you expect 100,000 unique users per day.
   • If each user shortens an average of 3 URLs per day, you would have:
     • Total URL Shortening Requests per Day:
     • [ 100,000 \text{ users} \times 3 \text{ URLs/user} = 300,000 \text{ URLs/day} ]
2. Concurrency Estimation:
   • Assume peak usage times when user activity is high. Let's say you estimate that during peak hours, around 20% of users are active at the same time.
   • If we consider 12 hours of peak usage, the number of concurrent users can be calculated as:
     • Concurrent Users:
     • [ 100,000 \text{ users} \times 0.20 = 20,000 \text{ users} ]
3. Request Rate:
   • Calculate the request rate during peak times. If each active user makes 5 requests per hour on average, you can estimate the requests per second:
     • Requests Per Hour:
     • [ 20,000 \text{ users} \times 5 \text{ requests/user} = 100,000 \text{ requests/hour} ]
   • Requests Per Second:
   • [ \frac{100,000 \text{ requests/hour}}{3600 \text{ seconds/hour}} \approx 28 \text{ requests/second} ]
4. Storage Requirements:
   • Each URL mapping might take up around 100 bytes (for the long URL and the short alias). If you are generating 300,000 short URLs per day:
     • Total Storage per Day:
     • [ 300,000 \text{ URLs} \times 100 \text{ bytes} = 30,000,000 \text{ bytes} = 30 \text{ MB/day} ]
   • Assuming you want to keep data for 5 years, the total storage required would be:
     • Total Storage:
     • [ 30 \text{ MB/day} \times 365 \text{ days/year} \times 5 \text{ years} = 54,750 \text{ MB} \approx 54.75 \text{ GB} ]

Summary of Capacity Estimation

• Total daily URL shortening requests: 300,000
• Peak concurrent users: 20,000
• Requests per second during peak: 28 Requests/Second
• Total storage required for 5 years: 54.75 GB

API design

1. Shorten URL API

• Endpoint: /shorten
• Method: POST
• Request Body:

{ "longUrl": "https://www.example.com/some/very/long/url", "customAlias": "optional-custom-alias", "userId": "optional-user-id-for-authenticated-users"}

• Response:

{ "shortUrl": "https://short.url/abc123", "shortUrlExpire": "2023-12-31T23:59:59Z" // If expiration is set}

2. Redirect Short URL API

• Endpoint: /redirect/{shortUrl}
• Method: GET
• Path Parameter:
  • shortUrl: The unique alias of the shortened URL
• Response:
  • Redirects to the original long URL; if the URL doesn’t exist, return a 404 response.

3. Retrieve URL Statistics API

• Endpoint: /stats/{shortUrl}
• Method: GET
• Path Parameter:
  • shortUrl: The unique alias of the shortened URL
• Response:

{ "shortUrl": "https://short.url/abc123", "originalUrl": "https://www.example.com/some/very/long/url", "clicks": 150, "createdAt": "2023-01-01T00:00:00Z", "lastAccessed": "2023-11-01T10:00:00Z"}

4. User Registration API (optional)

• Endpoint: /register
• Method: POST
• Request Body:

{ "username": "user123", "password": "securePassword", "email": "[email protected]"}

• Response:

{ "message": "User registered successfully", "userId": "newly-created-user-id"}

5. User Login API (optional)

• Endpoint: /login
• Method: POST
• Request Body:

{ "username": "user123", "password": "securePassword"}

• Response:

{ "token": "jwt-token-for-authentication"}

6. Get User's Shortened URLs API (optional)

• Endpoint: /user/urls
• Method: GET
• Headers:
  • Authorization: Bearer {token}
• Response:

[ { "shortUrl": "https://short.url/abc123", "originalUrl": "https://www.example.com/some/very/long/url", "createdAt": "2023-01-01T00:00:00Z" }, { "shortUrl": "https://short.url/def456", "originalUrl": "https://www.example2.com/another/long/url", "createdAt": "2023-01-02T00:00:00Z" }]

Summary of APIs

• The main API for shortening URLs is the /shorten endpoint.
• The redirect functionality is handled through the /redirect/{shortUrl} endpoint.
• Optional APIs for user management and retrieval of URL statistics enhance the service's functionality.

Database design

Entities

1. User:
   • Represents users of the system, especially if you want to allow user accounts.
2. URL:
   • Holds the mapping between the short URL and the long URL.
3. URL Stats:
   • Contains statistics for each shortened URL, such as click counts and timestamps.

ER Diagram

Here's an ER diagram representing these entities and their relationships:

USERstringidPrimary KeystringusernameUser's usernamestringpassword_hashHashed passwordstringemailUser's emaildatecreated_atRegistration dateURLstringidPrimary Keystringlong_urlOriginal long URLstringshort_urlGenerated short URLstringuser_idForeign Key referencing USERdatecreated_atDate URL was shorteneddateexpires_atExpiration date (if applicable)URL_STATSstringidPrimary Keystringurl_idForeign Key referencing URLintclick_countNumber of clicksdatelast_accessedLast accessed timestampcreateshas

Explanation of the Entities

1. USER: This table stores user information, including their username, hashed password, email, and registration date.
   • id: Unique identifier for each user.
   • username: Username chosen by the user.
   • password_hash: Securely stored hashed version of the password.
   • email: User's email address.
   • created_at: When the user registered.
2. URL: This table captures the relationship between the long URL and its shortened version.
   • id: Unique identifier for each URL entry.
   • long_url: The original long URL that needs shortening.
   • short_url: The generated short URL alias.
   • user_id: References the user who created the short URL.
   • created_at: When the URL was shortened.
   • expires_at: Optional expiration date for the shortened URL.
3. URL_STATS: This table maintains statistics for each shortened URL, allowing the retrieval of click counts and last accessed dates.
   • id: Unique identifier for each statistics record.
   • url_id: References the shortened URL's entry in the URL table.
   • click_count: Tracks the total number of times the short URL was accessed.
   • last_accessed: Records when the URL was last accessed.

Summary

• The ER diagram outlines how the data will be organized and related in the database.
• Each entity captures essential information for the URL shortening service, enabling efficient access and management of data.

High-level design

For a high-level design of a URL shortening service, we need to identify the key components required to handle all the functionalities we've discussed, ensuring that the system operates efficiently from end to end.

Key Components

1. API Gateway:
   • Acts as the entry point for all client requests, routing them to appropriate services.
2. URL Shortening Service:
   • Responsible for generating the short URLs from long URLs and handling the logic for custom aliases.
3. Database:
   • Stores user information, URL mappings, and statistics.
4. Redirect Service:
   • Handles incoming requests for short URLs and redirects users to the original URLs stored in the database.
5. Analytics Service:
   • Collects and provides statistics about how many times each URL has been accessed, providing insights to users.
6. User Management Service (optional):
   • Manages user accounts, authentication, and authorization for logged-in users.
7. Monitoring and Logging:
   • Monitors service performance and logs requests for debugging and analytics.

High-Level Block Diagram

Here's a block diagram that illustrates these components and how they interact:

HTTP Requests

Shorten URL

Save Mapping

Redirect

Lookup URL

Return Long URL

Redirect User

Analytics Data

User Requests

Collect Data

Client

API Gateway

URL Shortening Service

Database

Redirect Service

Analytics Service

User Management Service

Monitoring and Logging

Explanation of the Block Diagram

1. Client: End users interact with the system through a client application (web or mobile) that sends HTTP requests.
2. API Gateway:
   • It routes requests to the appropriate services (URL shortening, redirect, user management).
3. URL Shortening Service:
   • Handles the logic for creating and storing short URLs, returning the shortened URL to the API Gateway.
4. Database:
   • Stores user accounts, URL mappings, and statistics. It serves both the URL Shortening Service and the Redirect Service.
5. Redirect Service:
   • Processes requests for short URLs, looks them up in the database, and performs the redirection.
6. Analytics Service:
   • As URL accesses occur, this service collects data about usage patterns and statistics for each shortened URL.
7. User Management Service:
   • Manages user registration, login, and the retrieval of a user's shortened URLs.
8. Monitoring and Logging:
   • Observes the system's health, logs requests, and can trigger alerts in case of anomalies.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?