System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

1. URL Shortening: Given a long URL, the system should generate a unique shortened alias.
2. Redirection: When users access the shortened alias, they should be redirected to the original long URL.
3. Custom Alias: Option for users to create custom aliases for their URLs.
4. Usage Analytics: Track and provide analytics on the shortened URLs, such as the number of clicks.
5. Expiration: Ability to set expiration dates for shortened URLs.
6. API: Provide APIs for shortening URLs programmatically.
7. User Authentication: Authenticate users who want to create custom aliases or track URL analytics.

Non-Functional:

List non-functional requirements for the system...

1. Scalability: The system should be able to handle a large number of URL shortening requests and redirects efficiently.
2. Availability: The service should be highly available to ensure users can access shortened URLs without downtime.
3. Performance: The redirection from the shortened alias to the original URL should be fast and seamless.
4. Security: Ensure that the system is secure against malicious attacks, such as injection attacks or unauthorized access to URLs.
5. Persistence: Store shortened URLs and analytics data reliably to prevent data loss.
6. Monitoring: Implement monitoring and logging to track service performance and detect issues proactively.
7. Compliance: Adhere to privacy regulations and best practices in handling user data and URLs.

Capacity estimation

Estimate the scale of the system you are going to design...

Traffic: 1.000.000.000 / 86.400 (~100.000) = 10.000 requests per second

Number of shortened url's: 365.000.000.000 shortened url's a year

One url size on average: 100 bytes

365.000.000.000 * 100 bytes = 36.500.000.000.000 bytes = 36.500.000.000 kb = 36.500.000 mb = 36.500 gb = 36,5 tb storage a year

Backup across 3 blob storages = 36,5 tb * 3 = 100~ tb

API design

Define what APIs are expected from the system...

GET /api/v1/url {

"shortened_url": ,

}

Response:

200: {

"shortened_url"

"redirect": ,

"expiry": ,

"visitor_count":

}

4xx: {

"error":

}

POST /api/v1/url {

"redirect": ,

"vanity_url": ,

"expiry": ,

"idempotent_key": # to avoid duplicate requests

}

Response:

201: {

"shortened_url"

}

4xx: {

"error":

}

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

classDiagram

    class ShortenedUrl{

      +String ownerUuid

      +String shortenedUrl

      +String redirect

      +int expiry

      +int visitor_count

    }

    class User{

        +String uuid

        +String email

    }

    User<--ShortenedUrl

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

flowchart TB

    CDN -- Retrieves Static Files --> Client

    Client -- Sends Request --> LB1(Load Balancer 1)

    LB1 -- Sends Request --> Backend1

    LB1 -- Sends Request --> Backend2

    LB1 -- Sends Request --> Backend3

    Backend1 -- Queries --> LB3(Load Balancer 2)

    Backend2 -- Queries --> LB3

    Backend3 -- Queries --> LB3

    LB3 -- Writes --> Relational_DB_Master(Master Database)

    LB3 -- Reads --> Relational_DB_Slave1(Slave Database 1)

    LB3 -- Reads --> Relational_DB_Slave2(Slave Database 2)

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

sequenceDiagram

participant ClientB_Slave2 as SlaveDB2

Client ->> LB1: Send Request

LB1 ->> Backend1: Forward Request

Backend1 ->> LB3: Query

LB1 ->> Backend2: Forward Request

Backend2 ->> LB3: Query

LB1 ->> Backend3: Forward Request

Backend3 ->> LB3: Query

LB3 ->> Relational_DB_Master: Write

LB3 ->> Relational_DB_Slave1: Read

LB3 ->> Relational_DB_Slave2: Read

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

For the client, we will use a CDN to serve the static files. Static files can include the frontend application which users can generate shortenend URLs. CDN also have the benefit to make the client available globally, providing edge servers near users for fast website loads.

For the backend, these will be stateless servers which will allow you to horizontally scale which will be able to handle the 10.000 requests per second. A web server on average can handle 10.000 requests per second on it's own. But for resillience, you should add additional servers to ensure all the requests are served.

For the database, I chose a relational database because our workload is read-heavy. Additionally, we have a user to shortenedUrl relationship which relational database can help maintain data consistency. Lastly, we can use a master-slave architecture to improve read performance for the database. We can also add shard partitioning to distribute the shortenedUrl's in multiple databases, further horizontall scaling the system.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

I'm choosing SQL instead of NoSQL because there are data relations in my system which I'd like to keep consistent.

I'm using CDN to improve speed at the cost of the system becoming more expensive. However, it will result in less traffic to my backend servers.

I'm using a second load balancer between the backend and database so the backends only need to be aware of the second load balancer while the second load balancer takes care of distributing the queries among the slaves.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Replicating relational databases can be difficult to do in longer term. You'd have to redistribute the data if the workload is unevenly distributed. Additionally, some url's can be really popular, so adding some sort of caching to these url's will significantly help.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

Lastly, we can add multi-region redundancy by replicating the setup to a different region. If we choose active-active redundancy, in an event of a region failure, we can quickly switch to the other region using DNS load balancing. However, active-active redundancy basically duplicates the cost.

We could otherwise add a active-passive redundancy where in an event of a failure, it takes time for the passive region to spin up. But this isn't as expensive as the active-active setup.