Loading...
The main function of the URL shortening service is for user to convert longer URLs to shorter URLs, it will create an alias for the current URL and use the shorter one to redirect to the site.
Above is the main functional requirement, but we also need users to keep a record of the URL they have, the shorter version of it, so the shorted URL will work as long as our service exists.
In conclusion, we need to redirect URLs, and keep a record of the key value pair for the original URL and shorted URL for each user.
The non-functional requirement has the following:
we need to make sure the user can store and retrieve the shorter URL quickly and reliably
we have to make sure the system scales well if there are many users doing the operation at the same time.
we can estimate during peak hours, there will be 1000 requests per second, each key-value pair will be 1 KB in size.
Then we do some calculations, one minute will have 60000 requests, and let's say maybe half of them are redirecting, so the other half will be storing the key-value pair. Thus there will be 500 requests per second to store key-value pairs, and that results in 500 KB each second, for a minute we will have 60 * 500 KB = 30000 KB, an hour we will have 30000KB * 60 = 1800000 KB, which is 2 GB per hour, that is 48 GB per day and 1440 GB per month.
The total storage needed needs to be enough in the long term. So several TB would be a good starting point to handle potential future growth.
For bandwidth, if we have 1000 RPS, and each request/response packet size is 1KB, so the bandwidth will be at least 1 MB/S.
There will be multiple servers to handle the large traffic, and load balancing will be used to direct traffic in between.
There are several APIs, and the core one is on the APIs for shortening
Getting short URL:
POST '/api/shorten'
{"long_url":"www.original.com/longggggg"}
and the response will be
{"short_url":"www.short_url.com/hash_value"}
Redirect:
GET '/api/urls/hash_value'
{"URL":"www.original.com/longggggg"}
And there will be other CRUD APIs on the User information. Since it's not the core functionalities, we will skip it.
I am thinking the database can be Relational database, there will be two tables, one is User table, the other one is URL table with a foreign key to the User to indicate which user has the current URL.
The user table has info about user like User ID, name, email, encrypted password, DOB
URL table will have columns like User (foreign key), URL_ID, long URL, and shortened URL. Each long URL can have multiple shortened URL redirecting it.
We can use Redis as a cache layer for quicker access between the database and the server.
Update: we can set up index on both ID attributes of two tables for more efficient search, and set up a unique shortened URL field to maintain data integrity because each short URL should be unique. we can use database sharding on the user_id to make sure no single database server is overwhelmed.
we will have a frontend, a backend (a cluster of servers), a cache, and a database (cluster of shards)
If a user sends a request to convert the tiny URL, then the client will send a post request to the server with the long URL, server will hash it to a unique short URL, and store it to the cache and the database, with a foreign key of the current User.
then return the short URL back to client.
if a user put the short URL in the browser, 'www.short_url.com/shortdemo', it will send a get request on the endpoint shortdemo, and the server will look in the cache if there's a corresponding field, if there is, then get the original URL, if not, look in the database and see if there's a field, if not, return 404, if yes, redirect to original URL.
Let's talk about things that are related to database, we already talked about database schema, and for sharding we can shard using the User_id, for example: user_id % 3 = 0 and user_id % 3 = 1 and others, or we can shard the database by user location, and store different shards in corresponding location, this way we can avoid putting too much stress on one DB server and make it more robust if using location sharding, we can improve latency as well.
For fault tolerance, we can add data replication to the database, we can use the most common one, master-slave replication to avoid losing information. Each shards have one or more replication, the main database accepts write and read, and the replication accepts read for scalability, we can use another load balancer for the database replication read process.
master node handle the write request, all replications handle the read request
For frontend, the js CSS HTML would work, and the load balancer we can use nginx, for servers it can be any backend framework that serves the API like Spring boot with java or Django with Python, for caching we can use Redis.
Database we can use MySQL or PostgreSQL.
The Failure scenarios can be:
cache goes down, single point of failure
load balancer goes down, single point of failure
data inconsistency in replication reads, or replication lag.
Without a CDN it takes time to load the static resource for the client.
Add analytics for each component of the system.
customizable shortened URLs
API rate limiting to avoid request abuse