Loading...
17 million URLs per day
~20 GB of URLs per day
~4 TB storage
create(long_url)
POST: domain.com/api/create
body: {url=long_url}
Return 200 OK with shortened URL in body
read(short_url)
GET: domain.com/{short_url}
body: none
Return redirect response to the long URL
We can use a single NoSQL database for this for better scalability and latency, as we don't need ACID guarantees. This is a read heavy workload, so data replication is important to manage all read requests.
Schema:
shortened url - string
long url - string
creation time - datetime
URL readers and creators both hit a load balancer which hit web servers. We have our NoSQL database that stores our URLs with the schema mentioned previously. This database In front of this database is a write-through cache and then a load balancer. We also have a daily chron job that reads through the db and removes URLs older than 6 months.
Creation: Hit load balancer, then go to web server. The web server turns the long URL into a shortened URL using random characters. It then checks the database to see if this URL is used or not (with base 62 encoding this is a rare but possible occurence with 8 characters). If this URL has not been used before, write to cache and database. Database is sharded on short URL. Write to one database and it will replicate on other duplicates of that shard over time (not consistent, but this is fine for our use case)
Read: Hit load balancer, then hit cache. If not in cache, hit database load balancer then hit a replica in the proper shard of the database.
Delete: as mentioned earlier, this is a chron job. This reads through the entire DB (only around 4 TB * ~ 4 replication), so this isn't too expensive. It deletes anything from the DB that is too old. This job is ran every day during quiet hours.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
The hash can
The database is interesting. This is a very read heavy workload, and only needs to be written once and never updated. Also, there is no need for relational system as we only have one list
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?