Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck)
1M DAU
20K Reqs/second
2 9s
Eventual Consistency
Multi-region ( US, Europe, Asia )
80% of traffic contributd by 20% of URLs
Creation Metadata
Estimate the scale of the system you are going to design...
3.65 GB/year of creation data
182 GB/year of access data .. around 0.5 GB per day
Analytics done per hour basis
Define what APIs are expected from the system...
generateLink(link,expirationTime=defaultVal)
fetchLink(shortenedLink)
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
SQL database for storing the raw data:
The database will be spread out across the 3 regions -> Asia, US and Europe.
We will also be using sharding for these regions, and since eventual consistency is fine, these shards will sync up in some time.
Since we want to avoid naming conflicts here, we will preload the all possible shortened links into the 3 shards, there will be some delay.
We will also have a central cache in Redis ( to store the 20% ), we can use an LRU policy for cache eviction.
This way the popular links will be accessed in no-time for all regions using the cache, while local links will also be accessed soon. There might be some delays if a person from US tries to access a link which is not popular from Asia.
For analytics, we would be performing aggregations on the raw data in access table and storing it the way we want in a document store like MongoDB, a job will update our MongoDB data and aggregations every hour, this will decrease the lag introduced by complex joins and aggregations on the analytics dashboard. We choose mongodb here as it is a document store with high availability.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Client -> Load Balancer 1 Load Balancer 2-> Server Cluster 1,2,3 -> Cache -> Sharded Database
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Client -> Load Balancer 1 Load Balancer 2-> Server Cluster 1,2,3 -> Cache -> Sharded Database
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Client -> Load Balancer 1 Load Balancer 2-> Server Cluster 1,2,3 -> Cache -> Sharded Database -> Eviction process -> Mongo
Explain any trade offs you have made and why you made certain tech choices...
We have used NoSQL mongodb to store analytics, this increases our cost and introduces denormalization, but its required to quickly show performance dashboards to users.
Storing all the raw data in SQL allows us to do detailed deep dives on attacks and usage. While this increases storage required significantly, we get a level of detail which we require to make optimizations later on in the system.
We can evict the URLs not accessed in a long time and mark them as expired.
Try to discuss as many failure scenarios/bottlenecks as possible.
We can try to improve experience for users in US trying to access a link generated in India which is not in the cache.
We can also introduce systems like rate limiting for particular IPs to stop DoS attacks
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?