Detailed Component Design
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
- Write paths - the write path deals with the following - creating pre-signed URLs, creating short URL and updation in DB. It also needs to deal with deletion. We can reduce latency through auto-scaling since this is stateless. We can also use a queue to moderate traffic spikes.
- Read paths - the read path deals with providing the blob storage URL with read only permissions for reading the content from blob storage. Similar to the write service, the latency can be reduced by autoscaling since this is stateless. We can also use a queue to moderate traffic spikes. We use a CDN to cache data near the user. A cache will also be used in addition to the metadata DB to service highly accessed data. Both of these will further reduce the read latency. CDN and cache will be on warm standby.
- In this manner, the latency would be reduced for both the read and the write pathways along with moderation in traffic spikes. High volumes of read and write can be addressed through horizontal auto-scaling the required servers.
- Availability will depend on the availability of our servers, the unique ID svc, the metadata DB, cache and the blob storage. We will address this through horizontal scaling. The DB will have multiple replica with a leader-follower architecture - writes to leader and reads to followers. In case the leader goes down, a leader election protocol (ex: raft) will elect a new leader and spawn a follower which is brought up to date through synchronization. Similarly, we will have a distributed cache in place of a single cache node. Similar for the ID generation service. The availability then depends on the availability of blob storage - we will use GCP/AWS for this - their overall availability is generally in the order of 6 nines and thus would not reduce/jeopardise our availability requirements.
- We need user authentication and auth for protecting the users and the system. All the API would have to be used with a JWT token that identifies the user. We should add rate limiting to reduce the fallout of bad actors. The user identification would also allow us to know the owner for honoring the delete request. Since we are not running the pasted code, we can avoid running the code in a sandbox container to check for malware.
- Metadata DB will be a simple no SQL database optimised for reads (we expect read >> writes) with TTL functionality. In case the DB does soft deletion, we should run a periodic clean up job which hard deletes expired entries and clears the cache and CDN.
- The unique identifier service can use the blob storage link to generate a unique hash, shorten it and then make it unique by appending a random identifier like timestamp or random ambient temperature of the machine.