Requirements


Functional Requirements:


  • Allow users to upload and store text or code snippets.
  • Generate a unique shareable URL for each paste.
  • Enable retrieval of paste content by URL.
  • Support expiration and TTL for pastes.
  • Allow paste owners or the system to delete a paste before its natural expiration.



Non-Functional Requirements:


  • The system should support low latency uploads and downloads (~500 ms)
  • The system should support high upload/download volumes (~100 upload/s, ~1000 download/s)
  • The system should be highly available (99.99% uptime)
  • The system should be robust from attacks to support real users.


API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...


  1. GET v1/getPreSignedURL -> returns a pre-signed URL to upload content to blob storage
  2. POST v1/paste: {blobStorageURL: <url>, title: <title>, ttl?: timestamp} -> shortened URL
  3. GET v1/paste/<shortened URL> -> blob storage URL
  4. DELETE v1/paste/<shortened URL> -> success



High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.


  1. The CDN is used for read requests to reduce the read latencies by providing responses near the caller. This is the first step to reduce read latency.
  2. The request is then routed to the load balancer which routes loads based on traffic. No sticky sessions are needed as the requests and flows are stateless.
  3. Router - routes read and write req to appropriate microservices.
  4. write service - when a user wants to paste something, they send a GET v1/getPreSignedURL req, which returns them a pre-signed URL. The client frontend - browser/app - then uploads the content to the blob storage and calls the POST v1/paste: {blobStorageURL: <url>, title: <title>, ttl?: timestamp} -> unique shortened URL endpoint. The write service persists this information into a metadata DB and returns a shortened URL to the client.
  5. Read service - when the user calls the GET v1/paste/<shortened URL> -> blob storage URL, the read service takes in the shortened URL and returns the blob storage URL for read access to the blob storage. The read service uses a cache for quick lookup and in case of a hit returns the response directly. It looks up into the metadata DB for cache miss.
  6. Metadata DB - a simple no-sql DB with short URL -> {title, user, blob-storage-link, TTL?}
  7. The delete operation is routed to the write DB which checks for permissions from the DB - the user and then deletes the response from the DB, invalidates the cache entry if present, invalidates the CDN entries and returns the response.
  8. There should be a unique ID service which can generate unique ID for short URL and can do so in a high volume distributed setup. For example: snowflake ID can be used since it works well in a distributed setup although we do not need sortability here.


Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.



  1. Write paths - the write path deals with the following - creating pre-signed URLs, creating short URL and updation in DB. It also needs to deal with deletion. We can reduce latency through auto-scaling since this is stateless. We can also use a queue to moderate traffic spikes.
  2. Read paths - the read path deals with providing the blob storage URL with read only permissions for reading the content from blob storage. Similar to the write service, the latency can be reduced by autoscaling since this is stateless. We can also use a queue to moderate traffic spikes. We use a CDN to cache data near the user. A cache will also be used in addition to the metadata DB to service highly accessed data. Both of these will further reduce the read latency. CDN and cache will be on warm standby.
  3. In this manner, the latency would be reduced for both the read and the write pathways along with moderation in traffic spikes. High volumes of read and write can be addressed through horizontal auto-scaling the required servers.
  4. Availability will depend on the availability of our servers, the unique ID svc, the metadata DB, cache and the blob storage. We will address this through horizontal scaling. The DB will have multiple replica with a leader-follower architecture - writes to leader and reads to followers. In case the leader goes down, a leader election protocol (ex: raft) will elect a new leader and spawn a follower which is brought up to date through synchronization. Similarly, we will have a distributed cache in place of a single cache node. Similar for the ID generation service. The availability then depends on the availability of blob storage - we will use GCP/AWS for this - their overall availability is generally in the order of 6 nines and thus would not reduce/jeopardise our availability requirements.
  5. We need user authentication and auth for protecting the users and the system. All the API would have to be used with a JWT token that identifies the user. We should add rate limiting to reduce the fallout of bad actors. The user identification would also allow us to know the owner for honoring the delete request. Since we are not running the pasted code, we can avoid running the code in a sandbox container to check for malware.
  6. Metadata DB will be a simple no SQL database optimised for reads (we expect read >> writes) with TTL functionality. In case the DB does soft deletion, we should run a periodic clean up job which hard deletes expired entries and clears the cache and CDN.
  7. The unique identifier service can use the blob storage link to generate a unique hash, shorten it and then make it unique by appending a random identifier like timestamp or random ambient temperature of the machine.