Requirements


Functional Requirements:

  • Allow User to paste any text and submit the pasted text
  • Upon submission user recieves a unique sharable link with their pasted text
  • User can optionally set expiration for the pasted text
  • Allow any user with the shareable link to access the pasted text
  • User should be able to delete pastes


Scale Estimation:

  • System should handle approx 1B unique pasted texts
  • 100M DAU - 90% reads, 10% writes
  • So ~10M daily pasted text submissions


Non-Functional Requirements:


  • Available > Consistency - The system should not fail when the user tries to retrieve any pasted texts via links with 99.9% uptime
  • Highly Scalable - System should be able to scale to support high traffic scale 1B pasted texts and 100M DAU
  • High read throughput - System is read heavy so read throughput should be high
  • Low latency - The retriveal of pasted texts should be quick and efficient with no significant delays.
  • Unique URLs with proper url generation that is hard to guess to prevent unauthorized access.


API Design

Data Entities -


PastedText

Id,

Expiration?,

UrlCode,

IsDeleted,

StorageKey


PastedTextContent

PastedIdText,

PastedIdContent


APIs


POST /pasted-text -> Creates a new pasted text and redirects to url

Request - {

text,

expiration?

}


This api will be behind a rate limiter, we will apply rate limit on API keys


Response - HTTP 301 - redirects to the unique sharable url


PUT /pasted-text -> Soft deletes the text

request - {

isDeleted = true

}


response - HTTP 301 - delets and redirects to home page


GET /{url_code} - HTTP 301 - redirects to the the url, and fallbacks to an error page when the url code is invalid



High-Level Design

Our design consists of two main services:

  • Read Service
  • Write Service

Components

API Gateway

Acts as the entry point for all client requests.

Responsibilities:

  • Authentication (if required)
  • Rate limiting
  • Request routing

Load Balancer

Distributes traffic across multiple instances of the Read and Write services to improve scalability and availability.

Write Service

Responsible for:

  • Creating pastes
  • Deleting pastes
  • Generating unique URLs

When a new paste is created:

  1. The service generates a unique short URL.
  2. The paste content is stored in object storage.
  3. Metadata is stored in the database.

Example metadata:

PasteId UrlCode StorageKey CreatedAt ExpiresAt IsDeleted

Read Service

Responsible for retrieving pastes using the URL code.

Read flow:

  1. Check Redis cache.
  2. If cache hit, return content immediately.
  3. If cache miss:
    • Fetch metadata from the database.
    • Retrieve content from object storage.
    • Populate Redis cache.
    • Return content.

Redis Cache

Used to cache frequently accessed pastes.

Benefits:

  • Reduces database load.
  • Reduces object storage reads.
  • Improves response latency.

Redis stores:

UrlCode -> Paste Content

Database

Stores metadata about each paste.

Example:

PasteId UrlCode StorageKey CreatedAt ExpiresAt IsDeleted

Object Storage

Stores the actual paste content.

Benefits:

  • Low cost
  • High durability
  • Virtually unlimited storage capacity

Background Worker

Runs asynchronously to manage expired pastes.

Responsibilities:

  • Identify expired pastes.
  • Mark them as soft deleted in the database.
  • Remove corresponding cache entries from Redis.

Since the records are soft deleted, the content may remain in object storage until a future cleanup process permanently removes it.


Detailed Component Design


We will now deep into these topics:


1) How to support 10k+ reads/sec i.e. High read throughput:

  • System is read heavy so it is essential to make system able to manage massive read scale.
  • This can be achieved via combination of caching layers, caching strategy and scaling read service horizontoally
  • Caching layers:
    • CDN: We will use CDN for reads, so any read request will first closest CDN/edge location. If cache hit, the response is returned back, if cahce miss then the request is routed through API gateway eventually gets routed through read servers.
    • Redis: Redis is second layer of caching implemented at read service.The flow is:
      1. If cache hit, return content immediately.
      2. If cache miss:
        1. Fetch metadata from the database.
        2. Retrieve content from object storage.
        3. Populate Redis cache.
        4. Return content.
    • We will use Redis with an LRU eviction policy to keep frequently accessed pastes in memory. it gives us benefits like:
      • Reduces database load.
      • Reduces object storage reads.
      • Improves response latency.


    • If Redis becomes unavailable, the Read Service can fall back to the database and object storage. This increases latency but keeps the system functional.


    • Text updates rarely happens and to ensure freshness of data upon text deletion, we will invalidate cache whenever the Delete API is called or the background job that manages expire cache deletes data. This ensures data is correct always when read.
  • Scaling Horizontally:
    • Additonally, we will horizontally scale our read servers and use our Load balancer to route traffic and distrubute traffic across these servers


2) How to handle duplicate paste creation request and ensure URL is unique and not easily guessable:

  • This is one of the most crucial part of the system. We need to ensure that all URL that are generated they are unique, and not easily guessable to avoid unauthorized access.
  • Approach we will take to solve this is:
    • Generate random 7-10 characters base62 string that will be used as url code, so url would be like https://my-service/{url_code}
    • However, using this there are chances for collision. So we need a mechanism to avoid these collision. The simple solution for these is the UrlCode column will have a UNIQUE constraint. If an insertion fails because of a uniqueness violation, the Write Service generates a new code and retries. This also ensures race condition does not end up creating duplicates
    • But the chances of this are very rare, as if we use lets say 8 characters base62 then 62^8 ~ 218 trillion possibilities which is more than enough for our 1B unique text paste scale.


Rate limiting

Handled at API Gateway using token bucket per API key/IP to prevent abuse and control write traffic.

Idempotency

For create operations, idempotency keys can be used to prevent duplicate pastes during client retries.

Cache stampede

Cache stampede can be mitigated using request coalescing or short-lived distributed locks during cache misses.