Loading...
List the key functional requirements for the system (Ask the AI for hints if stuck)...
The user can store the copied text
The copied text will still there until user paste it somewhere
List the key non-functional requirements (performance, scalability, reliability, etc.)...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Suppose we anticipate on the order of 1 million new pastes per day. This is about ~12 paste creation requests per second on average. Traffic will not be uniform; at peak times we might see perhaps 10x the average load (hundreds of writes per second during spikes). We assume each paste is created once (no updates).
The system is read-heavy. Each paste may be read many times after creation. A read-to-write ratio of around 5:1 or higher is reasonable (many references assume between 5× and 10× more reads than writes). For example, if we have 1M new pastes/day, we might see ~5–10 million paste retrievals per day. This comes out to roughly 60–120 read requests per second on average, and potentially bursts of thousands of reads per second at peak if a particular paste goes viral. In a more extreme scenario (100:1 read/write ratio), read traffic could reach ~100M/day (~1200 reads/s), so our design should be prepared for high read throughput.
Data Size per Paste
We must decide on maximum and average paste sizes. To prevent abuse, we can cap the size of a paste (for example, maximum 1 MB or 5–10 MB of text). Realistically, many pastes are much smaller (snippets of code or logs). Let’s assume an average paste size ~10 KB (some may be just a few hundred bytes, some could be larger, but 10 KB is a typical order of magnitude).
Storage (Daily & Total)
At 1M pastes/day * 10 KB each, that’s about ~10 GB of new data per day that needs to be stored. Over longer periods:
These figures assume we retain everything indefinitely. If we implement expirations, the actual stored data would depend on how long pastes live. For instance, if we only retain data for 3 months (90 days), we’d store roughly the last ~900 GB of pastes at any given time. Given millions of users, we should design for multiple terabytes of storage in the long run.
Total Number of Pastes
Over five years at 1M pastes/day, we could accumulate about 1.8–2 billion paste entries (if none expired). Even with expirations, the database could contain on the order of hundreds of millions of records. This impacts how we choose our data store and how we generate unique keys (IDs).
ID Space and Collision
We need a strategy to generate unique paste IDs that won’t run out. If we use an alphanumeric ID of length 6 (using 62 characters [0-9, a-z, A-Z]), we have 626≈56626≈56 billion possible IDs. This is plenty for our needs (e.g. 2 billion IDs used is only ~3.6% of that space). Using base64 (64 characters) with 6 characters gives ~68.7 billion possibilities. Even in the distant future with tens of billions of pastes, 6-character IDs suffice. We can always extend to 7 characters (62^7 = 3.5 trillion combos) if needed. The probability of random ID collisions with 56+ billion possibilities is very low for our usage, but we will still handle collision cases just in case (see Key Generation in detailed design).
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
POST/paste
GET/paste/{id}
DELETE/paste/{id}
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
At a high level, our Pastebin like service follows a standard web service architecture with decoupled components to achieve scalability and reliability. The major components include:
Clients
End-users or applications that send requests to create or read pastes. This could be a web browser (visiting a website or using a web form) or any HTTP client using the API.
Load Balancer (LB)
Distributes incoming HTTP requests across multiple application servers. The LB ensures no single server is overwhelmed and enables horizontal scaling. It also health-checks servers and stops sending traffic to any that are down.
Application Servers
These run the core application logic (the Pastebin service). They handle HTTP requests for creating or retrieving pastes. We can design them as stateless web servers (e.g. Node.js, Python Flask/FastAPI, Java Spring, Go, etc.) so they can be easily scaled out. We might logically separate Write operations and Read operations for clarity or even deploy them as separate services (since the read path can be optimized separately from the write path). However, they could also be unified in one codebase. Each app server will:
ID Generation Service
A specialized service or component responsible for generating unique IDs for new pastes. We mention it separately because ensuring uniqueness and avoiding collisions at scale can be non-trivial. The ID generator could be implemented as an internal library (generating random IDs and checking the DB) or a dedicated microservice (e.g. a Key Generation Service (KGS) that doles out unique keys). We will detail this in the component design, but at a high level, this service ensures each paste gets a unique short ID and that we can generate them at a high rate without conflicts.
Metadata Database
The database cluster storing paste metadata (as designed in section 4). This is typically a distributed NoSQL store like Cassandra or DynamoDB. It holds entries mapping paste IDs to content references and metadata. It is designed to handle a huge number of records and many requests per second. We will configure it with replication for fault-tolerance and possibly sharding/partitioning by ID (which is usually automatic in Cassandra/Dynamo).
Object Storage Service
This stores the actual paste text content (blob data). For instance, an AWS S3 bucket or a distributed file store holds files named by paste IDs. This service is highly durable (e.g. S3 redundantly stores data across facilities). The application servers will store to and read from this storage when handling pastes. To improve performance, we might integrate a CDN in front of the object storage for reads (so content can be cached closer to users).
Cache Layer
A distributed in-memory cache (Redis) to cache frequently accessed data. We can use the cache to store recently created pastes and frequently read pastes. The cache is typically keyed by paste ID and stores the content (and possibly metadata) as value. This significantly reduces read latency and load on the database for hot keys. The cache cluster can be scaled and should also be replicated (or use clustering) to avoid it being a single point of failure.
Background Job Scheduler
A component (or cron job) for maintenance tasks. Specifically, a scheduled job will periodically scan for expired pastes and delete them from the system. If we rely on DB TTL, this job may not be heavily needed for DB cleanup, but it might still handle removal from object storage (e.g. deleting S3 objects for expired pastes, if not automated) and cache invalidation. This scheduler could also handle other tasks like generating usage analytics, or pre-computing any stats if needed.
miss
Client
Load Balancer / API Gateway
Stateless App Servers
Redis Cache
Metadata DB
(Cassandra / DynamoDB)
Object Storage
(S3)
TTL Cleanup Worker
Write path
A user submits a POST /paste containing the text and an optional expiry time. The load‑balancer forwards the call to any stateless write‑server. The server validates size, asks its local ID‑generator (or retries random generation until unique) for a six‑character token such as abc123, streams the text into object storage under that key, then records a metadata row (id, content_ref, created_at, expires_at) in the distributed database. If either step fails the operation is rolled back. Finally it primes the Redis cache with the content and TTL matching the expiry and returns https://…/paste/abc123 to the client—total latency tens of ms.
Read path
A browser or script issues GET /paste/abc123. The request lands on a read‑server, which first looks in Redis; a hit returns the text in a few ms. On a miss it loads the metadata row (O(1) key lookup); if the record is absent or expires_at is in the past it responds 404/410. Otherwise it fetches the blob from object storage, streams it to the client, stores the text in Redis with the residual TTL, and finishes. Subsequent reads are memory‑speed; a CDN in front of the object store can off‑load bandwidth for viral pastes.
Expiry and cleanup
Each row is written with a database TTL, and each object is tagged with an S3 lifecycle expiry. A periodic job merely evicts any lingering cache entries and double‑checks that both DB row and blob have disappeared, keeping storage use bounded.
This condensed flow shows how the load‑balancer, stateless servers, Redis, the database and object storage cooperate to give fast, collision‑free writes, sub‑millisecond cache hits, and automatic expiry without manual intervention.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
The data model for a paste is simple. Each paste is essentially a small document with an ID as the key. We can separate metadata (ID, timestamps, etc.) from the actual content for efficiency. The design will include:
Paste Metadata: This will be stored in a fast, persistent database keyed by the paste ID (which is the unique URL token).
We can represent the Paste metadata table as follows:
Paste Content Storage: For storing the actual text content, a common pattern (given potentially large volumes of text) is to use an object storage or blob storage service. The idea is to offload the heavy lifting of storing and serving large text blobs to a system designed for it. Options include cloud storage like Amazon S3, Google Cloud Storage, or a distributed file system. The metadata table will store a reference (e.g. an S3 key or URL) to the content.
id as primary key). It provides strong consistency and ease of querying, but scaling to billions of rows might require sharding or federation. It’s feasible (companies do shard MySQL/Postgres for large data), but adds complexity. Since our access pattern is simple and mostly primary-key lookups, a NoSQL store can give scalability with less management overhead.Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
Now, let’s dive deeper into each major component, discussing how they are implemented and how we ensure they meet our requirements.
The application layer can be designed as a stateless, horizontally scalable tier. We can run many instances (on VMs or containers) behind the load balancer. Because they are stateless, any instance can handle any request, which simplifies scaling and failover.
Separation of Concerns
We might logically separate the write path and read path handlers. They could be different endpoints on the same service, or even split into two deployable services if we want to scale them independently. For instance, if reads become 100x more frequent than writes, we might allocate more servers to running the read handler. In microservices style, we could have a Write Service and a Read Service, each with its own autoscaling policy. However, initially it might be simpler to have one service that does both, and scale it out as a whole.
ID Generation Integration
When a write server needs a new ID, how does it get it? A few design options:
AUTO_INCREMENT or a separate table for IDs). This centralizes ID generation but can become a bottleneck at scale (every new paste hits one sequence). Also, sequential IDs are predictable.For simplicity and given our moderate write QPS (~12/s), we can start with the random-and-check approach, which doesn’t require a separate service. We enforce uniqueness via the DB’s primary key and handle the rare collision by retrying. If down the line the collision rate or performance becomes an issue, we could introduce a more elaborate KGS.
Data storage calls
The app server will interact with the DB and object storage. We should use efficient clients:
Caching
Each app server will also interface with the cache:
SET id -> content (EX seconds_till_expiry). On writes, we might do an add: SET id -> content so that an immediate subsequent read finds it.Statelessness and Scaling
Because all state (pastes) is in the DB/storage and cache, the app servers don’t hold any persistent state between requests. This means we can scale out by just adding more servers behind the LB. If one server fails mid-request, the request fails but the client can retry and another server can handle it. For sticky scenarios like maintaining a batch of IDs, each server may have some cached IDs, but losing them is not critical aside from a tiny performance hit.
Autoscaling
We can set up the infrastructure to monitor CPU, memory, or request rate on the app servers and automatically launch more instances during high load. Because of the LB, new instances can start receiving traffic as soon as they’re up. Similarly, in low load, some instances can be shut down. This elasticity ensures we handle millions of users at peak but don’t waste resources at off-peak.
We choose a NoSQL distributed database (like Cassandra) for metadata, as justified. Here’s how we configure and use it:
Cluster Setup
The DB will run on multiple nodes (servers). For example, if we have 6 nodes, Cassandra will partition the key range among them. Each paste ID, when hashed, maps to a token that determines the node responsible. Data is replicated to (say) 3 nodes for fault tolerance. This means any given piece of data resides on 3 nodes (RF=3), and the cluster can tolerate up to 2 node failures without data loss.
Keyspace and Table
We create a keyspace (namespace) in Cassandra for our service, and within it a table for Pastes with columns as per our schema. The primary key is the id. In CQL, it might look like:
CREATE TABLE Pastes ( id TEXT PRIMARY KEY, content_ref TEXT, created_at TIMESTAMP, expires_at TIMESTAMP );
We might also include content_size or other fields as needed. If we plan to support user accounts in the future, we could add a user_id to link to a User table (but that would complicate the key since no accounts now, we skip it).
Access Patterns
PutItem.SELECT * FROM Pastes WHERE id = ?. This hits the nodes that have the key’s partition. If consistency is ONE or QUORUM, one replica’s result is returned. We likely want strong read consistency for newly written data: we can either do a QUORUM read, or in Cassandra, do a ONE read under the assumption that the client hitting the same region will get the local replica which likely has it (if replication is across racks rather than distant data centers). In Dynamo, we can specify strongly consistent read option for that request if needed.Partitioning & Load Distribution
Because our keys (IDs) are essentially random strings (especially if generated by a good random process), they will naturally distribute evenly across partitions. This avoids hotspots (unlike a sequence, where recent IDs all go to one shard, here “abc123” and next “qwe456” will likely hash to different nodes). This uniform distribution is great for scaling. We just have to ensure the partitioner (like Murmur3 in Cassandra) spreads keys well, which it does for random input.
Scaling the DB
If our usage grows (say to 10M writes/day and hundreds of millions of active records), we can add nodes to the Cassandra cluster. Thanks to consistent hashing, data rebalances with minimal impact. In Dynamo, we’d request a higher throughput from AWS. The important point is that this DB design can scale horizontally without a major redesign (in contrast, a single MySQL instance would start struggling, forcing a sharded MySQL approach).
Durability & Backups
We will configure multi-datacenter or at least multi-AZ replication for resilience. Also, periodic snapshots or backup of the data (in case of catastrophic failure beyond 2 nodes) should be done. If using Dynamo, AWS handles backup with point-in-time recovery.
Alternative
If we had chosen a relational DB, we would likely have to partition it ourselves by range or hash of id. We’d also set up replicas for reads. It can be done, but NoSQL simplifies it here. Moreover, no complex query (like no JOIN or scanning by user, etc.) is needed, so we don’t lose much by not having SQL relations.
We decide to use an external object storage service for paste content:
For example, AWS S3 bucket named “pastes” can store objects where each object’s key is the paste ID (or we could use a folder structure if needed, e.g. partition by first 2 chars, but S3 handles large number of objects fine nowadays).
When a paste is created, we do an S3 PUT request with key = pasteID, content = text blob. We might also set metadata on the object, such as an HTTP header for expiration date (for S3’s lifecycle policy).
We enable a lifecycle rule on the bucket: any object with an expiration tag or older than X gets deleted. Alternatively, set each object’s Expiry property individually if possible. This way, when a paste expires, S3 will automatically remove the content after some time (usually S3 lifecycle is not to-the-second precise, but within a day of expiration it will remove).
On retrieval, we have two approaches:
Capacity
S3 can handle essentially unlimited data, so storing 18+ TB over years is fine. We just pay for what we use. It’s also highly durable (99.999999999% durability) meaning it’s practically impossible to lose data once stored. That aligns with our durability requirement.
Data format
We are storing plain text. We should ensure to set the correct content-type (text/plain; charset=utf-8) so that if the content is fetched via web or CDN, it’s served correctly. We might also compress the content (text compresses well). S3 can store gzipped content and we could mark it as such. But since pastes are not huge, compression is optional. It could save bandwidth if a lot of large texts are served.
Multi-region
If we deploy our service in multiple regions, we could either use separate buckets (one per region) or an S3 replication. But that’s an advanced scenario – by default, we might keep all data in one region’s bucket. A CDN would mitigate cross-region access issues.
We use Redis as an in-memory cache to speed up reads. The design considerations:
Deployment
A Redis cluster of multiple nodes (to have more memory and also avoid single point of failure). We could use Redis with replication (master-slave) and partitioning (sharding keys across nodes). There are also managed services like AWS ElastiCache. For simplicity, assume we have a few Redis nodes and a consistent hashing or cluster mode to distribute keys.
What we cache
Primarily the paste content (the text) indexed by the paste ID. We might also cache the fact that a paste ID is expired or not found (to prevent hitting DB repeatedly for a non-existent ID). This is known as caching negative results. We could store a marker like id -> NULL with a short TTL if a paste wasn’t found, so that subsequent requests in a short time don’t slam the DB. But careful with that: if a paste is just created, we don’t want a cached “not found” to persist. So maybe small TTL (a minute) for negatives.
Expiration
We set the cache entry to expire at the same time as the paste. Redis supports setting an absolute TTL on key. For example, if a paste will expire in 2 hours, we set the cache to evict it in 2 hours as well. If the paste never expires, we could set a long TTL (or no TTL, relying on LRU eviction). But in theory “never” could overload memory, so we might set some upper bound TTL like 30 days and it will be refreshed if accessed often.
Write-through vs Lazy population
We will do a bit of both:
Cache Size
If we allocate, say, 10 GB to cache, that can hold ~1 million entries of 10KB each. If our daily active set of pastes (that are being read) is in that ballpark, cache will be effective. We assume a Zipf-like distribution where some pastes get a lot of hits (those will stay in cache due to frequency).
Eviction policy
Use LRU or LFU (least recently/frequently used) so that unused pastes get evicted to make room for currently hot ones. Redis by default has LRU.
Consistency considerations
Since we store data also in DB, we have to ensure if data changes, cache updates. In our system, data is immutable after creation (no updates to content), and deletion/expiry we handle by TTL. So consistency is easy – once cached, content doesn’t change. The only consistency issue is serving an expired paste from cache. By aligning TTLs, we avoid that. Also on deletion, the deletion job should delete from cache if it finds an entry. So our cache consistency with the source data is manageable.
Failure scenario
If the cache cluster goes down or is flushed, the system will still work (just every request goes to DB/storage, with higher latency). It’s not catastrophic, just less optimal. We should monitor cache health and maybe have redundant caching if needed (but usually not required since loss just means recalculation).
If we implement a dedicated ID generation service (KGS), details would include:
Batching keys
As mentioned, one optimization is each app server requests e.g. 1000 new IDs from KGS at once, and caches them locally.
If we implement this instead of random generation, here’s how it would work:
Preventing KGS as a SPOF
We’d replicate the KGS or allow app servers to fall back to random gen if KGS is unavailable. Also caching keys on app nodes helps. Essentially, if KGS fails, the system shouldn’t grind to a halt; we could temporarily allow local generation until KGS is back.
Not explicitly asked, but in any large system:
These support maintainability and operations of the system at scale.
Throughout the design, we made certain decisions. Here we discuss trade-offs and alternatives considered, justifying why we chose one approach over others:
SQL vs NoSQL for Metadata
We chose a NoSQL key-value store (like Cassandra) over a SQL database.
SQL (like MySQL) would give strong consistency and ACID guarantees easily, and can be simpler for developers used to relational schema. However, at the scale of billions of records and high throughput, a single SQL instance would not suffice; we’d need sharding and/or replication, which adds complexity. NoSQL (Cassandra/DynamoDB) is built to scale horizontally with minimal effort and can handle our access pattern (simple PK lookups) efficientl】. We traded some convenience of SQL and maybe complex query ability for the scalability and performance of NoSQL. We also accept eventual consistency in some NoSQL systems, but we can configure for near-strong consistency in this use case (reads and writes to quorum). DynamoDB offers strong consistency per request if needed, Cassandra’s eventual consistency is usually fine for our scenario since we don’t do multi-step transactions. Conclusion: The choice prioritizes horizontal scalability and write throughput (NoSQL) over strict relational features.
Strong Consistency vs Availability
According to the CAP theorem, in a distributed system we often trade consistency for availability. Our system emphasizes that once a paste is created and confirmed, any read after that should get it (consistency). But in a network partition, what do we prefer? If using Cassandra, we can choose consistency levels. We might choose availability (AP) by serving stale data if needed (though stale in our case mostly means missing newly created paste if a partition prevents seeing it). However, missing data might be considered as downtime for that paste. Given the nature of pastebin, it’s likely acceptable to be eventually consistent for a very brief window (user likely not sharing the link the microsecond after creation). Some designs explicitly mention eventual consistency is okay because the link isn’t used immediatel】. But realistically, the user might create and then click the link right away. We ensure our process makes it consistent by the time they get the link (since we waited for DB commit).
Object Store vs Storing in DB
We opted to store content in an object store rather than directly in the DB.
Keeping everything in one database (e.g. using a blob column in a SQL DB or storing text in Cassandra value) simplifies architecture (fewer moving parts). But the database would then handle a lot more data and I/O, which can slow down queries and make scaling harder. Object stores are optimized for heavy I/O and are cheaper for large data. We trade some complexity (we now have to manage two storage systems and the reference between them) for a big win in scalability and cost. It also simplifies DB replication (since it’s not replicating huge blobs). The object store approach slightly increases read latency (two calls instead of one), but caching and possible direct serving mitigate that. So we prioritize scalability of storage and throughput by using object storage, at the cost of system complexity (two datastores and eventual consistency between them if not careful). Given our scale, that trade-off is justified.
Cache Aside vs Write-through
We are using a cache in front of DB/storage primarily in a cache-aside pattern (the application populates cache on demand. We also do a bit of write-through (populating on new paste).
A pure cache-aside means first read is always a miss and hits DB, subsequent are fast. Write-through means even the first read could be fast, but it adds overhead on write path (every write goes to cache too, even if no one reads it). We chose a combination: write-through newly created content because it’s cheap to do and likely beneficial, but otherwise rely on cache-aside for other content. This gives a good balance of not overwhelming the cache with every single paste (many might never be read even once), but ensuring popular content is cached. The trade-off here is minimal; it’s an optimization detail. Also, we must accept that after a restart of cache or cold start, the initial requests will populate cache (slightly slower), which is okay.
Generating IDs (Random vs Central Service)
We decided on random generation with collision-check.
This is simple and fast but has a theoretical risk of collisions (which we mitigate by check). The alternative, a central ID service or a deterministic generation, removes collision risk but adds complexity and a potential bottleneck/single point to guard. By going with random IDs, we favor simplicity and decentralization at the cost of a negligible probability of collisions and an eventual need for a more coordinated solution if rate skyrockets. We also avoid sequential IDs which could be guessable. If we had chosen sequential (like a DB sequence), we’d likely need to obscure it. So random is good for security. If our service needed absolutely zero chance of collision without a check, KGS would be needed. We deem the trade (some code to retry on collision) worth it for less infrastructure.
Single Region vs Multi-Region Deployment
For MVP, we likely deploy in a single region data center (with redundancy across AZs). That means users far from that region might experience higher latency. Multi-region (with data replication across continents) would improve latency and availability (if one region goes down, others still serve). But multi-region replication for both DB and storage complicates consistency. Many pastebin services might just run in one region and rely on CDN for global reads. We choose to keep data in one region (to ensure consistency and simplicity) and possibly use a CDN for global performance. The trade-off is some users get slower response (e.g., an Asia user hitting a US server might see 200ms latency instead of 50ms). For a text fetch, this is usually acceptable. In the future, if user base is truly global and latency sensitive, we can explore multi-region active-active (with a more complex DB setup or multi-master). That would trade more complexity for lower latency and redundancy.