Loading...
Capacity estimation
Traffic (Writes)
Suppose we anticipate on the order of 1 million new pastes per day. This is about ~12 paste creation requests per second on average. Traffic will not be uniform; at peak times we might see perhaps 10x the average load (hundreds of writes per second during spikes). We assume each paste is created once (no updates).
Traffic (Reads)
The system is read-heavy. Each paste may be read many times after creation. A read-to-write ratio of around 5:1 or higher is reasonable (many references assume between 5× and 10× more reads than writes). For example, if we have 1M new pastes/day, we might see ~5–10 million paste retrievals per day. This comes out to roughly 60–120 read requests per second on average, and potentially bursts of thousands of reads per second at peak if a particular paste goes viral. In a more extreme scenario (100:1 read/write ratio), read traffic could reach ~100M/day (~1200 reads/s), so our design should be prepared for high read throughput.
Data Size per Paste
We must decide on maximum and average paste sizes. To prevent abuse, we can cap the size of a paste (for example, maximum 1 MB or 5–10 MB of text). Realistically, many pastes are much smaller (snippets of code or logs). Let’s assume an average paste size ~10 KB (some may be just a few hundred bytes, some could be larger, but 10 KB is a typical order of magnitude).
Storage (Daily & Total)
At 1M pastes/day * 10 KB each, that’s about ~10 GB of new data per day that needs to be stored. Over longer periods:
In one month (~30 days): ~300 GB of data.
In one year: ~3.6 TB of data.
In five years: ~18 TB of data.
These figures assume we retain everything indefinitely. If we implement expirations, the actual stored data would depend on how long pastes live. For instance, if we only retain data for 3 months (90 days), we’d store roughly the last ~900 GB of pastes at any given time. Given millions of users, we should design for multiple terabytes of storage in the long run.
Total Number of Pastes
Over five years at 1M pastes/day, we could accumulate about 1.8–2 billion paste entries (if none expired). Even with expirations, the database could contain on the order of hundreds of millions of records. This impacts how we choose our data store and how we generate unique keys (IDs).
ID Space and Collision
We need a strategy to generate unique paste IDs that won’t run out. If we use an alphanumeric ID of length 6 (using 62 characters [0-9, a-z, A-Z]), we have
62
6
≈
56
62
6
≈56 billion possible IDs. This is plenty for our needs (e.g. 2 billion IDs used is only ~3.6% of that space). Using base64 (64 characters) with 6 characters gives ~68.7 billion possibilities. Even in the distant future with tens of billions of pastes, 6-character IDs suffice. We can always extend to 7 characters (62^7 = 3.5 trillion combos) if needed. The probability of random ID collisions with 56+ billion possibilities is very low for our usage, but we will still handle collision cases just in case (see Key Generation in detailed design).
These estimations guide our design: we need a system that can handle tens of writes per second, hundreds to thousands of reads per second, and store terabytes of data over time. The design should support scaling out horizontally as these numbers grow.
The data model for a paste is simple. Each paste is essentially a small document with an ID as the key. We can separate metadata (ID, timestamps, etc.) from the actual content for efficiency. The design will include:
Paste Metadata: This will be stored in a fast, persistent database keyed by the paste ID (which is the unique URL token).
We can represent the Paste metadata table as follows:
Since we have no user accounts, we don’t need a user ID in this table. Each paste stands on its own.
Paste Content Storage: For storing the actual text content, a common pattern (given potentially large volumes of text) is to use an object storage or blob storage service. The idea is to offload the heavy lifting of storing and serving large text blobs to a system designed for it. Options include cloud storage like Amazon S3, Google Cloud Storage, or a distributed file system. The metadata table will store a reference (e.g. an S3 key or URL) to the content.
Why not store content directly in the database?
Storing everything in one SQL or NoSQL database is simpler initially, but it can become a bottleneck:
Database Choice
We need to pick a database for the metadata. Key considerations:
id as primary key). It provides strong consistency and ease of querying, but scaling to billions of rows might require sharding or federation. It’s feasible (companies do shard MySQL/Postgres for large data), but adds complexity. Since our access pattern is simple and mostly primary-key lookups, a NoSQL store can give scalability with less management overhead.Alternatively, DynamoDB (if using AWS) could be used with a similar data model. DynamoDB can handle a very high scale of requests and also has a TTL feature for items. Both choices meet our needs; we’ll proceed with the concept (the design is similar either way).
Search Indices
Our primary access pattern is by ID (exact lookup), which is the primary key. That lookup will be O(1) or O(log N) depending on the DB, but essentially very fast via hashing. We should also consider an index on the expires_at field if we want to efficiently query for expired records for cleanup. In Cassandra, one might use a time-series table or a separate mechanism for expiration, but since TTL will handle automatic expiration, we may not need a secondary index. If using a system without TTL, we might need a scheduled job to find expired pastes (e.g. scanning by time or keeping a sorted set of expiring IDs).
Summary
Each paste operation will result in:
This separation allows us to scale and manage the two concerns (metadata vs content) independently. The metadata DB remains lean and efficient for queries, and the object store handles large data storage with high durability.
External API
1.Create Paste
Method: POST
Endpoint: /{version}/paste
Request Body:
paste_text: string (required) : text included in paste
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
paste_id: id: id of paste
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
2.Get Paste
Method: GET
Endpoint: /{version}/paste/{paste_id}
Path Variable:
paste_id: {id}: id of the paste
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
3.Update Paste
Method: PATCH
Endpoint: /{version}/paste/{paste_id}
Path Variable
paste_id: id: id of the paste
Request Body:
new_text: string: new text
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
4.Delete Paste
Method: DELETE
Endpoint: /{version}/paste/{paste_id}
Path Variable:
paste_id: id: id of the paste
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
Service Component:
Diagram
Gen UID Service:
Database:
Paste expiration:
As we allow the user to set expiration condition for every paste we need efficient way to invalidate a balk of paste. For time condition we just need to map expiration_date with the id. If there is request after expiration timestamp we can easily know that this paste is invalid and delete it. In addition we can have the cron job that will periodically check if at the current time are there any paste that is expired if so we can delete them all for reducing storage cost.
Cache:
Scalability:
We will deploy stateless service on container orchestration service such as AWS ECS. In this case we can scale up/down our container on a whim. We can also set up condition for scaling such as when the request amount of certain service exceed our predefine value. They will add more instance to serve the upcoming load. Also we can adopt fault tolerant architecture by using spot instance to save the cost of infrastructure.