Loading...
Capacity estimation
Traffic (Writes)
Suppose we anticipate on the order of 1 million new pastes per day. This is about ~12 paste creation requests per second on average. Traffic will not be uniform; at peak times we might see perhaps 10x the average load (hundreds of writes per second during spikes). We assume each paste is created once (no updates).
Traffic (Reads)
The system is read-heavy. Each paste may be read many times after creation. A read-to-write ratio of around 5:1 or higher is reasonable (many references assume between 5× and 10× more reads than writes). For example, if we have 1M new pastes/day, we might see ~5–10 million paste retrievals per day. This comes out to roughly 60–120 read requests per second on average, and potentially bursts of thousands of reads per second at peak if a particular paste goes viral. In a more extreme scenario (100:1 read/write ratio), read traffic could reach ~100M/day (~1200 reads/s), so our design should be prepared for high read throughput.
Data Size per Paste
We must decide on maximum and average paste sizes. To prevent abuse, we can cap the size of a paste (for example, maximum 1 MB or 5–10 MB of text). Realistically, many pastes are much smaller (snippets of code or logs). Let’s assume an average paste size ~10 KB (some may be just a few hundred bytes, some could be larger, but 10 KB is a typical order of magnitude).
Storage (Daily & Total)
At 1M pastes/day * 10 KB each, that’s about ~10 GB of new data per day that needs to be stored. Over longer periods:
In one month (~30 days): ~300 GB of data.
In one year: ~3.6 TB of data.
In five years: ~18 TB of data.
These figures assume we retain everything indefinitely. If we implement expirations, the actual stored data would depend on how long pastes live. For instance, if we only retain data for 3 months (90 days), we’d store roughly the last ~900 GB of pastes at any given time. Given millions of users, we should design for multiple terabytes of storage in the long run.
Total Number of Pastes
Over five years at 1M pastes/day, we could accumulate about 1.8–2 billion paste entries (if none expired). Even with expirations, the database could contain on the order of hundreds of millions of records. This impacts how we choose our data store and how we generate unique keys (IDs).
ID Space and Collision
We need a strategy to generate unique paste IDs that won’t run out. If we use an alphanumeric ID of length 6 (using 62 characters [0-9, a-z, A-Z]), we have
62
6
≈
56
62
6
≈56 billion possible IDs. This is plenty for our needs (e.g. 2 billion IDs used is only ~3.6% of that space). Using base64 (64 characters) with 6 characters gives ~68.7 billion possibilities. Even in the distant future with tens of billions of pastes, 6-character IDs suffice. We can always extend to 7 characters (62^7 = 3.5 trillion combos) if needed. The probability of random ID collisions with 56+ billion possibilities is very low for our usage, but we will still handle collision cases just in case (see Key Generation in detailed design).
These estimations guide our design: we need a system that can handle tens of writes per second, hundreds to thousands of reads per second, and store terabytes of data over time. The design should support scaling out horizontally as these numbers grow.
External API
1.Create Paste
Method: POST
Endpoint: /{version}/paste
Request Body:
paste_text: string (required) : text included in paste
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
paste_id: id: id of paste
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
2.Get Paste
Method: GET
Endpoint: /{version}/paste/{paste_id}
Path Variable:
paste_id: {id}: id of the paste
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
3.Update Paste
Method: PATCH
Endpoint: /{version}/paste/{paste_id}
Path Variable
paste_id: id: id of the paste
Request Body:
new_text: string: new text
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
4.Delete Paste
Method: DELETE
Endpoint: /{version}/paste/{paste_id}
Path Variable:
paste_id: id: id of the paste
Response:
Success:
status: ENUM: SUCCESS
status_code: int: 200
Error:
status: ENUM: ERROR
status_code: int: 4xx | 5xx
Service Component:
Diagram
Gen UID Service:
Database:
Paste expiration:
As we allow the user to set expiration condition for every paste we need efficient way to invalidate a balk of paste. For time condition we just need to map expiration_date with the id. If there is request after expiration timestamp we can easily know that this paste is invalid and delete it. In addition we can have the cron job that will periodically check if at the current time are there any paste that is expired if so we can delete them all for reducing storage cost.
Cache:
Scalability:
We will deploy stateless service on container orchestration service such as AWS ECS. In this case we can scale up/down our container on a whim. We can also set up condition for scaling such as when the request amount of certain service exceed our predefine value. They will add more instance to serve the upcoming load. Also we can adopt fault tolerant architecture by using spot instance to save the cost of infrastructure.