Loading...
Writes -> 1M / day = 10^6/10^5 = 10 QPS [Peak at 30 QPS]
Read -> 10 * 10^6 / 10^5 = 100 QPS [Peak at 300 QPS]
Storage estimation: Max size of a bin = 10MB, however on an average it will be 10Kb -> 10^6 * 10 = 10^7Kb = 10GB / day = 3.65 TB / year.
Since the expiry is capped at 1 year, we expect the storage to increase during the first year, and then plateau from there on.
POST /paste
Request:
{
"title: string,
"content" string,
"expires_at": string,
"visibility": ENUM,
"alias": string
}
Response:
{
"paste_key": string,
"delete_key": string,
"success": True
}
GET /paste/{paste_key}
Response:
{
"title": string,
"content": string
}
DELETE /paste/{paste_key}
Response:
{
"success": True
}
delete_key is used to validate the user for anonymous posts. For logged in users, only they can delete the posts. This will be passed through header.
We will have a rate limiter for post and get requests for users based on their status.
Anonymous -> Write = 10 req/hr, Read = 100 req / hr
Logged in User -> Write = 100 req/hr, Read = 1000 req/hr
Paid User -> Write = 1000 req/hr, Read = unlimited
Client calls the Load balancer, which also takes care of rate limiting the request.
Write path ->
Write Service validates the size of the paste, checks against known bad words / malicious links -> generate unique paste key -> writes the data to S3 bucket if the size > 64kb, finally adds the metadata in postgres + data (if <64kb).
The data is replicated to the Read replicas, which will be eventually consistent.
The read service reads from the redis cache (cache aside method), else from the read replicas.
Redis stores the keys and the data = {"paste": "content", expires_at: stirng}.
We will use LRU cache eviction policy.
Any read that is served from the system, will be checked for expiry and only served if that criteria is met.
For CDN, we will set the max_age with the remaining TTL.
Deleting a record, will also remove the key from Redis and send a purge request to CDN.
In case of partial outage at DB level, we will continue to serve the cached results, while we will have a circuit breaker on the DB calls and return 503 to the user
We will store 2 types of data in the DB: paste content, metadata.
Metadata content is very small compare to what the paste content could be (10MB for paste, 500kb for metadata).
Hence we will divide the metadata storage and object storage.
We have different types of query patterns here - query by the paste key, query by expiry (for cleaning up), query by visibility (for pastes that people can explore) - hence we can go with Postgres SQL for the metadata storage.
For the actual paste storage, we will use an object DB, like Amazon S3. They provide high availability at lower costs than Postgres.
But this approach will require 2 network hops -> fetch the paste link from postgres and then fetch the paste from S3.
So, for smaller pastes (< 64kb, we can store it in Postgres), while bigger pastes are stored in S3.
Key generation will be handled by 64 bit keys ->
10 bits for shard id and 54 bits for counter -> hash SSH-256-> take 48 bits -> Base 62 conversion.
Duplicates are rare but possible, so to mitigate, we can add salt while hashing in case of duplicates.
Since it is a read heavy operation, we need to add a caching layer and read replicas to handle the read queries.
We will deep dive into the following components:
We will create unique keys by following way.
Each shard will have a 10bit shard id and will keep its own counter in redis -> 10 bit shard_id+ 54 bit counter -> Mask it with XOR of a fixed num to randomize the serialized counter -> SSH-256 hash -> take 48bits and encode to BASE62 to create new links.
This way, each shard is stateless and keep its own count. If the Redis for any shard is down, it will let the load balancer know, and the load balancer can route the traffic to other shards.
Clean up tasks run in the background. It will check for expired keys with the expire_at timestamp and check it with current timestamp. We will limit the number of rows to 1000 at a time, so that all the DB resource isn't captured by the delete job. It will then add the keys to a kafka topic and invalidate the Redis entries and purge data from the CDN
We can add another field called deleted_at and add the timestamp for the expired keys, and then remove the actual data after a week or so, if we want to have a backup mechanism even after deleting (for restoring in case something goes wrong)
We have 2 layer of caching -> CDN (roughly 50m) + Redis (1ms) + Postgres + S3 calls (20ms).
We will add read replicas to speed up the reading latency
To avoid cache stampede, we can coalesce the requests, and 1 request can fetch the data and update the cache, while others wait or serve stale data (in our case, the data never changes, so there is no stale data). Apart from that we can have eager caching for the hot keys when they are close to expiring.
Each client will send an idempotency key, which will be stored and checked against, in case of retry