Loading...
business objects:
relationships:
I'm going to assume we do not have accounts and pastes are immutable for the time being. we can explore those aspects in more detail later. right now there's just pastes and links. Anyone can create a new paste and share the link, and you only need the link to open it. I'm also going to assume there's no password protection or anything on the pastes
I will also assume a 1k character maximum on pastes for now. That might be too small, but it's a good first guess
consistency can be eventual
availability is very important
performance should be reasonably fast
this will be more read heavy than write heavy. We'll want to be horizontally scalable so we can handle peak demands
for security, we already have a size limit, which is the most concerning portion. We'll also need to make sure we sanitize any pastes so we and are users are not vulnerable to sql injection, xss, etc
we might want to consider having a TTL for pastes so our database doesn't get too big over time
Let's say 50M daily reads and 100k daily writes, for a 500:1 read/write ratio. So mostly read heavy
qps
If each paste is 1kb, then 50M/86400*5=2900 reads/s peak that we should be prepared for
100k/86400*5=6 writes/s peak that we should be prepared for
we can have far fewer write replicas than read replicas/services
storage
if each paste is 1kb on average, then
100k*365*1kb=36GB stored per year. We probably don't need sharding since that's so small, but good to start thinking about nonetheless
POST /paste
body: {text: ""}
returns:
200 created successfully, and returns the URL you can use to see the paste in the future. The ID it generates can probably just be a UUID. It won't be easy to parse, but you're almost guaranteed no collisions and people will just be pasting the IDs anyway
400 body was malformed
GET /{id}
returns:
200 with the content of the paste that was requested
404 id doesn't exist
since this uses a primary key lookup, this should be very fast
we start with the database, which should be nosql so that we can have very fast reads and horizontal scalability is easier. We are fine with being eventually consistent on writes
The database sits behind read and write microservices. Since we're so tilted in favor of reads, we should probably have read replicas of the db so it doesn't get overwhelmed. The write microservices write to the main db, which then gets replicated to the read dbs. The read microservices only ever talk to the read replicas. This also gives us fault tolerance, as if we lose a read db it's no big deal. We can rebuild it using a quorum from the remaining dbs
the microservices live behind the load balancer. simple round robin is probably sufficient for now for the load balancer, since each request should be roughly the same amount of work
we will need a cache to take pressure off the db and to serve frequently used pastes very quickly. A write aside cache with LRU should be sufficient. query the cache for an ID first, and if it doesn't exist there then go to the read replicas. Should aim for 80% of traffic going to cache. we should monitor user behavior as well. If it turns out that the normal usage pattern is someone creates a paste, 5 people immediately read it, and then it's never used again, then a write through cache might actually be more efficient.
we can also have a cdn in front of everything, since pastes will probably be geographically distributed in hot spots
the only thing we need to store are the pastes themselves
ID: the UUID for the paste, primary key, shard key
text: the text of the paste
createdAt: DATETIME, nice to have but not necessary
we can use nosql for this, since we want reads to be very fast and we're fine with being eventually consistent on writes. It also makes horizontal scalability easier
the ID generation should be very simple. our nosql database will have a built in function to generate a UUID, and we can just send that back to the client when they create a paste. We do not need a separate uuid generation service
let me expand on the decision to limit pastes to 1k characters. We might want to not do that in the future, or we might want to allow people to paste something like an image. In that case, we should probably shift to blob storage instead of trying to put everything into the db. In that case, the db would not hold the data itself, but just a uri to the correct place in the blob. In that case the read service would have to first look up the paste in the cache, then the db if it misses, then actually retrieve the object from blob, then write the uri to the cache if necessary and return the data to the client. I strongly think we should keep some limit though, even if it's very high, to prevent resource exhaustion attacks
performance is mostly kept high by the performance of the cache, the read replicas, and the access pattern of only getting things by primary key. Additionally, we should have automatic horizontal scaling, especially on the read replicas, to make sure we can handle load