Loading...
DB:
Assuming the Size of text on average = 50kb per paste
Assuming 100,000 pastes per day * 50 kb = 5GB per day of storage
5GB *365 = 1.825TB of storage for text files in our data base.
Traffic:
Writes:
(100,000/(24*360))=1.16 pastes per second
1.16* 50 kb, roughly 60 kb per second for writes.
Reads:
Assuming 1:1000 write to read ratio:
60 kb*1000= 60 MB
Cache:
1.825TB * 0.2= 256 gb
a 20% cache size is a pragmatic rule of thumb—it’s large enough to cover the working set of frequently accessed content (maximizing hit rates) yet small enough to keep in‑memory costs under control.
We will use REST API's
Since we assume the pastes are anonymous we don't need a users table. We will have one table for pastes as such:
Paste Info Table (SQL Database)
Paste Table: (Key value store such as S3)
MongoDB (NoSQL document database) is preferred for a fitness tracking app because it excels at handling dynamic, structured data with frequent updates and complex query needs. (Fitness app data (workout logs, user progress, and goals) is naturally represented as documents with structured fields. MongoDB stores data in a JSON-like format (BSON), which makes it easier to work with complex data structures and perform rich queries)
Amazon S3 (Object storage service) is optimized for storing and serving static, large objects and doesn't provide the rich query capabilities or flexible update mechanisms that a fitness app requires.
(
)
Since this is a heavy read operation we will use single leader replication(all writes go to the leader node, read replicas (followers) can be horizontally scaled to handle heavy read loads).
To address concurrency
We will repopulate the table with paste_key and paste_url.
"repopulate" means that for each new paste entry, you generate (or calculate) the paste_key and paste_url for that new record and then add that entry to the table—rather than re-generating or rewriting the entire table with every write
This will be created using MD5 hashing. (Rather than waiting for the user to finish writing the paste and then hashing the complete text, the system precomputes a pool of MD5-generated hash pairs. This means the MD5 algorithm is applied to some predetermined input (like a sequence number, counter, or a random seed) ahead of time. These precomputed values are then stored and ready to be assigned as new pastes are created.)
We can take the first 8 values of the hash as paste_key and the second 8 as paste_url. We will not be creating the hash values upon write but pre-populating them.
In concurrent writes, each write can grab a lock for each index, ensuring its thread is safe.
To address deadlocks the locks will only lock the primary index which also serves as the paste_key and then move to paste_url so there is a lock ordering. We can also add timeouts
Lock Acquisition on Indexes:
Enforcing Lock Ordering to Prevent Deadlocks:
Timeouts to Further Prevent Blocking:
If you choose to use just the paste_url as the key—eliminating the separate paste_key—you simplify the schema, but you may lose some benefits:
To address expired rows we have created an index on the created_at field and can run a job every given period to detect old rows.
We will use file storage such as Amazon S3 for the paste themselves. The paste_key will serve as the key to the content of the paste. We are confident that we can horizontally scale a S3 service if needed.
To address latency between the leader node and follower nodes, we can use a quorum strategy to address the delay between followers not being updated.
A quorum strategy can help mitigate the effects of this latency on read and write operations in the following ways:
Let's assume you have 5 nodes labeled 1, 2, 3, 4, and 5. A common quorum configuration is to set both the write quorum (W) and read quorum (R) to 3, ensuring that:
R+W>5(3 + 3 > 5)
This guarantees that the read and write quorums will have at least one node in common.
Write Quorum:
Read Quorum:
Note:
if one node (for example, node 3) is slow, overloaded, or temporarily unavailable, the system might choose another set of nodes (like 2, 4, 5 )
What matters is that any read quorum you form must intersect with the write quorum such that at least one node has the most recent update
Scalability and fault general tolerance:
1. To prevent abuse of our system we can impose a rate limiter based on ip addresses and a bucketing strategy, we can have a higher number for read requests and lower number for writes. If users exceed this amount we can display a message letting them know. The bucketing strategy will prevent distributed types of attacks
A bucketing strategy is a common method used for rate limiting that groups or "buckets" incoming requests—typically by a source identifier such as an IP address—and controls the rate at which those requests are allowed.
2. All read/write requests go through a load balancer that distributes the load based on a server with minimum amount of load. We can use a weighted approach to more effectively address the balancing. We will have multiple instances of our micro services to handle increasing load. The load balancer will distribute the load first via health checks and second by the load each server is having
3. For data base, we will use single leader replication and follower nodes will act as replicas.
4. The S3 bucket will automatically scale but we can have a backup of s3 with features like Amazon AWS back up as a service
5. To help with our read/write throughput we can implement consistent hashing to shard the database. Since our paste_key is a hashed value we can assume a somewhat even distribution amongst the shards and we won't have to worry about hot spots
6. To address expiration of old pastes we will run a job once a day to delete old records. Since expires_at is a key in our db this will be relatively easy.
7. Since the hashes are prepopulated there will be no cases of collisions during the write. To genrate hashes we can use a md5 hash since its cheaper and easier to generate.
Caching:
Writes:
Reads:
same as high level design
same as high level design