Loading...
Capacity Estimation:
Traffic:
Storage:
Bandwith:
Create Paste:
Endpoint: POST /v1/bin
Description: Create a new paste and generate a unique URL code
Message Body:
Success Response:
Error Response:
Authentication: Token in the Authorization header. authentication is optional for creating pastes, the rate limit for creating pastes is higher and better managed when users are authenticated, also users would be able to edit and delete the pastes they create.
Rate Limiting:
Guest Users: 10 Requests per minute per IP
Authenticated Users: 20 Requests per minute per user
Returns 429 (Too Many Requests) when exceeded, with a Retry-After header indicating when the client can try again.
Validation: The server should validate the body fields to make sure they are correctly formatted and to avoid any abuse
Retrieve Paste:
Endpoint: GET /v1/bin/{code}
Description: Get a bin by code, if it was set to burn on read, also change the status do inactive
Path Parameters:
Success Response:
Error Response:
Rate Limiting:
Guest Users: 20 Requests per minute per IP
Authenticated Users: 100 Requests per minute per user
Returns 429 (Too Many Requests) when exceeded, with a Retry-After header indicating when the client can try again.
Edit Paste:
Endpoint: PATCH /v1/bin/{code}
Description: Edit a paste text and name
Message Body:
Success Response:
Error Response:
Authentication: Token in the Authorization header. authentication is required for editing pastes.
Rate Limiting:
Authenticated Users: 20 Requests per minute per user
Returns 429 (Too Many Requests) when exceeded, with a Retry-After header indicating when the client can try again.
Validation: The server should validate the body fields to make sure they are correctly formatted and to avoid any abuse
Delete Paste:
Endpoint: DELETE /v1/bin/{code}
Description: Delete a paste
Path Parameters:
Success Response:
Error Response:
Authentication: Token in the Authorization header. authentication is required for deleting pastes.
Rate Limiting:
Authenticated Users: 30 Requests per minute per user ( at a rate of deleting 2 pastes per second )
Returns 429 (Too Many Requests) when exceeded, with a Retry-After header indicating when the client can try again.
Database:
The data flow relies heavily on generating URL short codes and on creating paste text related to these codes; the main workload would be focused on the generation of these short codes and then on retrieving specific paste text that is related to them. The flow will be read-heavy, retrieving shared pastes through the code from the URL. The estimation is 20:1 for 20 reads per 1 created paste.
In this scenario, the use of a key-value store database makes the most sense as the data would be heavily reliant on retrieving pastes by short code or on writing to the database with the short code as the key. No heavy join operations are required, relational constraints, and the data does not need to be normalized.
Write Flow:
The database writes need to be consistent; a small delay would be ok, yet consistency and ensuring the data is available and correct are of high importance. The flow would be Client --> Load Balancer --> Code Generation Service --> DynamoDB --> Redis. The write flow should be synchronous, as when a user creates a paste, they should only get a successful response after the paste has been persisted.
DynamoDB should use conditional writes to make sure no collisions in URL generation occur as another safety mechanism on top of the code generation service.
Read Flow:
The Read flow requires low latency and eventual consistency, we need to make sure the data is available when a URL is used, in the case of a burn after read event, we need to make sure that the paste is read only once, we need strong consistency for this case and this would take a synchronouse path with atomic database read and update. we need to make sure no cache is stored for burn after read pastes, yet for the normal flow of normal pastes using cache is key to ensure optimal reads and low server load.
Normal Flow: Client --> CDN --> Load Balancer --> Redis Cache --> DynamoDB
Burn After Read Flow: Client --> Load Balancer --> API --> DynamoDB (Cache-Control: no-store)
Cache:
Different layers of caching to make sure best response time for reads while making sure no cache is stored for burn-after-read operations.
CDN level caching with regional edge locations.
Application layer cache with a distributed Redis cluster, implementing cache partitioning by code, cache-aside with locking to make sure that only 1 request reaches the database in case of a cache miss, and writing the value to the Redis cache for subsequent queries, and preventing the risk of a cache stampede.
burn-after-read pastes bypass caching and go directly to the database.
Application Layer:
Code Generation Service (Multiple Instances):
Generate URL short codes for pastes at write time. The key purpose is to avoid collisions in code generation while still keeping low latency when creating pastes.
Collision handling at the level of the database is also an extra layer of protection. Conditional writes using PutItem with the condition.
Range-based allocation for ID assignment means writes succeed on the first time, as no 2 generated IDs can be the same, while each server has an ID pool to use, and the pools are tracked in the database
Paste Retrieval Service (Multiple Instances):
Look up paste URLs and return results to the user.
The request goes through CDN -> Load Balancers -> Application Servers -> Redis Cache -> Database
The first cache layer is the CDN for the fastest retrieval of request information. If the CDN cache is a miss, the request continues to the LB, which directs the traffic to the Retrieval service, which then looks up the system cache layer (Redis), in case of another miss, the service performs a database lookup for the data.
Redis:
A cache-aside strategy is used to ensure data is cached only after it is requested, and cached data does not pollute the cache with unused pastes.
Cache Management:
Eviction Policy: LFU (Least Frequently Used) to retain high traffic pastes, explicit invalidation is performed on deleted or expired entries.
Caching:
Multiple layers of caching are implemented to ensure the load is managed at the application layer and at the Database layer, with the database connections being the bottleneck.
1- CDN (hot tier):
The CDN caches user requests at edge locations closer to users, reducing latency and server load. Cache policies are implemented to manage the stored requests using the Cache-Control header to make sure only appropriate data is cached. TTL based expiration ensures updates are timely propagated.
2- Application Layer, Redis (warm tier):
The Redis layer ensures warm requests are served from this cache. multiple shards based on the short code, a cache-aside pattern to ensure data is cached only when used, and no cache pollution with never-requested data. LRU (Least Recently Used) eviction policy to ensure the cache has data that is used most frequently and most recently.
Partitioning:
Data in both Redis and the database is partitioned by shortcode; this means each lookup path would hit one Redis shard, and if not found, one database partition to find the requested data.
Database:
DynamoDB, an AWS-managed key-value store type database, is recommended in this scenario for the simplicity of its use, and the data is also best stored in such a database for fast lookups and reliable writes.
CassandraDB would be another choice, but it would require much heavier operational resources and knowledge.
Both databases are highly scalable and support automatic partitioning/sharding out of the box, without the need for manual sharding like in SQL databases.
Code Generation Service:
Collision Handling:
Collision handling using generated range-based pools of IDs and unique writes to the database. This approach allows simplicity as there will be minimal runtime coordination between machines, and ranges would be tracked in the database.