Requirements


Functional Requirements:

  1. Users should be able to create new pastes of text, code etc.
  2. Users should be able to search and view pastes created by them and others
  3. Users should be able to delete pastes
  4. Pastes should get deleted after their TTL expires



Non-Functional Requirements:

  1. The application should be highly available
  2. Creating new pastes should be fast and happen in less than 1 second
  3. Search should be semantic and run in under 50 milli second



API Design

We will need the following endpoints:


  1. /createUser - creates a new user
  2. /createPaste - creates a new paste
  3. /editPaste - edits an existing paste
  4. /deletePaste - deletes an existing paste
  5. /viewPaste - shows details of a paste
  6. /search - returns semantically related pastes



High-Level Design


Capacity estimates:

For a user we store user name, password so each record will be less than 100 Bytes. Assuming 100,000 users, this will be less than 100 MB.


For a post we store created on, TTL, created by, paste body. Paste body can be very small to very big depending on post. Let us assume on average a post will be 5 MB, if we end up storing 500k posts we will need 2.5 TB of storage. Due to the size of our data an in memory database like Redis does not make sense.


Most of our data is going to be of blob type mainly paste text body. We can use MongoDB to store this data and efficiently query it. MongoDB also gives us inbuilt TTL expiry capability, which we will use to delete expired posts.

Application components:

  • Load balancer to distribute traffic
  • Servers running application logic
  • Database


Design:

We will design stateless services for each of the endpoint, so that we can easily scale our app depending on traffic. The database is the biggest bottleneck, so it makes sense to shard our database. Sharding on user_id makes sense as all pastes of a user should be grouped together for faster access, although this will slow down searching. To counter this we can save search results in a Redis cache to improve performance. We can also cache hot pastes on CDNs for faster access and update the CDNs once everyday.



Detailed Component Design