System requirements


Functional:

  • User authorization - users can create accounts and log in and out
    • For simplicity, I'll leave this functionality as a black box and won't dive into its implementation
  • Paste creation - an authorized user can create a pate
  • Paste sharing - a user can share a link with another user so the other user can view the contents
  • Paste search - users can search for public pastes based on title
    • For simplicity, the system will not support tags
  • Expiration handling - once a paste expires, it should no longer be accessible to users
    • For simplicity, the expired pastes will be hard deleted. So they won't be visible in a user's history
  • Paste deletion - the creator of a paste can delete a paste they created



Non-Functional:

  • Abuse prevention - malicious users should not be able to negatively impact the performance of the system. For instance, DDOS attacks and spam traffic should be stopped
    • This can be handled by a service like CloudFlaire
  • Security - Content of private pastes should be encrypted end-to-end
  • Availability - the system should be resilient to failure and continue to serve traffic even if an instance of a component fails




Capacity estimation

  • 10,000 pastes per day
  • 500 characters per paste
  • 1,000 bytes per paste at 2 bytes per character (UTF-8)
  • 10MB / day -> ~4GB / year
  • 8:1 read-to-write ratio
  • 7 writes per minute
  • 56 reads per minute


By these estimates, the traffic is low enough to be handled by standard components (databases and servers) without much, if any, optimization like caches and message queues



API design

  • Paste CRUD endpoints
  • Search endpoint




Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...








High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

  • API Gateway - routes traffic and handles security, DDOS protection, rate limiting, and other utility functions
  • Paste CRUD service - interface to pastes CRUD. It posts messages to the delete priority queue so that pastes can be deleted when they expire without having to perform a table scan on a schedule
  • Delete priority queue - append-only priority queue for pastes to be deleted. The priority is set based on the expiration of the paste, with pastes with lower value for `expires` field being at the front of the queue
  • Delete worker - handles the deletion of expired pastes
  • Paste search service - interface to paste search functionality
  • Search engine - engine used to power the search features in the system
  • Paste index service - service responsible for updating the search index for the paste functionality
  • Event bus - messaging queue used for keeping the search index up to date. Using a messaging queue decouples the CRUD service from the indexing service




Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...






Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

  • Delete functionality - this works by using a priority queue to efficiently determine which pastes have expired recently and need to be deleted. By using a priority queue, the system doesn't have to perform a table scan to find the expired pastes. In addition, the queue is used only to append; if the expiration date for a paste is changed, then the update is simply pushed to the queue and the Expiration worker checks against the database before deleting a record. In the case that the expiration of a paste is in further in the future than the original, when the original is picked from the queue, it'll be checked against the current expiration date in the database. Since the current expiration date is in the future, then the queue item will be dropped as stale. In the case where the date is moved forward, then the queue item will be deleted and the stale item (that will be picked up later) will be dropped as stale
  • Search functionality - the Paste CRUD service pushes updates to the Event bus for the Search index service to pick up and update the Search engine. Since the only searchable field is title, then only operations that change the title or delete an object need to be pushed to the event bus


Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

  • Opted not to use a cache because it's not needed at the scale of the system. A database can easily handle the estimated 10 writes per minute and 60 writes per minute





Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

  • The current system doesn't account for the latency incurred by having all of the infrastructure in a single location. To reduce latency for users farther from the data center, the infrastructure should be replicated in more regions
  • The system doesn't account for concurrent operations. Although this should be fine given the light write traffic (only 7 writes per minute)





Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

  • Replicate the infrastructure to multiple regions
  • Design more thoroughly around non-functional requirements