System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

1. Allow users to input a large text and store it in the system.
2. Generate a unique URL for each text input that users can access (the user can also name the text).
3. Automatically delete the stored text after 30 days.

Non-Functional:

List non-functional requirements for the system...

1. The system should be able to handle a large number of concurrent users and store a significant amount of text data efficiently
2. Text retrieval and storage operations should be fast and responsive to provide a seamless user experience.

Capacity estimation

Estimate the scale of the system you are going to design...

Assume we have 1000 initial users, and an average user would create 3 pastes per day. That makes 3 * 1000 / 100000 = 0.03 write request / second in the initial month.

Everyday 3 * 1000 = 3000 pastes would be created initially, assuming each paste would be accessed 5 times in its lifetime, the read traffic could be estimated as 3000 * 5 / 100000 = 0.15 read request / second.

Initially, the traffic is very small, however, we should assume the number of users would grow gradually at 10% per month. The read / write traffic should grow linearly according to this.

Assume each text and its metadata would take 1KB in storage, the highest storage we need initially would be 3000 * 30 * 1KB = 90MB

Again the storage needed initially is small, but as the number of customers grow over time, we will need greater storage.

API design

Define what APIs are expected from the system...

storeText API:

request: customerId (string), textToStore (string)

response: uriOfText

fetchText API:

request: uriOfText

response: text (string)

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

Table: Text

Id (varchar): id of text

ownerId (varchar): customerId of the creator of the text.

textToStore (varchar): the text to store

uri: uriOfText

insertTime (DateTime)

expireTime (DateTime)

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

The system should consist of a client, a server, a database to serve the basic requests. In order to remove the expired texts from the database, there should be a seperate cronjob that is executed daily.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

The Client should issue a storeText API call on behave of a customer. This API call contains the customerId of the owner and the actual text to store.

The server should then calculate the expiry time of the text based on the current date, as well as generate a uri to fetch this text. The server then inserts all these information as a new record in the database.

After database insertion succeeded, the server should then returns the uri to fetch the text to the client.

The client should issue a fetchText API call to the server. The server then use this uri to query database to get the text. After the text is retrieved, it returns the text to the client.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

1. The server. We need to use a good algorithm to generate the url of the text. We could use [0-9][a-z][A-Z] characters in the url, and that makes it 62 characters. To make sure all urls are unique, initially we need to make sure the url hash is at least 3 characters in length, however as more and more users are joining and using the service, we can maintain longer urls. We should generate a random string for the url. If there is a collision, re-generate the string.
2. Database. Initially, the database is very small and can be fitted into a single machine, however, in order to maintain high availability, we should provide duplications of the database. Initially we can have 1 redundant duplication of the database, and use a master-slave config. We should always write to master database machine and have it send its data synchronized to slaves. The slaves could serve reads.
3. When user base has grown enough and the database size has grown large, sharding of the database should be introduced. As the number of users is expected to be growing continuously, consistent hashing should be used to distribute the stored content.
4. Similarly, our server should have duplication, to serve all the requests. Besides, when more and more users are joining the platform, more machine should be added in the server layer.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

In the design of random url generators, we choose to generate the url completely randomly. There are to other way to do this.

1. Use a hash function to compute the hash value of the text, and then use the first n characters (n being the number of characters currently needed to accommodate uniqueness of all urls), if there is a collision, choose the next n characters, and so on... if the whole hash is used up, fallback to randomly generating the whole string. This is not chosen as its computational cost is very high.
2. Use the database row number in the url. This method is not as secure as randomly generating url.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

If one of the servers fails, the clients could call other servers and get their requests served.

If one of the databases replication fails, and it is a slave machine, the other slaves machine can come in and serve the requests.

If the master database replication fails, one of the slaves can be elected as the master.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

As mentioned above, since there will be more and more users joining the platform, database sharding should be introduced.