Loading...
Traffic Estimations
Bandwidth Estimation
Read
Write
Storage estimation
post_tweet(user_id, tweet_content) - stores the tweet into the system's storage
search_tweet(user_id, keywords, target_id=None) - searches tweets based on the search query, the default target_id will be None unless the user wants to search tweets specific to a certain user
The most important question would be how can we store the tweets as efficiently as possible? A simple way is to store the tweets in something like a SQL database like MySQL or a documentDB like MongoDB, when a user enters the search query, we can search the database for tweets that contain some of the keywords and return them to the user. However, this would be largely inefficient as this method requires having to query the entire database multiple times each time a search query is sent through.
We need another type of DB that would allow us to retrieve the tweets more efficiently. A possible solution would be having certain keywords as the key value and an array of tweet_ids as the value. We can store this key-value pair relationship in a key-value DB like a DynamoDB. This would be better because as we can just return the tweets that are in the array of the corresponding keywords.
Client connects to the backend server, which connects to the key-value DB, and also the tweets DB
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
In terms of the DB design, we do not want to store every single word in the query in the DB. Words like 'the' 'a' 'is' do not provide any value in search queries as they would be present in about 95% of tweets. Hence for these types of words, we will not create a key-value entry in the DB. However words, like 'explosion', 'Excited', 'Concert' provides more value, and hence will be kept in the DynamoDB database
When the system attempts to retrieve the tweets from the corresponding tweet_ids, what we dont want to do is to retrieve ALL the tweets at once, we should do them in chunks to avoid large latencies. Hence, for each keyword in the search query, the system will retrieve about 10 of them at a time. In addition to this, we must arrange the tweets based on how many keywords are present in the tweet, meaning a tweet containing the words 'Taylor' 'Swift' 'Concert' will be placed ahead of a tweet containing the words 'Taylor' 'Swift'. To do this, we could maintain a HashMap where the key is the tweet, and the value is the count of how many keywords are present in the tweet.
In order to return the chunks to the user with little latency, we want to be able to kind of pre-load some of the chunks even before the user requests for them. What we could do is to maintain a linked list of chunks for every user, every time a user requests a chunk, the front most chunk will be returned. In the background, the system will continue to load chunks and add them to the linked list, this is so that even as the user requests for more chunks, the chunks would already have been pre-loaded, hence allowing for better user experience.
We would also want to have a Cache to cache the results of some of the more popular keywords. This can be done by caching the top 20% of keywords searched a day, the eviction algorithm would be least recently used.
In order to make the system more robust, we need additional servers and databases that would serve as load distribution and contingencies just in case a server crashes.
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
For future improvements, we could devise an algorithm that returns some of the more popular tweets based on the keywords, or tweets that might be of interest to the user