Loading...
Assumptions
Bandwidth Estimations
Capacity Estimations
Cache Estimations
post(user_id, photo_content)
get_feed(user_id)
get_feed(user_id, target_id)
We want to have a DB to keep the relationship between users, meaning who are users following. We would use a relational SQL DB like MySQL or PostgresSQL will likely be needed to support this functionality. For photos, we will use a cloud storage like Google Cloud to stores these entities.
In terms of cache, we will use a memcache or redis to support this. We will use the LRU algorithm for data eviction.
The biggest issue here is how can we load the 10 most recent photos for a user in the best way possible?
A naive solution would be to use a SQL query to retrieve a list of tweets based on the user's following and sort them by time. This works, but it is terribly slow for users following a large number of users. How can we do better?
We can do an offline generation of a user's feed. Basically we can have dedicated servers that are continuously generating users’ newsfeed and storing them in memory. So, whenever a user requests for the new posts for their feed, we can simply serve it from the pre-generated, stored location. Using this scheme, user’s newsfeed is not compiled on load, but rather on a regular basis and returned to users whenever they request for it.
Whenever these servers need to generate the feed for a user, they will first query to see what was the last time the feed was generated for that user. Then, new feed data would be generated from that time onwards.
We can store these posts in a Linked List fashion where a post leads to another post.
We should do this generation for users that are online since majority of our users will be offline at any given time, we do not want to waste resources on the offline users. When the user comes online, the system will then start, users would have to wait a few seconds before the feed is generated for the first time but that is a trade off that we are willing to make.
users can continually make new request for feeds by scrolling, as the user reaches the bottom of the page, then the next 10 post are requested
In terms of data partitioning, we could potentially shard the data in terms of userID, but that would mean that for DBs that contain 'hot' users, the DB will be overloaded when users request for the posts of the 'hot' user. What we could do is to use consistent hashing to distribute our tweets evenly among the DBs. this way the load is distributed across the DBs
The bad thing about our current data partitioning strategy is that the user's posts are located in different DBs meaning that we could potentially be searching across all DBs to retrieve a person's posts. But this is a tradeoff that we are willing to make a the moment
Instead of showing posts on feed based on time, we could show tweets based on importance or popularity, meaning that we could come up with an algorithm to determine what posts would interest the target user the most
We could also show other posts of accounts that are not followed by the user if our algorithm determines that this would be of interest to the target user. Of course, this would mean that there will be privacy settings as well.
We can also allow users to post videos. This would take a larger portion of memory in the google cloud.
We can also allow users to comment on other people's posts. This will need additional mechanisms and servers and storage