Requirements

Functional Requirements:

• Users see a feed of posts from people/pages they follow (ranked or recent).
• Support text, images, video, shared links, and other rich media in posts.
• Users can like/react, comment, and share.

Non-Functional Requirements:

• low latency - the users should fetch the feed within 200ms
• scalability is a must - in case of high spikes, the platform should be able to quickly scale out and increase the number of servers; also for large DBs we should be able to implement sharding for faster data retrieval
• high availability: an up-time of at leat 99.99%
• eventual consistency for posts, there's no need for strong consistency as it is too consuming

API Design

GET /feed/me/ - fetches feed for the active user

POST /feed/me?content={content}&auth_token={auth_token} content is the data the user wants to post; authToken is used to check user's permissions. body={media:JSON} on body we can send any other media like images or videos for that post

POST /post/{postId}/react/{reactType}&auth_token={auth_token} used by users for reacting to a post

POST /post/{postId}/comment/{comment}&auth_token={auth_token} used by users to leave comments to a post

POST /post/{postId}/share&auth_token={auth_token} used by users to share a post

POST /notify/user/{userId} body:{notificationMessage:string} used to notify a user of a new post or interactions

High-Level Design

see feed flow:

1. client calls dns to get the ip address corresponding to the domain, it calls then with it the load balancer
2. load balancer distributes evenly the work accross web servers and in case of high spikes, it increases the number of servers
3. web servers take care of authentication and rate limiting and call notification service for any posts notifications. also they call fanout service
4. fanout service firstly pushes to a queue the messages, which are consumed by workers (news feed services)
5. news feed service checks news feed cache to retrieve the {postId, userId} pairs in order to know exactly what to retrieve from users and posts caches/dbs.
6. users service checks the cache for user info and if missing it checks the db.
7. fanout workers then fetch the newsfeed based on its cache and the users and returns it to the clients; we could use a hybrid approach of fanout on read and write; for inactive users, fanout on read is a better approach, this way we don't make too many useless calls to keep the cache in sync with the db and we only fetch data on demand. for active users we can use fanout on write and whenever the db is updated, we update cache too (on each post, interaction, etc);
8. we could use a cdn to deliver faster the newsfeed data like images, videos;

post and interact with posts flow:

1. client calls endpoint for posts and interactions which hits dns, load balancer and web servers;
2. web servers call fanout service and it calls afterwards users graph db and fetches the closest connections of the active user first and afterwards the more distanced ones, in order to know who to show the post to. it then reaches users service where it filters out any blocked/muted accounts or accounts marked not to see the post.
3. then it pushes to fanout queue and to workers in order to update news feed cache for users that get fanout on write
4. web servers call posts server and notification service which emits notifications to the close friends that a new post was created
5. posts server updates checks the cache, updates the db and for fanout on write also updates the posts cache

in case of system failures, we have 5 levels of caching:

1. news feed caching where we store newsFeed ids
2. content: hot cache + normal cache; we store the hot cache separately to prevent hot keys from happening (eg celebrities posts)
3. social graph: separated in followers and followings;
4. action: like/react, comment, share
5. counters: like count, comment count, share count

if any of the db fails, we temporarily use its cache, in case of fanout workers failures, then the message still remains in the fanout queue and can be picked up by other workers. in case the queue fails, it persists its data on disk and when it comes back up it will still have its data

Detailed Component Design

when a post is viral, we will add its content to the hot cache, and we will seperate it from the normal cache so it can be accessed multiple times, if needed we can also store it multiple times accross different caches and regions so it will let more people access it (randomly access one of the caches, eg 4 caches).

users save their preferences for each post using usersService, whenever we compute their feed (after a new post is created), we first check if they have any blocked/muted accounts and filter them out.

to minimize cold caches, we will use a fanout on write approach for active users and whenever a new post is created we update the subscribers newsfeed caches too. this way we will have less cold caches and the system will be able to answer them quicker