Loading...
List the key functional requirements for the system (Ask the AI for hints if stuck)...
List the key non-functional requirements (performance, scalability, reliability, etc.)...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
POST /api/v1/uploadImage
POST /api/v1/post/{postID}/like
POST /api/v1/comment
POST /api/v1/follow?userID
POST /api/v1/unfollow?userID
POST /api/v1/register?name=&password=&email=
POST /api/v1/login?username=&password=
GET /api/v1/notify
GET /api/v1/feed?limit=20&after=
DELETE /api/v1/post?postId=
ReadPath: Client -> CDN -> API Gateway -> Feed Service -> Redis -> MongoDB
CDN: It is used to host the frontend of the service. The response is fast because it is in edge locations. Most of the time it handles 80% of the reads since it can cache as well.
API Gateway: To route traffic based on the endpoint getting hit. Used for authentication and for rate limiting the requests.
Kafka: It is used as a high throughput queue to store the requests that are received. With this even if there is a down time it is easy to recover as kafka will have the missed requests. User will receive 200 OK
Redis:To cache the requests and take care of concurrent redundant requests and reduce load on the db and response time. and to store sorted sets so that delivery is fast.
Blob Storage: Great way to store the media that is uploaded into the platform we can use services like S3 for this.
Feed Generator: Generates feed by taking in the incoming data from the CreatePost service and is used by the Feed service. We will be using the Hybrid fan-out strategy where we will push to every follower's feed for normal users and pull at read time for celebrity.
Feed service: Uses the Feed Generator service generated data. Checks the redis first if not hit goes to Mongo and then makes an entry in redis so as to the next hit happens successfully at redis itself.
Mongo: Document DB used for this use case.
Register/Login Service: Use to register or login into the application
Create Post Service: Creates a Post for the user.
Follow/Unfollow Service: Follows or unfollows the user id. when a user follows another user an event is sent to kafka which sends data to notification service notifying the user they got a new follower at the same time feed generator will include the following users post.
Each operation is going to be having an idempotency key to avoid duplicate executions. such as following user twice or posting twice when there is an interuption.
The idempotency keys are stored in redis key value store and will have a ttl of around 24 to 48 hrs after which it will be discarded.
Notification service is used to notify the respective users of their subscribed activities. It will receive data from Kafka.
If there is a failure we can failover to the regional replicas keeping the availability high.
We will be going with mongo for this case.
User Collection:
UserID
Followers
Following
PostCount
Post Collection:
PostID
UserID
CreatedAt
UpdatedAt
Likes
Comments
Media presigned URL
Follows Collections:
FollowerID
FolloweeID
Timestamp
Likes Collections:
PostID
LikesCount
Comments Collection:
PostID
CommentID
Comment
CreatedAt
Feed Collection:
UserID
Posts
Will be having recent N thousand
User Graph will also be stored in this itself.
Stories will live in a store with built-in expiry.
Feed Generator: Generates the feed for the user. Generates feed by taking in the incoming data from the CreatePost service from the following of the user and is used by the Feed service. We will be using the Hybrid fan-out strategy where we will push to every follower's feed for normal users and pull at read time for celebrity.
Notification Service: This is used to notify the users about their posts likes, dislikes, comments, followed etc., We will be fanning it out to the followers if its a post through push strategy.
If a action fails we even retry it for a set amount of times lets say 3. If it fails more than that set limit it will go to the DLQ. We will be retrying using exponential backoff with jitter to avoid multiple retries at the same time. This will prevent any load on the respective service.
We will be having replicas to adhere to high availability so that when ever a service fails it's replica takes over. We can go even further by having regional failovers so that if a region goes down we can switch to the other region.
We will be using sharded counters to avoid hot spots created due to viral posts. we will be spliting it into N shards and increment independently. We will flush aggregates to durable storage on a timer.