Requirements

Functional Requirements:

• Allow users to tweet messages up to 140 characters.
• Enable users to follow other users.
• Allow users to like tweets from other users.
• Display tweets from followed users in the home feed.
• Show top K popular tweets in the home feed based on likes and followers.

Non-Functional Requirements:

• Multi Cluster
• Availability over consistency
• 10000 tweets/day
• 1:10 write/read ratio
• p99 latency - 1s
• 10,000 QPS
• Requires 99.999% uptime and no data loss

API Design

1. /POST - post tweet
2. /POST - load
3. /activity - like or follow
4. /POST - ingestion of tweets

High-Level Design

This problem requireds 2 micro serivce, one is loading home page and another to store user info and maintain it's social graph

Talking about first:

okay, going step by step on the API

1. posting a tweet
   1. persist
      1. it should be a high available DB and durable
   2. Publish to other people feeds
      1. An async event to store this new tweet
   3. response back to customer
      1. success
2. loading homepage
   1. get curated feed for user
      1. A read call to a pre computed queue assigned to the user at that time by some external system

b. display topK

1. ingestion - trigger to precompute the queue for the user
   1. queueEMptyEvent() or newFeedEvent() can be a trgger
   2. will get the feed ready for the user by some algo
   3. push into a queue in order of priority
   4. cache for sometime

Regarding NFR

1. Assuming general request payload is , 4kb - so 10000*4 = 40 thousand kb - 500 KBps and same for response so, a total of 1000KBps or say 1 MBps -> any general ec2 can handle
2. Read/write 1:10 so, so, write heavy db which can scale better with data
3. keeping it multi clsuter behind an app gateway + elb (if any authz or authn or observability)
4. metrics and alerts setup - any telemetry rool can help
5. Multi cluster in differetn zones with each cluster is provisioned to handle 1.5x of the expected traffic

Talking about second:

1. activity - like or follow
   1. both users Id, activityType, triggeredBy
   2. persist
   3. event to maintain user relations

Detailed Component Design

First microservice:

okay, so let;s setup schema real quick:

Tweet

a. tweetid

b. user

c. tweet text

d. ts.

Structure of messafge stored inthe queue

TweetId

Tweet metadata

targetUserId

priority

Audit table to store sent msgs or some metadata in the queue(if business asks for it) - future scope

Now, for 1st should be NoSql - since tweet can explode, need vertical scaling and new params can come

2 is just a java classm

Regd User Relations microserfice

1. User table
2. A neo4j db call to update user relations

Regarding the ingestion part:

this should be a cron job, every 5 mins or 10 mins (configurabel by the business based on user activity probably or whatever business want to track)

and can be triggered nefore as well in case of queue empty() event or fasterConsumption event

it will call the below microservice and get the nearest K users for the given user and get their top tweet from the tweet table and feed it to the queue.

for this queue i am thinking kafka (since i need ordering) for smaller usercase rabbit mq can be used but in long term kafka is better here

regd the infra setup, probably 6 (3+3) general purpose ec2 servers (hosting 3 microservices of homefeed and 3 of userRelations)

1 MYsql

! neo$j

1 nosql

1 Redis

1 Kafka setup

1 Classic Load balancer + api gateway

A CDN too if global customers

1 noSql