System requirements

Functional:

1. 1 to 1 messaging
2. group messages
3. read acks
4. user presence
5. typing indicator
6. notifications
7. message history

Non-Functional:

1. availability - the system shoud be highly available
2. low latency - users should receive messages in real time
3. message order consistency - users should see messages in the same order they were sent so they won't lose context
4. scalability
5. durability - messages should not be lost

Capacity estimation

DAU - 0.5 billion

messages per user - 50

average text payload - 1KB

Storage day - 0.5*10^9 * 50 * 1KB = 25TB -> 25*365 = 9PB year

Bandwidth - 25GB/(24*60*60) * 8 = 2.3Gbps

QPS - 0.5*10^9 * 50 / (24*60*60) = 290k/sec

Server - 0.5*10^9/64k = 7812 servers for peak load

API design

POST /api/v1/{conversation_id}/message

{

auth_token,

text

}

POST /api/v1/messages/{msg_id}/ack

{

auth_token,

message_id

}

GET /api/v1/{conversation_id}/messages?offest&limit - returns

[message_ids]

GET /api/v1/messages/{msg_id} - returns

{

text

}

POST /api/v1/presence/heartbeat

{

auth_token

}

GET /api/v1/presence/{user_id}

Database design

users table SQL

{

user_id,

name,

email,

}

conversations table NOSQL

{

conversation_id,

user_id1,

user_id2

}

messages table NOSQL

{

pk: conversation_id,

sk: sequence_id,

}

acks table NOSQL

{

conversation_id

msg_id,

user_id,

}

presence table NOSQL

{

user_id,

websocket_server,

last_seen: timstamp

}

unread message queue table NOSQL

{

conversation_id,

msg_id,

receiver_id

}

High-level design

load balancer - used to distribute messages accross servers.

api gateway - used to authenticate, authorize, and route requests.

Chat service - used to process user message requests.

presnece service - used to check whether user is online and to which websocket server it's connected to.

Kafka - event storage to separate message events from critical path.

Notification service - send receiver the message if he is online and store the message in a queue.

Request flows

Send message flow:

1. client call POST /api/v1/{conversation_id}/message
2. Chat service gets sequence id for message
3. Chat Service stores message in Conversation DB
4. Chat service acknoledge message to sender
5. Chat service sends a message sent event to Kafka
6. Notification service checks if receiver is online
7. sends the message to websocket server that's connected to reveiver if online
8. stores messasge if unread queue

receiver acknoledge message as read:

1. receiver sends POST /api/v1/messages/{msg_id}/ack
2. chat service stores the ack in DB
3. chat service sends a "message read" event to Kafka
4. Notification service checks if sender is online and sends the websocket server that connected to sender a "message read" ack
5. if sender is offline, it will mark the message as "read" when it connects next time by calling GET /api/v1/{conversation_id}/messages?offest&limit

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?