Loading...
Optional:
Assuming we have 10M DAU, each user send 100 messages per day, message average size 100 bytes.
Total message capacity per day:
10 * 10^6 * 100 * 100 bytes = 100 GB
one year 36500 GB = 36.5 TB
5 year ~ 200 TB
api/v1/user/conversations/create
{
method: POST,
header: token
body: {
content,
members,
}
}
response:
{
conversationId
}
api/v1/user/conversations
{
method: GET,
header: token
}
response
{
conversation {
id,
lastMessageId,
},
}
Send from client to server
Web Socket
{
eventName: "send_message",
conversationId,
message
}
Receive from server
Web Socket
{
eventName: "message_received",
conversationId,
senderId,
message
}
api/v1/conversations/:conversationId
{
method: GET,
header: token
}
response: {
messages: {
{
messageId,
senderId,
content,
}
}
}
User Id | Conversation Id - Last Message Id
Since we need fast access to get all user conversations on initial load and want to auto-scale as each user could have many conversations, we could use Key Value DB like DynamoDB. the value is array of "conversation id + last message id"
ConversationId | Members | Created At| Updated At
This could be SQL as conversation metadata won't change very frequently.
ConversationId | Message |
The read & write throughput on message table is heavy, use wide column table here like Cassandra.
Parition key is ConversationId. And each row, we store message_id, sender_id and content. The message_id would be a unique id generated from message timestamp, sequence id.
We cluster messages based on message_id, which would sort messages by time and sequence.
We need a discovery service to build websocket connection for client to a "best" chat server.
The chat server will handle conversation creation, message send events send from client. Will send message to client on conversation created and message received.
For message sending, it could be sent to all conversation members which could be time consuming, so we need a message queue as a buffer for broadcasting message.
For tracking messages for each conversation, we could take use of client local storage (e.g SQLite). Users could read all history messages even they are off-line. We would tell if there is any unread new messages for each conversation by comparing local and server last message id.
1 o 1 chat is similar to group chat, main difference if the conversation only have two members.
Client builds connection with chat server.
Client sends message to server to create conversation.
Server sends back message to client when conversation is created in the DB. Client will get conversation id and store it locally, the initial last message id for the new conversation could be a negative value (indicates no message).
Clinet sends message to server, which includes conversation id and message content.
Server add sender id when receives the message, also generates message id, the message is persisted in conversation table. The composited message object would then be sent to a message queue. Workers would take message from the queue and get conversation members info, then send message to each conversation member.
If the member is online, server send the message to client directly through websocket, if not, we would send notification.
When client received the message, it would send acknowledge message to server and updates the local storage conversation last message id.
When user logs in, we get all user conversations, the returned conversations is ordered by last message timestamp, so we always have conversations with newest messages in front.
On client side, we could have local storage to store latest message id for each conversation, if the client latest message id is smaller than conversation last message id, it means we need to fetch messages for the conversation.
The chat server is stateful, which means a user would also connect to the same chat server unless it's broken. It makes it's easier to find user's corresponding chat server to send events, also we could make more efficient caching (e.g for conversation metadata, conversation messages). But also introduce more complexity to maintain the user state.
We use WebSocket for bi-directional communication between client and server, it's great fit for real-time messaging, but there is extra complexity to maintain and manage the connection.
We use NoSQL DB to store user conversations and conversation messages. This ensures the system is scalable when conversations and messages grow. However, the NoSQL DB is eventual consistency, which means user could miss some latest messages.
If some chat server has a lot of active users sending tons of messages, it may slow down the MQ process. We could add rate limit for sending messages on client. On server, we could add more workers to consume MQ depending on the MQ load.
When a conversation has a lot of members, new conversation convey could be slow. We could add another MQ to send individual member message. E.g, worker consume the conversation message MQ, from the message, worker produces new jobs and send to each member's MQ. Each member's MQ will be responsible to send message. This way, the sending message could be in parallel.
We could add online/offline indicator for user. We need a separate presense table to store the state. To avoid frequent online/offline changes, we could use web socket to send heartbeat and check user status at some interval.
We could add system health check for each chat server, better load balance.