System requirements

Functional:

• Register/login
• Send/receive real-time messages
• Offline delivery
• Message persistence
• Delivery acknowledgment

📌 Non-Functional:

• Real-time (low latency)
• Highly available
• Horizontally scalable to billions of users
• Secure (eventually support end-to-end encryption)

Capacity estimation

Assume:

1. The daily active users (DAU) is 500 million, average 10 messages have been sent, 1 / 10 is video, and 2 /10 are photos
2. Every text message is 256 Bytes, every photo is 1MB, every video is 5MB

Storage:

500m * (7 * 256B + 2 * 1MB + 5MB) = 4PB / day

Bandwidth

4PB / 86400 s = 46GB/s

send + receive: 46GB/s * 2 = 92 GB/ s

API design

/message

POST sendMessage(message_id, group_id, sender_id, receiver_ids, create_time, type, text, media_object=none, document=none)

GET getMessage(user_id)

/file

POST uploadFile(file_type, file);

GET downloadFile(user_if, file_id);

/list

GET getMessagelist(group_id, timeSpan) 分页

/group

POST group(group_id, users{}, owner_id, create_time)

GET group(group_id)

Database design

We need to store the metadata of groups, users and messages

For groups' and users' information, we can use the RDB (MySQL, PostgreSQL) because they are structured, but consider we will support 1B users; RDB is hard to horizontally scale. A NoSQL database like Cassandra would be better.

For the messages, we could also use Cassandra to store them, but for the media, we need to use a blob database.

Users:

userId

userName

createTime

lastLogin

Group:

groupId:

groupName

createTime

ownerId

members

High-level design

We have 3 ways to implement message sending and receiving:

1. pull: we will need a lot of request for the scenario, and it can not support the offline scenario
2. Long pull: better than pull, can support offline, but it still needs a lot of requests.
3. WebSocket: the best approach. When the connection is built, users can real-time connect each other during the session, quickly and efficiently

Components

ApiGateway: user auth

CDN: frequently used media

DB: for users and group metadata

Blob DB: media

cache: frequently used media and message

Load balancer: balanced forward the request to different server

Message queue: for those offline message

1. The user 1 and user 2 create a communication channel with the chat server.
2. The user 1 send a message to chat service, when the chat server receive the message, will ackowledge user 1 the message has been received.
3. The chat service store the message to a DB and forward the message to user 2 if it is online.
4. If user 2 is offline, chat server will hold the message to a transient until user 2's status is online
5. When the user 2 receives the message, it will also send an acknowledgment to the chat service.
6. The chat service acknowledges user 1's message has been successfully delivered
7. When user 2 reads the message, the application notifies the chat server.
8. The chat server notifies user 1 that user 2 has read the message.

Request flows

1. The user 1 and user 2 create a communication channel with the chat server.
2. The user 1 send a message to chat service, when the chat server receive the message, will ackowledge user 1 the message has been received.
3. The chat service forward the message to user 2 if it is online.
4. If user 2 is offline, chat server will hold and store message to a DB until user 2's status is online
5. When the user 2 receives the message, it will also send an acknowledgment to the chat service.
6. The chat service acknowledges user 1's message has been successfully delivered
7. When user 2 reads the message, the application notifies the chat server.
8. The chat server notifies user 1 that user 2 has read the message.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?