Loading...
total messages in a day = 10B / day = 10 * 10^9 / 10^5 = 100K messages/sec
Peak messages = 300K messages/sec
Storage = 10B * 1KB / day = 10TB/day = 3.65 PB / year
Throughput = 100K * 1KB = 0.1GB/sec, with peak at 0.3GB/sec
Online concurrent peak connections = 20% of 300M = 60M.
Total connection memory required = 60M * 50KB = 3TB
POST /v1/messages
HEADER:
Idempotency-Key: uuid
Authentication: Bearer
{
sender: uuid,
recipient: uuid,
content: string
group_recipient: group_uuid [EIther recipient or group receipient will be present]
client_ts: timestamp
}
{
message_id: uuid,
message_timestamp: timestamp [Given by server]
status: awaiting/delivered
}
GET /v1/presence-indicator/{uuid}
{
user_id: uuid,
status: online
last_present_at: timestamp #if user is offline
}
WS /v1/ws?user_id=uid
HEADER:
Idempotency-Key: uuid
Authentication: Bearer
{message_id, sender, content, group_id}
POST /v1/chat-group
HEADER:
Idempotency-Key: uuid
Authentication: Bearer
{group_name, group_members: [uuids], description}
{group_id, group_name, group_members, status}
PUT /v1/chat-group/{group_id}
{group_name, group_members: [uuids], description}
{group_id, group_name, group_members, status}
DELETE /v1/chat-group/{group_id}
204 # This can be performed by admin only
POST /v1/chat-group/{group-id}/add_members
HEADER:
Idempotency-Key: uuid
Authentication: Bearer
{group_id, member_id: [uuids]}
{group_id, members: [uuid]}
The client's request is routed through Load balancer and then to API Gateway, where we also handle rate limiting.
We have have Auth Service for authentication, login/logout
Group Service is responsible for creating/updating groups
Chat Service takes users data and stores it in Cassandra. It also emits an event to Kafka, which is picked by Web Servers.
Bus is responsible for maintaining the websocket connection mapping.
When a msg arrives from user_1 for user_2, we check which chat server user_2 is connected to and forward the msg to that particular chat server.
This way we send the msgs to the users in real time.
This also updates the cassandra asynchrounously.
In case user is offline, a notification service picks up the msg and connects with android/ios hooks to deliver push notificiations to the user.
We have S3 buckets when users wants to send images/videos in the chats. We will store the link in the cassandra, while the actual image is stored in S3.
Kafka topics are also subscribed by Notification, Monitoring Services.
Kafka is only used to carry msg and not for handling routing state. Routing is handled by Redis Cluster, where we store the connection link
We receive continuous pings from the user's devices every 30sec, which we keep track of. When a ping comes, it sets the TTL to curr_time+90sec. If 3 continuous pings are missed, the key is expired and the user is deemed as offline. A worker listens to these events and updates the DB and Redis (for caching).
Redis hold presence state only. Cassandra is authoritative, a stale cache entry never loses a message
When a client loses connection, it can send a connection request and also read the msgs from the last known message_id. Kafka provides the functionality of replaying the events.
User Info: Postgres
Group Info: Postgres
Status: Postgres
Chats: Cassandra + S3 (for images)
Redis: Idempotency Keys, connection status, online presence status
User Info:
(user_id, name, email, phone_number, address, ...., created_at, updated_at)
Group Info:
(group_id, name, description, created_at, updated_at)
Group-User Table:
(group_id, user_id, added_at)
Index at (user_id, group_id), (group_id, user_id_
Status:
(user_id, status, last_seen, created_at, updated_at)
Chats:
(message_id, sender, recipient, chat_id (individual/group), status, timestamp, created_at)
The messages will be sharded by chat_id and indexed by chat_id
PK - (chat_id, timestamp)
Redis will be used for caching
Real time Chat:
A clients sends a HTTP request, which is upgraded to Web Socket connection. We maintain the connection list (user to chat_server mapping) in redis.
When a message comes for a particular recipient, we check which chat server the user is connected to forward the msg to that particular chat_server and the msg is delivered.
For images, the link is stored in the chat content and images are served through CDN -> S3.
Group msgs - When a user sends a msg for a group, we will check the status of all the users in the group and connect with their chat servers. This is the reason, we will keep the size of the group small as each msg fans out to all group members.
If a worker fails, connection drops, the client can replay the msgs using Kafka from the last known message_id
The msgs in the Kafka are partitioned by group_id / chat_id so that all the msgs for a particular group are processed parallelly. Also, each msg contains a server assigned timestamp, which is the source of truth for ordering. Client will order the chats based on this timestamp, even if a msg arrives late.
When the member list of a group changes, we invalidate the cache.
The messages are encrypted before sending/saving.
Failures: