Loading...
Authentication
Managing chats
Handling messages
Receiving messages is an event sent by websocket connection, managed by kafka event bus, which then pulls the message data and the app renders them in timestamp order also relative to the user's writes
One websocket connection per user. All chats will go through this websocket connection (as per our capacity planning)
Sending messages is a websocket connection event that groups together text and media data.
If a message contains media, handle it separately:
so that the api doesn't have to deal with GB/s uploads of media, if a message contains media then make a call to generate a temporary pre-signed blob storage link to upload the media to concurrently on a separate plane from the api gateway. expire the link after the media is done uploading (use events of successful finished uploads routed by kafka event bus)
When a client uses the service from their phone or laptop, they should be able to hop to another device and have the message history sync up using the get /v1/msg/:chatId call
Blocking users
To start a session, clients first hit a regional load balancer and get routed to an api gateway by round-robin (all session starts are equal). Api gateway handles user auth/token generation, input sanitization/validation, rate limiting before sending to a microservice replica.
Session microservice: Manages websocket connections. Creates a websocket connection for a user starting a session and passes back the handler for it, talks directly to Kafka event buses to receive "message received" events to then forward to clients. Cleans up websocket connections after enough inactivity.
Event microservice: Replicas of Kafka event buses that send message received events to session microservice when a new message is committed to a user's message database.
Message microservice: When a client sends a new message through their websocket connection to any chat, that client's table (1 message table per client) gets a new message (row) appended to it with the textual components, references to media, a timestamp, and a bool flag for if it's been deleted by that user or not. A redis key/value store maps chat ID to metadata like chat name and chat profile picture. If a user deletes one of their messages, the message's "deleted" col is set to true here and is reflected immediately for the user (read your own writes).
Media microservice: When a client attaches images or videos or other arbitrary files to share over a chat, the separate api call for media hits this microservice and generates a pre-authorized blob storage url, sending it back to the client for the client to upload to directly. This keeps load of several GB/s off the api gateway. Larger files are partitioned into chunks and sent sequentially to be resilient to network failures. Once the media are uploaded to storage, an event is sent through kafka that is watched for here, where the image media is redownloaded to create smaller resolution thumbnails for previews in the messages. Finally the message microservice is contacted to edit the row for this particular message ID to have references to the relevant blob storage links. This service ensures that the pre-authorized upload urls expire once the upload is done and read-only upload urls are generated and edited into the relevant message row.
User microservice: Handles blocking of users. If a user blocks another, a key/value store containing uuids as keys and an array of blocked IDs is updated. When a user tries to add someone to a chat, this redis db is checked first to see if it's a valid request. When a user unblocks a user, it takes effect here too.
Clients making request first hit an edge CDN to primarily serve images or media, this can save a lot of latency for group chats that all see the same media shared among everyone.
This is a high throughput system (more than 200M writes per second) so use NoSQL database like DynamoDB for easy and adaptive horizontal scaling for the messages database. Shard by uuid which should be randomly generated, and use consistent hashing to gracefully horizontally scale up new db replicas.
For messages there will be 1 table per user. Index will be messageID which is unique per chat for all participants.
Use a PostgreSQL db for the user db since it's much less frequently written to and it's important to keep up-to-date and consistent data about who blocked who, for safety.
For message deletion, use a soft delete: hide it for the users but set that message's status to "deleted" in the db. Let a periodic background job that runs daily clean up these messages as deletion can be an expensive process.
Media microservice: When a client attaches images or videos or other arbitrary files to share over a chat, the separate api call for media hits this microservice and generates a pre-authorized blob storage url, sending it back to the client for the client to upload to directly. This keeps load of several GB/s off the api gateway. Larger files are partitioned into chunks and sent sequentially to be resilient to network failures. Once the media are uploaded to storage, an event is sent through kafka that is watched for here, where the image media is redownloaded to create smaller resolution thumbnails for previews in the messages. Finally the message microservice is contacted to edit the row for this particular message ID to have references to the relevant blob storage links. This service ensures that the pre-authorized upload urls expire once the upload is done and read-only upload urls are generated and edited into the relevant message row. When a client retrieves the latest messages because of receiving a Kafka event, first try to retrieve it from the CDN edge servers - this will save a lot of latency on group chats where up to 500 people view the same media.
Message microservice: Ensure that all messages are sent with a unique message ID (unique per chat) that serves as an idempotency key, along with a timestamp. The message database will store these message IDs in redis cache replicas to quickly check if an incoming request is a duplicate or not. This will then ensure that no duplicate messages are sent. Another important point is to make sure that messages are presented and stored in time order. This is done by logging the timestamp with which each message was sent (timezone of client and the time it was received by api gateway). That way the source of truth is the message db which stores an absolute timestamp of each message. In the event of an outage or needing to restore from a backup, the ordering and contents of messages stored in the msg db will ensure a faithful and accurate recreation of messages without duplicates and in the correct order. Clients will use retries with a limit and exponential backoff + jitter to retry sending messages until they go through - if the message fails to send within these bounds, notify the user and let them manually resend the message. This gives us more bandwidth to work with in our api gateway and services.
Event microservice: Kafka event bus is used to update clients when new messages are received from any of their chats; one websockets connection per user so they all flow through here. The event bus can transmit batches of events instead of individual updates to minimize repetitive overhead of single small events. It also facilitates communication of updates of when media is finished processing.