Loading...
business objects:
message
user
group (a dm is a subset of a group)
messages will be immutable, but deletable
groups are mutable
users a mutable
relationships:
send a message from a user to a group
delete a message
create a new group with certain users
delete a group
given a group, get the most recent messages (should be paginated, but behind an infinite scroll)
add a user to an existing group
remove a user from an existing group
create a new user
delete a user
given a user, get the groups they are in
I am not going to talk about read receipts in this, though if I can if desired. I'm also not going to talk about friends right now
eventual consistency is fine
high availability is very important
messages should be sent quickly
we should be able to scale horizontally to meet demand
security is important in that people can't access groups they're not part of
people should be able to log in and see their chats as well as the most recent message in each group
let's say 100M DAU? so maybe 20M peak
so if each user sends 50 messages a day and reads 200 messages
that's 100M*50=5B writes/day
20B reads/day
at peak, we'll have 20M users, each sending and receiving messages. say it's over an hour, they send 20 messages and read 80.
so 20M*20/60/60=100k writes per second peak
20M*80/60/60=400k reads per second peak
so slightly read oriented, but we're going to have to make sure we're very well scaled for both
for storage, say each message is 1kb on average, since sometimes people will send pictures or videos
5tb/day or 2pb per year on messages.
for group and user metadata, 500M users/groups=1.5tb of data
So a massive amount of storage, we will need to shard aggressively
POST /message send a message from a user to a group
body: {groupId: groupId, text: string, timestamp: DATETIME, media: blob, userId: user}
200 message sent successfully, here's the messageId
403 user not in group
404 group not found
DELETE /message delete a message
body: {messageId: id}
201 message deleted
404 message doesn't exist
403 not your message
400 malformed
importantly, we shouldn't actually delete the message row in the db. We should remove the data but we should keep a reference to it so we can show that something was deleted
POST /group creates a new group
body: {groupName: text, users: [list of users]}
200 group created, here's the groupId
400 malformed
DELETE /group/{groupId} delete a group
the user deleting the group must themselves be in the group
201 group deleted
404 group doesn't exist
403 user does not have permission to delete group
GET /group/{groupId}/messages?before={timestamp}&limit={numMessages} given a group, get the most recent messages (should be paginated, but behind an infinite scroll)
body: {timestamp: DATETIME, numMessages: number}
both things in the body are optional, and if neither are provided then it just gets the most recent message. Otherwise it starts with everything before the given timestamp and gives you numMessages number of messages. When the user scrolls up, the client has to use the oldest message's timestamp to send the right request for older messages
200 here's numMessages message objects
404 group doesn't exist
403 user is not in group
POST /group/{groupId}/user add a user to an existing group
body: {userId: user}
201 user added
403 user not authorized to add to this group
404 user doesn't exist
400 user is already in group
DELETE /group/{groupId}/user remove a user from an existing group
body: {userId: user}
201 user deleted
403 unauthorized
404 doesn't exist
400 user not in group
POST /user creates a new user
body: {username: text, password, first name, last name, profile picture (linking to blob storage), etc}
200 user created successfully, here's the userId
400 malformed data or username already exists
DELETE /user/{userId} deletes a user
201 user deleted
404 user doesn't exist
GET /user/groups/{userId} gets all the groups for a given user (really we shouldn't have userId in the slug, and instead should just get the userId from the auth cookie of the logged in user. This prevents IDOR, but for the sake of argument I'm just putting the userId in the slug)
200 returns a list of groupIds that the user is in
404 user doesn't exist
WS /ws websocket endpoint for real time push updates
client -> server
subscribe {groupId, userId} for the user to get all message updates for a given chat
send {groupId, text, timestamp} same as the POST /message
server -> client
message {messageId, groupId, senderId, timestamp, text} a new message in a group
ack {messageId, status} for when a message is successfully sent
clients connect by hitting the API gateway/load balancer. Maybe we have a CDN to serve logos or anything else every user might see, but it's not super relevant for this architecture.
Load balancer has rate limiting to prevent abuse, handles authentication, and routes requests to the appropriate services. For example we should have an auth service to handle all authorization and user info, a read service for when someone is getting caught up on messages or querying old messages, a write service for when someone sends a message
The services all connect to the db and to the websocket cluster. We should have multiple ws nodes for scalability, and each open connection lives on one node. One node can serve many connections. When a user connects to a chat, we open a websocket with that chat.
When a user sends a message, it first goes to the write service. This adds the message to the db, including all the inboxes, and forwards the message to the pub/sub service. From there, all the ws nodes that have a connection open to that group can listen to that topic and forward it to their clients
To read messages, first it hits the user's inbox to get the n most recent messages to display them. Once the conversation is open, it no longer does polling, and only listens to the websocket. The client also keeps track of the last message its seen, so that if it disconnects and reconnects, it can query the read service to get the most recent messages before opening a new websocket
media goes into blob storage, which is uploaded asynchronously when someone sends a message, and pub/sub and ws just keep track of the url
I realize I've made a bit of a mistake here. Writes come in from the web socket cluster, and so need to flow back up to the database to be persisted from the web socket. That means they need to go through the write service, which interacts with the db. We can also send things straight from the web socket nodes into pubsub, to bypass the expensive db write and get things to clients more quickly, while still persisting in the backend
For the messages, we should hold them inbox style. We have one table called messages containing this
userId*, messageId (p), senderId*, groupId*, sendTime, text, media
where userId is the inbox for a specific user. When someone sends a message to a group, we write the same message to the messages table n times, where n is the number of people in the group. Each row is identical, except for the userId, which is different for every person in that group. When a user wants to retrieve their messages, they can just go to their userId, which is indexed, and get the most recent ones. This avoids having to do large scans based on their group membership to get all the necessary messages. Since getting all of a user's most recent messages, either across all groups or in a single group, will be the most common db access pattern, we should optimize for this
other tables we need. * means it is an indexed column, (p) means it is the primary key
groups
groupId (p), name, timeCreated, createdBy*
memberships
userId (p), groupId*
users
userId (p), username*, password (hashed), first name, last name, email*, etc
ws node
each node keeps a map in memory of which groups it is responsible for and which connections are mapped to that group. user connections should be pinned to a certain node, not group connections. If it's pinned by groupId, then a few large groups could cause a lot of pressure on particular nodes, while per user traffic should be better spread out. Each websocket connection also sends a 30s heartbeat to keep connections open. If we miss 2 heartbeats, then we close the connection and mark them as offline. On reconnect, the client sends the last seen message, and the service can send all the missed messages since then.
pub/sub
we can use kafka to send messages to ws nodes
the partition key for kafka is the groupId, so that all messages land on one partition and you never get messages out of order. Combining this with the userId pinning allows us to keep load even distributed while still grouping together groupIds and making sure things don't get out of order. the kafka key is the messageId, and should be idempotent so you don't get duplicates
cache
should be write aside for reads, both for messages and for group membership. Whenever something updates membership, the cache is immediately invalidated for that group membership.