Loading...
users should be able to send messages either directly to another user or to a group chat
users should be able to quickly receive messages sent in a chat
support up to 10 users per group chat
users should be able to receive messages quickly
we prioritize reads over writes
messaging service should have high availability
1 billion users
each user sends 100 messages per day
each message is around 100 characters so 100 bytes
1 billion x 100 messages x 100 bytes = 10 PB per day
4 PB per year
send_message(user_id, chat_id, message_text)
read_chat(user_id, chat_id)
add_user_to_chat(user_id, chat_id)
create_new_chat(user_id)
chat members database:
user_id, chat_id
we want to be able to quickly fetch which chats a user is apart of. so we should partition the database on the user_id. and index on user_id.
we do not want write conflicts of adding or removing a user from a chat, so we should use single leader replication. so we can stick to mysql.
users database:
user_id, password_hash, email, other metadata
this database will be huge, but we don't expect to be updating it that frequently. so we can just index on user_id and partition on user_id. and use mysql for simplicity
messages database:
message_id, chat_id, timestamp, user_id, message_contents
we want to partition on chat_id so that we can quickly get the messages for a particular chat. we will sort on timestamp so that we know the consistent ordering in which we should display a chat's messages.
distributed timestamp isn't reliable, but it doesn't really matter in this case, as long as users all see the same order. if a user's message takes a little longer to send and is displayed later, then that's fine.
we want to optimize for reads in this case, which means that we probably want a database with a B tree index (as opposed to LSM Tree). this is also a good fit for column oriented storage since we may just need the message text and user columns. so HBase would be a good database choice. however, this is single leader replication, which means that we give up some write speed potential (if we had used Cassandra instead which is leaderless replication).
a user registers itself with an account service that writes to the users database
we have many potential servers for users to connect to and a load balancer in front of the servers to route the user to the correct server. this is all managed by a zookeeper coordination service.
each user should just maintain a single websocket connection to a server at a time. to do this the user first reaches out to the load balancer which tells it which server to establish the websocket connection with. the server will send occasional heartbeats to the user and to zookeeper. so if the server goes down, then zookeeper will know and use its consistent hashing algorithm to determine the new server for the user. the user who is now disconnected from the server will reach out to the load balancer again (at some random interval to avoid thundering herd), who will use zookeeper to tell it which server the user should now connect to.
once a user has established a websocket connection with a server, it can create a new chat or join an existing chat. the server will write this update to the chat members database. we'll use change data capture to propagate this update to a Kafka queue and then to flink which will maintain an internal mapping of chats to users. now if a user sends a message to a chat, the server will assign the message a unique id (for idempotence). and then the message will go to a the kafka queue (partitioned on chat id). the message will then get consumed by flink (parititoned on chat id) which guarantees exactly once processing. and flink will write the message to the messages database. if any component fails, we can get back to the state because of the log based message broker and flink's checkpointing. and then use the unique message id for idempotency when writing to the messages database.
flink also will send the message to the load balancer which will determine via zookeeper which servers the chat members are part of. then once the message is sent to the corresponding servers they will be send to the read user.
a user registers itself with an account service that writes to the users database
we have many potential servers for users to connect to and a load balancer in front of the servers to route the user to the correct server. this is all managed by a zookeeper coordination service.
each user should just maintain a single websocket connection to a server at a time. to do this the user first reaches out to the load balancer which tells it which server to establish the websocket connection with. the server will send occasional heartbeats to the user and to zookeeper. so if the server goes down, then zookeeper will know and use its consistent hashing algorithm to determine the new server for the user. the user who is now disconnected from the server will reach out to the load balancer again (at some random interval to avoid thundering herd), who will use zookeeper to tell it which server the user should now connect to.
once a user has established a websocket connection with a server, it can create a new chat or join an existing chat. the server will write this update to the chat members database. we'll use change data capture to propagate this update to a Kafka queue and then to flink which will maintain an internal mapping of chats to users. now if a user sends a message to a chat, the server will assign the message a unique id (for idempotence). and then the message will go to a the kafka queue (partitioned on chat id). the message will then get consumed by flink (parititoned on chat id) which guarantees exactly once processing. and flink will write the message to the messages database. if any component fails, we can get back to the state because of the log based message broker and flink's checkpointing. and then use the unique message id for idempotency when writing to the messages database.
flink also will send the message to the load balancer which will determine via zookeeper which servers the chat members are part of. then once the message is sent to the corresponding servers they will be send to the read user.
a user registers itself with an account service that writes to the users database
we have many potential servers for users to connect to and a load balancer in front of the servers to route the user to the correct server. this is all managed by a zookeeper coordination service.
each user should just maintain a single websocket connection to a server at a time. to do this the user first reaches out to the load balancer which tells it which server to establish the websocket connection with. the server will send occasional heartbeats to the user and to zookeeper. so if the server goes down, then zookeeper will know and use its consistent hashing algorithm to determine the new server for the user. the user who is now disconnected from the server will reach out to the load balancer again (at some random interval to avoid thundering herd), who will use zookeeper to tell it which server the user should now connect to.
once a user has established a websocket connection with a server, it can create a new chat or join an existing chat. the server will write this update to the chat members database. we'll use change data capture to propagate this update to a Kafka queue and then to flink which will maintain an internal mapping of chats to users. now if a user sends a message to a chat, the server will assign the message a unique id (for idempotence). and then the message will go to a the kafka queue (partitioned on chat id). the message will then get consumed by flink (parititoned on chat id) which guarantees exactly once processing. and flink will write the message to the messages database. if any component fails, we can get back to the state because of the log based message broker and flink's checkpointing. and then use the unique message id for idempotency when writing to the messages database.
flink also will send the message to the load balancer which will determine via zookeeper which servers the chat members are part of. then once the message is sent to the corresponding servers they will be send to the read user.
we want to optimize for reads in our messages database, which means that we probably want a database with a B tree index (as opposed to LSM Tree). this is also a good fit for column oriented storage since we may just need the message text and user columns. so HBase would be a good database choice. however, this is single leader replication, which means that we give up some write speed potential (if we had used Cassandra instead which is leaderless replication).
once a user has established a websocket connection with a server, it can create a new chat or join an existing chat. the server will write this update to the chat members database. we'll use change data capture to propagate this update to a Kafka queue and then to flink which will maintain an internal mapping of chats to users. now if a user sends a message to a chat, the server will assign the message a unique id (for idempotence). and then the message will go to a the kafka queue (partitioned on chat id). the message will then get consumed by flink (parititoned on chat id) which guarantees exactly once processing. and flink will write the message to the messages database. if any component fails, we can get back to the state because of the log based message broker and flink's checkpointing. and then use the unique message id for idempotency when writing to the messages database.
flink also will send the message to the load balancer which will determine via zookeeper which servers the chat members are part of. then once the message is sent to the corresponding servers they will be send to the read user.
each user should just maintain a single websocket connection to a server at a time. to do this the user first reaches out to the load balancer which tells it which server to establish the websocket connection with. the server will send occasional heartbeats to the user and to zookeeper. so if the server goes down, then zookeeper will know and use its consistent hashing algorithm to determine the new server for the user. the user who is now disconnected from the server will reach out to the load balancer again (at some random interval to avoid thundering herd), who will use zookeeper to tell it which server the user should now connect to.
once a user has established a websocket connection with a server, it can create a new chat or join an existing chat. the server will write this update to the chat members database. we'll use change data capture to propagate this update to a Kafka queue and then to flink which will maintain an internal mapping of chats to users. now if a user sends a message to a chat, the server will assign the message a unique id (for idempotence). and then the message will go to a the kafka queue (partitioned on chat id). the message will then get consumed by flink (parititoned on chat id) which guarantees exactly once processing. and flink will write the message to the messages database. if any component fails, we can get back to the state because of the log based message broker and flink's checkpointing. and then use the unique message id for idempotency when writing to the messages database.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?