System requirements

Functional:

• Support users posting and viewing comments in real time
• Organize comments in different channels
• Allow users to view comment history
• Out of scope
  • Support searching comments
  • Content moderation
  • Support threads

Non-Functional:

• System is highly available (99.99%)
• Eventual consistency is okay
• System is highly scalable: 10M DAU, 100k channels, 10 messages per channel per second, with peak load 100 messages / channel * sec
• Latency: <500ms for new messages to arrive on clients

Capacity estimation

Storage

• Assume we need to store all messages, and each message is 100 bytes.
• 100k * 10 * 100 = 100MB / sec. Daily storage is 10TB. In one year it is 3.6PB.
• User db: 10M users, each 1k metadata: 10GB

Network

• Need some kind of persistent connection to the server for real-time updates.
• Assume at any given time, up to 5M concurrent users.
• Assume each server can handle 100k connection max, and we want 50% capacity, then need: 5M / 100k * 2 = 100 servers

Bandwidth

• Each channel 50 users: total bandwidth: 100k * 50 * 10 *8 = 400Gbps
• Assume each server can handle 25Gbps, this is well within the limits of 100 servers

API design

// Create channel

POST /channels -> Channel

// Omit other CRUD methods

// Joins a channel and retrieves the last N messages

POST /channels/?action=join&limit=

// POST message

POST /channels//messages -> HTTP 200

Database design

User

• id (PK)
• username
• fullName
• ...

Channel

• id (PK)
• name
• owner (FK -> User)
• createTime
• ...

Message

• id (PK)
• creator (FK -> User)
• channelId (FK -> Channel)
• timestamp // server timestamp
• message

High-level design

ChannelService

• Manages channel lifecycle

MessageService

• Receive messages posted by client
• Handles queries for a range of messages when clients connect for the first time, or reconnects

MessageQueue

• Enqueues posts for further processing

MessageProcessor

• Stores messages for persistence and caching

BroadcastService

• Sends new posts to other clients listening on the same channel

Request flows

Channel management

• This is standard CRUD operations against a database. We omit the details here.

Client joins a channel

• Client first issues a join request, and reads the last 50 messages to display.
• In parallel, Client issues a SSE connection request, which is routed to a SSE server (perhaps by consistent hashing) and establish a SSE connection for the client
• The SSE service determines the Kafka topic that it should subscribe to, and is ready to send new messages to client.

Client rejoins a channel after disconnection

• This works similar to joining, except that it needs to make sure there is no gaps between the last message and new messages. There are several options, like reading all messages in between in a separate API call.

Send message

• Client sends a message through MessageService, which delivers the message to the topic for that channel.
• SSE service that listens to the topic picks up the message. It keeps a configuration of SSE connection to channel, so it knows which connection should get the message.

Detailed component design

Database

• The database is write heavy since most reads are not needed (i.e handled by SSEService)
• For the scale of messages, we need to partition the database. Say we are keeping 1 month worth of messages in database (older data can be moved to cold storage), that is 300TB, and we can create 10 partition so each is ~30TB.
• For high write, we choose something like Cassandra for its high write throughput.
• Partition key: channelId

SSE service

• We need ~100 SSE services. Each handles a number of clients.
• We client establishes SSE connection, APIGateway routes the request via consistent hashing to SSE service. This way there is both determinincity and when servers are added or removed, we need to migrate mimimal number of connections.

Message queue

• Message volume is 100MB/sec and 1M message/sec.
• Message/sec is on the edge for 1 broker.
• We want to keep topics to ~2k. That means we need to combine channels into smaller number of topics (by hashing), and SSE server need to ignore messages that it doesn't care about.
• We probably aim to have ~10 brokers. That means we should combine topics for low volume channels.
• For hot channels that can reach 1k messages / sec (100KB/sec), this is still within the limit of Kafka.

Caching

• If there are a lot of reads for hot channels (e.g. lots of users on a sport channel), we should consider adding a caching layer for the messages.

Trade offs/Tech choices

We choose SSE instead of websocket because most of the traffic goes from server to client (unidirectional).

Failure scenarios/bottlenecks

ChannelService, MessageService and MessageProcessor are stateless and we can horizontally scale it.

As discussed, we reduced the number of topics of MessageQueue to ~2000. For 1M message / sec, it averages to 500 per topic, and if we create say 5 partitions. We should tune this based on performance testing.

SSE services should be in a consistent hash ring so we can scale the servers out and in with minimal disruption of existing connections.

Client disconnections are handled as discussed.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?