Build a real-time Chat Pipeline

Question

Design a real-time chat system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Real-time Messaging**: Users should be able to send and receive messages in less than 100 milliseconds.
2. **User Presence**: Display online/offline status for users.
3. **Group Chats**: Support for both one-on-one and group chats.
4. **Message History**: Users should be able to retrieve the last 100 messages in chat history.
5. **Notifications**: Users should receive notifications for new messages.

### Non-Functional Requirements:
1. **Scalability**: The system should handle up to 10 million concurrent users with peaks of 1 million messages per second.
2. **Availability**: 99.99% uptime, ensuring that the chat service is almost always operational.
3. **Durability**: Messages should not be lost and must be stored durably.
4. **Latency**: The system should maintain a maximum latency of 100ms for message delivery.
5. **Compliance**: The system should comply with data protection regulations such as GDPR.

To estimate capacity, let's assume:
- **Concurrent Users**: 10 million users.
- **Messages per User**: 100 messages per day.
- **Peak Message Rate**: 1 million messages per second.

### Calculating QPS:
- **Total Messages per Day**: 10 million users * 100 messages = 1 billion messages/day.
- **Messages per Second**: 1 billion messages/day / 86400 seconds/day = ~11,574 messages/second (average).
- **Peak Load**: During peak times, we will see up to 1 million messages/second.

### Storage Estimation:
- **Average Message Size**: Assuming 256 bytes per message, total storage per day = 1 billion messages * 256 bytes = 256 GB/day.
- **Data Retention**: If keeping messages for 30 days, total storage needed = 256 GB/day * 30 days = 7.68 TB.

The architecture will consist of several components:
1. **WebSocket Server**: For real-time messaging, using technologies like Node.js with Socket.IO for handling connections and message broadcasting.
2. **Load Balancer**: Distributing incoming traffic across multiple WebSocket servers to handle scaling.
3. **Message Queue**: Using Kafka for decoupling message sending and processing, ensuring durability and scalability.
4. **Database**: A NoSQL database like MongoDB or Cassandra for storing user messages and chat history, optimized for read-heavy workloads.
5. **Cache**: Redis for caching frequently accessed data (e.g., user presence, message history) to minimize database load.
6. **Monitoring**: Prometheus and Grafana for monitoring system health and performance, with alerting configured through Alertmanager.

### User Schema:
```json
{
    "userId": "string",
    "username": "string",
    "status": "string",
    "lastActive": "timestamp"
}
```

### Message Schema:
```json
{
    "messageId": "string",
    "senderId": "string",
    "receiverId": "string",
    "content": "string",
    "timestamp": "timestamp"
}
```

### Group Chat Schema:
```json
{
    "groupId": "string",
    "members": ["userId1", "userId2"],
    "messages": ["messageId1", "messageId2"]
}
```

### Access Patterns:
- Fetch user messages by `userId` with pagination.
- Store and retrieve group messages efficiently.

1. **Consistency vs. Availability**: Given the real-time nature of chat, we may choose eventual consistency for message delivery. This allows for higher availability, but messages may not be immediately visible to all users.

2. **Database Choice**: Choosing a NoSQL database (like Cassandra) provides horizontal scalability and faster writes, but sacrifices some transactional consistency compared to traditional SQL databases.

3. **Caching**: Using Redis for caching introduces complexity with cache invalidation strategies, but significantly improves read performance for frequently accessed data.

4. **Message Delivery Guarantees**: Opting for at-least-once delivery via Kafka can lead to duplicate messages, requiring idempotency in the message processing logic.

5. **Monitoring and Alerting**: Implementing a robust monitoring solution increases operational overhead but is critical for ensuring system reliability and quick incident response.

Build a real-time Chat Pipeline

NVIDIA

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Calculating QP...

Submit Your Answer

NVIDIA Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search