System requirements

Functional:

Team Management

• Team Creation and Management: Users should be able to create teams, invite members, and manage team settings.
• Team Roles and Permissions: Implement a role-based access control system where team admins can assign roles (e.g., Admin, Member) and set permissions (e.g., who can add/remove members, manage files, create channels).
• Team Membership: Users should be able to join existing teams by invitation or by searching for and requesting to join public teams.

Communication

• Real-time Messaging: Enable real-time text-based communication within teams and through direct messages between users.
• Group Chats and Channels: Support the creation of group chats and channels where multiple team members can participate. Channels can be categorized (e.g., by project, department) and can be public or private.
• Threaded Conversations: Allow users to create threads within channels or group chats to keep discussions organized.
• Notifications: Implement a notification system that alerts users to new messages, mentions, and important updates. Users should be able to customize their notification preferences.

File Sharing

• File Upload/Download: Users should be able to upload files to channels, group chats, and direct messages. Likewise, they should be able to download shared files.
• File Versioning: Implement version control for files, allowing users to upload new versions and view previous versions.
• File Previews: Provide previews for common file types (e.g., PDFs, images) within the platform to avoid the need for downloading files unnecessarily.
• External Storage Integration: Allow users to link and share files from external storage services like Google Drive and Dropbox, enabling seamless access to documents and collaboration.

Non-Functional:

Scalability

• The system should scale horizontally to support 100,000 active users and the associated team activities, such as messaging and file sharing.
• Ensure that the platform can handle an increase in team creation, message volume, and file sharing as the user base grows.

Performance

• Real-time messaging should be delivered with a latency of less than 200ms to ensure smooth communication.
• The platform should handle up to 10,000 concurrent file uploads/downloads without significant performance degradation.

Security

• Implement data encryption at rest and in transit to protect user data, particularly sensitive communications and shared files.
• Ensure role-based access control (RBAC) is enforced to manage permissions within teams effectively.

Capacity estimation

User Base

• Active Users: 100,000 active users at peak times.
• Concurrent Users: Assuming around 10% of active users are online simultaneously, we expect approximately 10,000 concurrent users.

Messaging

• Messages per User: On average, each user sends/receives 200 messages per day.
• Total Messages per Day: 100,000 users * 200 messages = 20,000,000 messages per day.
• Message Throughput: At peak, we might expect 10% of these messages to be sent within the busiest hour:
  • 2,000,000 messages per hour / 3600 seconds = ~555 messages per second.

File Sharing

• Files Shared per User: Each user shares an average of 5 files per day.
• Total Files per Day: 100,000 users * 5 files = 500,000 files per day.
• Average File Size: Assuming an average file size of 5 MB.
• Total Data Transferred per Day: 500,000 files * 5 MB = 2.5 TB per day.
• File Upload/Download Throughput: Considering peak usage (10% in the busiest hour):
  • 50,000 files per hour * 5 MB = 250 GB per hour = ~70 MB per second.

Database Capacity

• User Data: User profiles, team memberships, and roles.
  • Estimation: 100,000 users * ~1 KB/user profile = ~100 MB.
• Metadata: Storing metadata for messages, files, and activity logs.
  • Estimation: Metadata size typically is small but grows with activity. For 1 year: ~1 TB.

API design

Team Management APIs

These group of api's will include api's for creating team, adding member to team, get team details and remove team member.

File Sharing APIs

These will include uploading file, downloading file and for listing files in a channel.

Communication APIs

Send Message

Get Messages

Create Channel

Database design

File Storage

Cloud Storage Service: Use AWS S3 (or equivalent) for storing files. This provides:

• Scalability to handle large volumes of files.
• Built-in redundancy and versioning support.
• Integration with CDN services to optimize file delivery.

Message Storage

NoSQL Database: Store messages in a NoSQL database such as MongoDB or DynamoDB. This approach provides:

• Flexibility in storing unstructured or semi-structured data like messages.
• Horizontal scalability to handle high write and read throughput.
• Efficient querying for search features (e.g., retrieving recent messages, searching by user or channel).

High-level design

Client (Web/Mobile): The frontend interacts with the API Gateway to access all backend services. It handles rendering the UI and managing user interactions.

API Gateway: This component routes requests to the appropriate backend services, ensuring secure and efficient communication between the frontend and backend.

Authentication & Authorization Service: Manages user authentication (e.g., login, logout) and ensures that users have the correct permissions to perform actions.

User & Team Management Services: Handle the creation, retrieval, and management of user accounts and teams. They interact with the relational database to persist data.

Communication Service: Manages real-time messaging and notifications. It stores messages in the NoSQL database for scalability and supports WebSocket connections for real-time updates.

File Storage Service: Manages file uploads, downloads, and version control. Files are stored in cloud storage, and metadata is kept in the relational database.

Notification Service: Sends notifications to users about relevant events, such as new messages or file uploads.

Search Service: Provides full-text search capabilities across the platform. It indexes messages, files, and other content to allow quick retrieval.

Request flows

Sending Message

Client: The user sends a message from the web or mobile application.

API Gateway: The request is routed to the Communication Service via the API Gateway.

Authentication & Authorization Service: The API Gateway first checks with the Authentication & Authorization Service to ensure the user is authenticated and authorized to send messages in the specified channel.

Communication Service:

• Receives the message and stores it in the NoSQL database.
• Broadcasts the message to all participants in the channel using WebSocket connections.

Notification Service: If the message mentions specific users, the Notification Service generates notifications and updates the notification table.

Client: The message is delivered in real-time to all participants in the channel.

Detailed component design

WebSocket Server Scalability:

• Horizontal Scaling: Deploy multiple WebSocket server instances behind a load balancer. This allows the service to handle more concurrent connections by distributing the load across multiple servers.
• Sticky Sessions: Ensure that once a user connects to a specific WebSocket server, subsequent requests from that user are routed to the same server. This reduces the overhead of re-establishing state across different servers.
• Auto-Scaling: Implement auto-scaling based on connection count and message throughput to dynamically adjust the number of WebSocket servers in response to demand.

Message Processing and Storage:

• NoSQL Database Sharding: Use sharding to distribute the data across multiple nodes in the NoSQL database. Sharding can be based on user IDs, channel IDs, or time periods (e.g., by month). This helps in distributing the load and prevents any single node from becoming a bottleneck.
• Write-Optimized Storage: Use a write-optimized storage engine (e.g., DynamoDB with optimized write capacity or MongoDB with a suitable shard key) to handle the high volume of message writes, especially during peak usage times.
• Message Caching: Implement an in-memory caching layer (e.g., Redis) to store recent messages or frequently accessed messages. This reduces the load on the database and speeds up message retrieval.
• Asynchronous Processing: Use a message queue (e.g., RabbitMQ, Kafka) to decouple message processing from immediate user interactions. This allows for better handling of burst traffic and ensures that the system can process messages even if there is a temporary spike in load.

File Access Performance:

• Content Delivery Network (CDN): Use a CDN to cache and serve frequently accessed files from edge locations close to the user. This reduces load times and decreases the bandwidth required from the origin cloud storage.
• Pre-Signed URLs: For secure and temporary access to files, generate pre-signed URLs that allow users to download or upload files directly from/to the cloud storage without going through the application server. This offloads the file transfer burden from your servers.

Trade offs/Tech choices

Operational Overhead vs. Performance: Choosing cloud-based solutions like AWS S3 and a CDN offloads much of the operational burden but introduces ongoing costs and reliance on external providers.

Consistency vs. Scalability: The choice between NoSQL for messages and relational databases for metadata reflects a trade-off between consistency and scalability, ensuring that each type of data is stored in a way that best matches its usage pattern.

Failure scenarios/bottlenecks

1. The CDN experiencing a failure or significant latency, will affect the delivery of cached files and static assets.
2. The message queue (used for decoupling and processing messages) can overload due to a sudden surge in traffic, causing delays in message processing and delivery.
3. The cloud storage service (e.g., AWS S3) can experiences high latency sometimes, affecting file uploads and downloads.

Future improvements

End-to-End encryption to use public key infrastructure (PKI) to encrypt messages and files on the client side before they are sent, and decrypt them on the recipient's side.