System requirements

Functional:

Real-time Collaboration:
Multiple users should be able to simultaneously view and edit a document in real time.
Changes made by one user should be immediately reflected for all other users.
User Authentication:
Users should be able to create accounts and log in securely.
Authentication mechanisms should be in place to control access to documents.
File Management:
Users should be able to upload and download documents.
Users should be able to sync files across multiple devices seamlessly.
Document Versioning:
The system should support document versioning. Users should be able to view and restore previous versions of a document.
Document Formatting:
Users should be able to format text, insert images, create tables, and make other formatting changes to the document.
Comments and Suggestions:
Users should be able to add comments, suggestions, and annotations to the document for collaboration purposes.
Revision History:
The system should maintain a detailed revision history of all changes made to a document.
The system should also show who made the changes and when.
Notifications:
Users should receive real-time notifications for changes made by other collaborators, comments, and mentions.

Non-Functional:

Scalability:
The system should be able to handle 10 million daily active users (DAU).
Should support a peak QPS of 480 for the upload API.
Reliability:
The system should have high availability, aiming for 99.9% uptime.
Data integrity and consistency should be maintained across all users and devices.
Security:
User authentication should be secure and follow best practices.
Document access should be controlled based on user roles and permissions.
Data transmission should be encrypted.
Performance:
The real-time collaboration feature should have low latency, ensuring a smooth editing experience.
File uploads and downloads should be optimized for efficiency.
Document Size and Frequency:
Support an average file size of 1 KB.
Assume users upload 2 files per day.
Auditability:
Maintain detailed logs for all user actions, document changes, and system events.
Support auditing and analysis of system usage.

Capacity estimation

Assumptions

Consider we have 10 million active users
Average file size is 10KB
Each user creates 2 files per day.

Daily Storage:

File size per user per day: 2 files * 10 KB = 20 KB

Total daily storage for all users: 20 KB/user * 10 million users = 200,000,000 KB

Yearly Storage:

Total daily storage * Days in a year: 200,000,000 KB/day * 365 days = 73,000,000,000 KB/year

Storage for 5 Years:

Total yearly storage * 5 years: 73,000,000,000 KB/year * 5 years = 365,000,000,000 KB

Storage in PB: 365,000,000,000 KB = 3.6 PetaByte

API design

User Authentication API:
Description: Authenticate users and provide access to the system.
Input: User credentials (username, password).
Output: Authentication token or error message.
Document Upload API:
Description: Allow users to upload documents to the system.
Input: Document file, user authentication token.
Output: Success message or error response.
Document Fetch API:
Description: Enable users to download documents from the system.
Input: Document ID, user authentication token.
Output: Document file or error response.
Share Document API:
Description: Facilitate sharing of a document document.
Input: Document ID, user authentication token, collaborator user id.
Output: Updated document state or error response.
Document Versioning API:
Description: Manage document versions, allowing users to view and restore previous versions.
Input: Document ID, user authentication token, version information.
Output: Document with specified version or error response.
Comments and Suggestions API:
Description: Allow users to add comments, suggestions, and annotations to a document.
Input: Document ID, user authentication token, comment text.
Output: Updated document state with comments or error response.
Revision History API:
Description: Provide access to the revision history of a document.
Input: Document ID, user authentication token.
Output: Revision history log or error response.
Notifications API:
Description: Send real-time notifications for changes made by collaborators, comments, and mentions.
Input: User authentication token, notification preferences.
Output: Notification messages or empty response if no new notifications.

Database design

Here are a few tables that we will use for our design. This is not an exhaustive list but this is a good starting point.

classDiagram
  class User {
    +UserID: int [PK]
    Username: string
    PasswordHash: string
    Email: string
    CreatedAt: datetime
  }

  class FileContent {
    +FileID: int [PK]
    Content: binary
    +ChunkID: int [FK]
    +VersionID: int [FK]
  }

  class FileMetadata {
    +FileID: int [PK]
    +UserID: int [FK]
    FileName: string
    FileType: string
    FileSize: int
    CreatedAt: datetime
  }

  class AccessControl {
    +AccessID: int [PK]
    +FileID: int [FK]
    +UserID: int [FK]
    AccessLevel: string
  }

  class ChunkInformation {
    +ChunkID: int [PK]
    +FileID: int [FK]
    ChunkNumber: int
    ChunkData: binary
  }

  class Notifications {
    +NotificationID: int [PK]
    +UserID: int [FK]
    NotificationType: string
    Content: string
    CreatedAt: datetime
  }

  class FileComments {
    +CommentID: int [PK]
    +FileID: int [FK]
    +UserID: int [FK]
    CommentText: string
    CreatedAt: datetime
  }

  class FileVersions {
    +VersionID: int [PK]
    +FileID: int [FK]
    VersionNumber: int
    CreatedAt: datetime
  }

  User "1" -- "n" FileMetadata
  FileMetadata "1" -- "n" AccessControl
  FileMetadata "1" -- "n" ChunkInformation
  FileMetadata "1" -- "n" FileComments
  FileMetadata "1" -- "n" FileVersions
  ChunkInformation "1" -- "1" FileContent
  User "1" -- "n" Notifications

Database Choices

User Data:
Database Type: Relational Database (e.g., PostgreSQL, MySQL).
Reasoning: User data involves structured information, and relational databases provide strong consistency and ACID transactions, ensuring accurate and secure user authentication and authorization.
Focus: Consistency-focused. Implications: Ensures the reliability and integrity of user authentication and authorization data, critical for security and access control.
File Content:
Database Type: Document Database (e.g., MongoDB).
Reasoning: File content, stored as documents, benefits from the flexible schema and scalability of NoSQL databases, supporting efficient handling of varying content structures.
Focus: Availability-focused. Implications: Prioritizes availability for real-time collaboration, allowing for efficient storage and retrieval of document chunks.
Metadata and Chunk Information:
Database Type: Relational Database.
Reasoning: Metadata and chunk information involve structured data and relationships, making a relational database suitable for consistency and relational queries.
Focus: Consistency-focused. Implications: Ensures reliable and consistent handling of metadata, access control, and versioning, supporting collaborative editing with a consistent version history.
Access Control Information:
Database Type: Relational Database.
Reasoning: Access control information requires maintaining relational integrity and consistency, making a relational database suitable for enforcing and managing access permissions.
Focus: Consistency-focused. Implications: Ensures accurate enforcement of access control rules, maintaining a consistent and secure access environment.

Overall Focus and Implications:

Overall Focus: Balanced (Consistency and Availability).
Implications: This balanced approach ensures a reasonable level of consistency while maximizing availability for real-time collaboration. Relational databases are used for structured data that demands strong consistency, while a document database is employed for the flexible and scalable storage of file content. This combination supports the collaborative document editing system's goals of providing a seamless user experience and maintaining the reliability and security of user and document-related data.

Data Partitioning

Best Partitioning Strategy:

Range-based partitioning based on user ID ranges would be the most suitable strategy for this problem. It allows for efficient distribution of user-related data across different nodes, ensuring that user-specific documents, access controls, and metadata are co-located, reducing cross-node communication.

Reasoning:

Range-based partitioning aligns well with the fact that users are likely to access and collaborate primarily on their own documents, leading to more localized data access patterns. This minimizes the need for data movement across nodes during user-specific operations, improving overall system performance.

The "jump" in Jump Consistent Hashing refers to the ability to quickly jump between partitions. This characteristic is essential for efficiently locating the partition associated with a given key while minimizing computational overhead.

Partitioning Algorithm:

A consistent hashing algorithm, such as the Jump Consistent Hashing (JCH) algorithm, can be employed for mapping user IDs to specific partitions. Consistent hashing ensures a balanced distribution of data and provides flexibility in scaling by minimizing the impact of adding or removing nodes in the system.

Sharding

Best Sharding Strategy:

Range-based sharding based on user ID ranges would be the most suitable strategy for this problem. It aligns well with the likely access patterns, as users are more likely to collaborate on their own documents, ensuring that related data is co-located on the same shard and minimizing cross-shard communication.

Reasoning:

Range-based sharding is efficient for distributing user-related data evenly across shards while maintaining the locality of data access for users. This strategy supports optimized retrieval of documents, access control information, and collaboration activities, contributing to better overall system performance.

Scaling Strategy: Horizontal or Vertical

The best scaling strategy for databases in the context of a collaborative document editing service is horizontal scaling. This allows for distributing the workload across multiple nodes, accommodating the potential growth in users and data, and improving overall system performance by adding more servers as needed.

Read/Write Separation

Read/Write Separation is beneficial for improving performance, especially in scenarios where there is a high volume of read operations compared to writes. In a collaborative document editing service, where users frequently read and collaborate on documents, implementing Read/Write Separation allows for optimized resource allocation, faster response times for read-heavy operations, and improved overall user experience.

High-level design

Web App / Mobile App:
User interfaces for accessing and collaborating on documents, supporting three user types: Owners, Commentors, and Editors.
API Gateway:
Facilitates communication between clients and backend services, routing requests to appropriate services like Authentication, Analytics, Document Service, Comment Service, and Notifications.
Auth Service:
Manages user authentication and authorization, ensuring secure access to the collaborative document editing service.
Analytics:
Provides monitoring and logging functionalities, offering insights into system performance and user interactions.
Document Service:
Handles document-related operations, such as storage, retrieval, and collaboration, interacting with Document DB and Relational DB.
Comment Service:
Manages comments and annotations on documents, storing data in both the Relational DB and utilizing an Event Queue for real-time updates.
Event Queue:
Enables asynchronous communication and event handling between different services, ensuring real-time collaboration and responsiveness.
Notifications Service:
Sends real-time notifications to users about changes, comments, and mentions in documents, enhancing collaboration awareness.
Document DB:
Dedicated database for storing document content, supporting efficient retrieval and updates during collaborative editing.
Relational DB:
Manages structured data, including user information, access controls, comments, and metadata, providing consistency for relational data.
In-Memory Cache:
Improves performance by caching frequently accessed data from Document Service and Comment Service, reducing latency in document-related operations.

graph TD

  subgraph Clients
    web_app(Web App)
    mobile_app(Mobile App)
  end

  api_gateway(API Gateway)

  subgraph Services
    document_service(Document Service)
    comment_service(Comment Service)
  end

  auth[Auth Service]

  event_queue[Event Queue]

  notifications(Notifications Service)

  subgraph Analytics
    Monitoring
    Logs
  end

  subgraph Databases
    document_db(Document DB)
    relational_db(Relational DB)
  end

  subgraph Cache
    in_memory_cache(In-Memory Cache)
  end

  subgraph Users
    owner(Owner)
    commentor(Commentor)
    editor(Editor)
  end

  owner --> |Collaborative Editing| document_service
  commentor --> |Commenting| comment_service
  editor --> |Content Editing| web_app & mobile_app

  web_app & mobile_app -->  api_gateway

  api_gateway --> |Auth Services| auth
  api_gateway --> |Data Cache| Cache
  api_gateway --> |Document Services| document_service
  api_gateway --> |Comment Services| comment_service

  document_service --> |Document Data| document_db
  comment_service --> |Comment Data| relational_db
  comment_service --> |Event Notifications| event_queue

  auth --> |User Events| event_queue

  event_queue --> |Notifications| notifications

  document_db --> |Analytics Processing| Monitoring 
  relational_db --> |Analytics Processing| Logs

Request flows

Have a look at the below sequence diagram for upload flow and add metadata flow.

sequenceDiagram
    participant Client1
    participant Client2
    participant BlockServers
    participant CloudStorage
    participant APIServers
    participant MetadataDB
    participant NotificationService
    
    Client1->>APIServers: Add File Metadata (Status: Pending)
    APIServers->>MetadataDB: Upload File
    MetadataDB-->>NotificationService: Notify New File
    NotificationService-->>Client2: Notify New File

    Client1->>BlockServers: Upload File Content
    BlockServers->>CloudStorage: Chunk, Compress, Encrypt, Upload
    CloudStorage-->>APIServers: Upload Completion Callback
    APIServers->>MetadataDB: Update File Status (Uploaded)
    MetadataDB-->>NotificationService: Notify File Uploaded
    NotificationService-->>Client2: Notify File Uploaded

Detailed component design

Since this design problems is complex and contains a lot of components, to simplify lets break down and discuss smaller flows and components.

For operations like, upload and download the design is very similar to google drive design, we have separate services that will help us with authentication, storing metadata, creating, uploading and downloading the files.

To efficiently handle upload, edit operation on files, the files are divided into chunks and then these chunks are uploaded to the server, each file is divided into multiple chunks, the file metadata table stores information about all these chunks. Storing files into multiple chunks also helps with versioning and providing user with the history of updates on the file.

Chunking and use of block servers

For large files that are updated regularly, sending the whole file on each update consumes a lot of bandwidth. Two optimizations are proposed to minimize the amount of network traffic being transmitted:

Delta sync: When a file is modified, only modified blocks are synced instead of the whole file using a sync algorithm
Compression: Applying compression on blocks can significantly reduce the data size. Thus, blocks are compressed using compression algorithms depending on file types.

In our system, block servers do the heavy lifting work for uploading files. Block servers process files passed from clients by splitting a file into blocks, compressing each block, and encrypting them. Instead of uploading the whole file to the storage system, only modified blocks are transferred.

Notification service and Real-Time collaboration

To maintain file consistency, any changes performed on the fule locally needs to be informed to other clients to reduce conflicts. Notification service is built to serve this purpose. At the high-level, notification service allows data to be transferred to clients as events happen. Here are a few options:

Long polling. We can use long polling in client so they can always poll the server for any changes.
WebSocket. WebSocket provides a persistent connection between the client and the server. Communication is bi-directional.

Even though both options work well, we opt for long polling for the following two reasons:

WebSocket is suited for real-time bi-directional communication and collaboration. WebSocket is a communication protocol that enables bidirectional, real-time communication between clients and servers. It facilitates instant updates and notifications.

Implementation: WebSocket can be used to establish a persistent connection between clients and the server, allowing for immediate transmission of edits and changes.

Access Control

Google Docs allows us to invite collaborators and assign them various levels of permission, such as read-only or owner. However, for more precise access control, we can use Role-Based Access Control (RBAC).

RBAC ensures that employees only have access to the information and resources needed to fulfill their specific job responsibilities, and cannot access information that is not relevant to their role. This helps to protect sensitive data and maintain network security.

In the RBAC model, access can be restricted to specific actions, such as reading, creating, or editing files, based on various variables including authorization, responsibility, and job expertise. This ensures that only those with the appropriate permissions can perform

certain actions, which helps to protect sensitive data and critical applications. RBAC has several advantages, including:

Increasing operational effectiveness
Increased visibility of the owner
Cost reduction
Reducing the risk of data breaches

Real time collaboration and algorithms to resolve conflicts

Real-time collaboration in a document editing system involves allowing multiple users to concurrently view and edit the same document while ensuring consistency and minimizing conflicts. Several algorithms and techniques can be employed to achieve real-time collaboration, we will discuss the following 2:

Conflict-Free Replicated Data Types (CRDTs):

How it works: CRDTs are data structures designed to be conflict-free and convergent. They allow concurrent updates without the need for extensive coordination, making them suitable for real-time collaboration.
Implementation: CRDTs are used for different types of collaborative data, such as text documents (Treedoc), sets (LWW-Element-Set), or counters (G-Counter).
Merge Semantics: CRDTs define specific merge semantics for each data type. These semantics ensure that when concurrent updates occur on different replicas, the replicas can be merged together in a deterministic and conflict-free manner. Merge functions for CRDTs are carefully designed to guarantee convergence, often using techniques like Last-Writer-Wins (LWW) , First-Writer-Wins (FWW) or counter-based approaches

Let's consider a simple text document shared between two users, Alice and Bob, who are concurrently editing the document. We'll use a basic CRDT called a "G-Set" (Grow-Only Set) to demonstrate how changes made by both users can be merged without conflicts.

Initial Document State:
Both Alice and Bob start with an empty document.
User Actions:
Alice: Adds the word "Hello" to the document.
Bob: Adds the word "World" to the document.
CRDT Representation:
Each user maintains a G-Set to track the words they have added to the document.
Alice's G-Set: { "Hello" }
Bob's G-Set: { "World" }
Merging CRDTs:
When Alice and Bob synchronize their changes, their G-Sets are merged:
Merged G-Set: { "Hello", "World" }
Resulting Document State:
Both "Hello" and "World" appear in the document, reflecting the combined changes made by Alice and Bob.
Concurrency Handling:
Even if Alice and Bob had added words simultaneously, the G-Set ensures that all additions are included in the merged state, preventing conflicts. Merge Semantics decide how the words are ordered in the resultant document.

Operational Transformation (OT):

Transform Functions: OT relies on transform functions to modify operations, ensuring that the effects of concurrent edits are properly integrated. Transform functions are applied both at the client and server sides.
Synchronization and Consistency: OT requires regular synchronization between clients and the server to maintain a consistent view of the document. Synchronization involves exchanging operation logs and applying transformations.
User Intent Preservation: OT aims to preserve user intent during concurrent edits. Even if operations are applied in a different order, the resulting document state should reflect the users' intended changes.

Let's consider a simple text document shared between two users, Alice and Bob, who are concurrently editing the document.

Initial Document State:
Both Alice and Bob start with a document that contains “BC”.
User Actions:
Alice: Inserts the character "D" at position 2
Bob: Inserts the character "A" at position 0.
Individual Representation:
Alice's Version: { "BCD" }
Bob's G-Set: { "ABC" }
Merging:
When Alice and Bob synchronize their changes, the OT functions are applied to create a merge, both of them pass a function to the server which applies transformation:
Alice => INSERT(“A at beginning”, “BCD”) => “ABCD”
Bob => INSERT(“INSERT D at position 2”, “ABC”) => Here the server compares the versions and transform Alice’s changes to be made at position 3 => “ABCD”
Resulting Document State:
Both of them get “ABCD” in the document, reflecting the combined changes made by Alice and Bob. Transformation function takes care of merging the changes.

Trade offs/Tech choices

Choice of Conflict Resolution Strategy:

Trade-off: Operational Transformation (OT) offers precise conflict resolution but can be complex to implement and may require more computational resources. Conflict-Free Replicated Data Types (CRDTs) offer simpler conflict resolution but may have limitations in certain scenarios.

Data Partitioning Strategies:

Trade-off: Partitioning data based on user IDs, document IDs, or geographical regions can improve query performance and reduce contention but may lead to uneven data distribution, hotspots, and increased complexity in managing data placement.

Choice of Messaging Protocol for Real-Time Updates:

Trade-off: Using WebSocket for real-time communication offers low-latency, bidirectional communication but may require additional infrastructure for load balancing and handling large-scale deployments.

Future improvements

Optimized Conflict Resolution Mechanism: Implement more advanced conflict resolution mechanisms beyond basic OT or CRDTs, leveraging machine learning or natural language processing algorithms to intelligently merge conflicting edits and enhance collaboration efficiency.
Integration of Real-Time Presence Indicators: Integrate real-time presence indicators to provide users with visibility into who else is currently viewing or editing a document, fostering better communication and coordination among collaborators.
Enhanced Versioning and Rollback Functionality: Enhance versioning capabilities to allow users to create snapshots of document states at specific points in time and provide intuitive rollback functionality, enabling easy restoration of previous document versions in case of errors or unwanted changes.
Geographically Distributed Storage and Caching: Implement geographically distributed storage and caching mechanisms to reduce latency for users accessing the system from different regions, improving overall responsiveness and user experience for global collaboration scenarios.

Solutions

alchemy1135 - 03/12/2024

2

echo7959 - 09/07/2024

0

kraken_pinnacle338 - 08/09/2024

1