System requirements

Functional:

1. Users ought to have the capability to register via their email and opt for a subscription plan. In the absence of a subscription, they will be granted 1 GB of complimentary storage.
2. Users must have the ability to both upload and download their files from any device.
3. Users should possess the functionality to share files and folders with fellow users.
4. Users should have the capability to upload files of up to 1 GB in size.
5. The system should facilitate automatic synchronization across all devices.
6. The system should support offline editing, enabling users to add, delete, or modify files offline. Once they reconnect to the internet, changes should synchronize with the remote server and other online devices.

Non-Functional:

1. Performance: The system should have low latency for file uploads, downloads, and synchronization operations.
2. Reliability: The system should have high availability, with minimal downtime for maintenance or upgrades.
3. Scalability: The system should be able to handle a large number of users and a high volume of data without experiencing performance degradation.

Capacity estimation

Assumptions

1. Number of registered users: 500 million
2. Daily active users: 100 million
3. Average files stored per user: 200
4. Average file size: 100 KB
5. Active connections per minute: 1 million

Storage Estimations

Total number of files = 500 million * 200 = 100 billion

Total storage required = 100 billion * 100 KB = 10 PB

API design

The service will expose API for uploading file and downloading file wihch we will discuss here. Other API like user sign-up, sign-in, sign-out, subscribing, unsubscribing are out of scope for this discussion as we want to solve the core problem in designing systems like Dropbox and Google drive.

Download Chunk

API used to download a chunk of a file stored in Dropbox.

The response will contain Content-Disposition header as attachment which will instruct the client to download the chunk. Note that Content-Length is set as 4096000 as each chunk is of 4 MB.

Upload Chunk

This API would be used to upload the chunk of a file.

Content-Type header is set as application/octet-stream to tell server that a binary date is being sent.

Get Objects

This API would be used by clients to query Meta Service for new files/folders when they come online.

The Meta Service will query the database and retrieve an array of objects containing the name, ID, and type of each object, along with an array of corresponding chunk IDs. Upon receiving this information, the client can utilize the Download Chunk API by providing the chunk IDs to download the necessary chunks and reconstruct the file.

Database design

The ER diagram provides a visual representation of the database schema, capturing the relationships between various entities. Let's dissect the components of our ER diagram:

• USER: Represents the users of the cloud storage platform, storing essential information such as usernames, emails, and passwords.
• FILE: Stores metadata about files uploaded by users, including filenames, sizes, and timestamps.
• DEVICE: Represents the devices owned by users, facilitating seamless access to stored files.
• OBJECT: Represents files or folders stored on a device, allowing for hierarchical organization of data.
• CHUNK: Stores data chunks of files, enabling efficient storage and retrieval.
• SUBSCRIPTION and PLAN: Handle user subscriptions to paid services, offering various plans tailored to individual needs.
• ACCESS_CONTROL_LIST: Manages the access control for shared files, ensuring secure collaboration among users.

High-level design

Following system diagram illustrates the architecture of Dropbox, showcasing the seamless interaction between client devices, the Dropbox application, servers, and databases. It highlights key functionalities such as user authentication, file management, synchronization, and sharing, ensuring efficient and reliable cloud storage services.

The following expanded diagram illustrates the intricate interactions within the Dropbox system, featuring multiple client devices seamlessly connected to various instances of the Dropbox application. These applications communicate with a cluster of Dropbox servers responsible for managing user requests, including authentication, file uploads/downloads, metadata management, and synchronization across the distributed server infrastructure. A central database coordinates metadata storage, ensuring consistency across all server instances. This detailed view highlights the robust architecture supporting Dropbox's reliable file storage and sharing services.

Core Components of the System

1. Client Devices: These are the devices used by users to access Dropbox, such as laptops, smartphones, or tablets. They interact with the Dropbox application installed on their devices to upload, download, sync, and manage files.
2. Dropbox Application: The Dropbox application runs on client devices and facilitates interactions between users and the Dropbox servers. It provides a user interface for accessing files, managing folders, and performing various actions such as uploading, downloading, syncing, and sharing files.
3. Dropbox Servers: These are the backend servers managed by Dropbox. They store and manage user data, handle user authentication, process file uploads and downloads, manage metadata, handle file synchronization between devices, and facilitate file sharing among users. Dropbox servers are distributed across multiple data centers for scalability, reliability, and fault tolerance.
4. Database: The database stores metadata and user account information. Metadata includes file names, sizes, timestamps, permissions, and other attributes associated with files and folders. The database is essential for efficient file management, search functionalities, and user authentication. Dropbox servers interact with the database to read and update metadata, manage user accounts, and perform various operations efficiently.

Interactions

1. User Authentication: The Dropbox application interacts with Dropbox servers for user authentication. When users log in to their Dropbox accounts, the application sends authentication requests to the servers, which validate user credentials and provide access to the user's data if authentication is successful.
2. File Upload and Download: Users interact with the Dropbox application to upload files from their devices to Dropbox servers or download files from Dropbox servers to their devices. These interactions involve transferring data between client devices and Dropbox servers over the network.
3. Metadata Management: The Dropbox application interacts with the database to read and update metadata associated with files and folders. Metadata management includes operations such as creating new files or folders, updating file attributes, retrieving file information, and performing searches based on file metadata.
4. Syncing: Dropbox servers handle file synchronization between multiple client devices associated with the same Dropbox account. When users make changes to files on one device, such as adding, modifying, or deleting files, Dropbox servers ensure that these changes are propagated to other devices to maintain consistency across all devices.
5. Sharing: Users can share files or folders with other Dropbox users. The Dropbox application sends sharing requests to Dropbox servers, which manage permissions, access control, and notifications related to shared files or folders. Recipients of shared files can access them through their Dropbox accounts based on the permissions granted by the file owner.

Request flows

The request flows within the Dropbox system play a crucial role in ensuring seamless user experience and efficient file management. Here's a breakdown of the key request flows involved:

1. Action Request Flow: Users can perform various actions within the Dropbox application, such as uploading, downloading, or deleting files. When a user initiates an action, the client application sends a corresponding request to the Dropbox server. The server processes the request and performs the necessary actions, ensuring data integrity and security.
2. Chunking Logic Request Flow: To optimize the upload process for large files, the client application may request chunking logic from the Dropbox server. Upon receiving this request, the server provides information on how to chunk the file effectively, allowing for efficient transmission and storage.
3. Syncing Request Flow: Dropbox enables users to sync their data across multiple devices seamlessly. When a user requests syncing between devices, the client application communicates with the Dropbox server to initiate the syncing process. The server identifies the devices associated with the user's account and coordinates the data transfer, ensuring that the latest changes are reflected across all devices.

Detailed component design

The system must manage a substantial amount of both read and write data, with their proportion remaining consistent. Therefore, during system design, emphasis should be placed on optimizing data communication between the client and server.

We are prioritizing the development of components located within the shaded area in the diagram above. The other components situated outside, such as the Users Service and Subscriptions Service, have already been extensively addressed earlier. Therefore, let's delve into the remaining components in greater detail.

Resiliency

Our system highly resilient owing to following:

Distributed Block Storage

Files are split up in to chunks and these chunks are replicated within data center for durability. Also these chunks are distributed across data centers in multiple geographic regions for redundancy. This makes sure that enough copies of chunks are available within data center is one machine goes down. Also if entire data center goes down, chunks can be served from a data center in other geographical location.

Queuing

We are using queuing in our system for sending out the notification to clients. Hence if any worker dies, message in a queue isn’t acknowledged and other worker picks up the task again.

Load Balancing

Since we are putting multiple servers behind the load balancer, there is redundancy. Load Balancer is continuously doing health check on servers behind it. If any server dies, load balancer stops forwarding the traffic to it and removes it from cluster. This makes sure that requests don’t fail due to a unresponsive server.

Geo-redundancy

We are deploying exact replica of our services in data-centers across multiple geographical locations. This ensures that if one data-center goes down due to some reason, the traffic could still be served from remaining data-centers.

Security

Our system is highly secure due to following:

HTTPS

The traffic between client and server is encrypted over HTTPS. This ensures that no one in the middle is able to see the data, especially the file contents.

Authentication

For each API request post log-in, we are doing authentication by checking the validity of auth_token in Authorization HTTP header. This makes sure that requests which originate from clients are legitimate.

Failure scenarios/bottlenecks

1. Authentication and Authorization: Authentication and authorization processes can become bottlenecks, particularly during periods of high user activity or when dealing with complex access control policies. Ensuring efficient authentication and authorization mechanisms is essential for maintaining security and performance.
2. File Chunking and Reassembly: Chunking large files for upload and reassembling them on the server side introduces overhead and complexity. Inefficient chunking algorithms or processing delays can impact the upload/download speeds and overall user experience.
3. Network Latency: Network latency can be a significant bottleneck, especially for file upload and download operations. Users with slower internet connections may experience delays in accessing their files, particularly when dealing with large files or high traffic volumes.

Future improvements

1. Enhanced Security Measures: Implementing additional security measures such as end-to-end encryption for user data could provide users with greater peace of mind regarding the privacy and security of their files.
2. Optimized Performance: Continuously optimizing the performance of the Dropbox system, such as by improving upload and download speeds, reducing latency, and optimizing resource utilization, can enhance the user experience and efficiency of file management operations.
3. Advanced Collaboration Features: Introducing advanced collaboration features such as real-time editing, commenting, and version control could further enhance Dropbox's capabilities for team collaboration and document management.
4. Intelligent File Organization: Implementing AI-powered algorithms to automatically organize and categorize files based on content, usage patterns, and user preferences could simplify file management and make it easier for users to find and access their files.
5. Seamless Integration with Third-Party Services: Enhancing integration with third-party services and platforms can provide users with a more seamless experience when working with files across different applications and ecosystems.
6. Offline Access and Synchronization: Improving offline access and synchronization capabilities to allow users to access and work on their files even when they're not connected to the internet can increase productivity and flexibility.
7. Cross-Platform Consistency: Ensuring consistency and feature parity across different platforms and devices, including desktop computers, mobile devices, and web browsers, can provide users with a cohesive experience regardless of the device they're using.
8. Advanced Search and Discovery: Enhancing search and discovery capabilities to allow users to easily search for and discover relevant files based on keywords, metadata, and contextual information can improve productivity and efficiency.
9. Data Lifecycle Management: Implementing robust data lifecycle management policies to automatically archive, delete, or migrate unused or obsolete files can help optimize storage utilization and reduce storage costs.
10. User Feedback Mechanisms: Establishing efficient mechanisms for collecting user feedback and incorporating user suggestions and feature requests into the development roadmap can ensure that Dropbox continues to evolve in line with user needs and expectations.