Loading...
Cloud file storage services have become very popular recently as they simplify the storage and exchange of digital resources among multiple devices. The shift from using single personal computers to using multiple devices with different platforms and operating systems such as smartphones and tablets each with portable access from various geographical locations at any time, is believed to be accountable for the huge popularity of cloud storage services.
Availability: The motto of cloud storage services is to have data availability anywhere, anytime. Users can access their files/photos from any device whenever and wherever they like.
Reliability and Durability: Another benefit of cloud storage is that it offers 100% reliability and durability of data. Cloud storage ensures that users will never lose their data by keeping multiple copies of the data stored on different geographically located servers.
Scalability: Users will never have to worry about getting out of storage space. With cloud storage you have unlimited storage as long as you are ready to pay for it.
Consistency: ACID is required. Atomicity, Consistency, Isolation and Durability of all file operations should be guaranteed.
Request JSON
{
"filename": "example.txt",
"path": "/user/directory/example.txt",
"block_hashes": ["hash1", "hash2", ...]
}
Response JSON:
{
"status": "success", // or "error"
"message": "Metadata saved successfully." // or error message
}
Response JSON:
{
"filename": "example.txt",
"path": "/user/directory/example.txt",
"block_hashes": ["hash1", "hash2", ...]
}
Request Data: Binary data of the block
Response JSON:
{
"status": "success", // or "error"
"message": "Block uploaded successfully." // or error message
}
Response: Binary data of the block
Response: Connection kept open until a change occurs.
Response JSON
{
"status": "change_detected",
"message": "A change has been detected.",
"changed_files": [
{
"filename": "example.txt",
"path": "/user/directory/example.txt"
}
// ... potentially other changed files
]
}
The core function of the Metadata Database in Dropbox is to store essential details about the files without actually storing the file content itself. One of the core requirements is maintaining strong ACID properties and consistency across devices. We use a Relational Database Management System (RDBMS) structure to manage and query the metadata generated by users.
To track file changes:
Devices can compare the LatestJournalID of files with the one in the File Metadata Table during syncing. If they're different, the system knows updates have occurred. The Journal table helps identify changes, aiding synchronization.
Client consists of main 4 parts:
Uploading a New File (Write Path)
When a user adds a new file to their Dropbox folder:
Downloading the New File (Read Path)
When another device attempts to sync the new file from Dropbox:
Data deduplication is a technique used for eliminating duplicate copies of data to improve storage utilization. It can also be applied to network data transfers to reduce the number of bytes that must be sent. For each new incoming chunk, we can calculate a hash of it and compare that hash with all the hashes of the existing chunks to see if we already have the same chunk present in our storage.
We can implement deduplication in two ways in our system:
a. Post-process deduplication
With post-process deduplication, new chunks are first stored on the storage device and later some process analyzes the data looking for duplication. The benefit is that clients will not need to wait for the hash calculation or lookup to complete before storing the data, thereby ensuring that there is no degradation in storage performance. Drawbacks of this approach are 1) We will unnecessarily be storing duplicate data, though for a short time, 2) Duplicate data will be transferred consuming bandwidth.
b. In-line deduplication
Alternatively, deduplication hash calculations can be done in real-time as the clients are entering data on their device. If our system identifies a chunk that it has already stored, only a reference to the existing chunk will be added in the metadata, rather than a full copy of the chunk. This approach will give us optimal network and storage usage.
1. Vertical Partitioning: We can partition our database in such a way that we store tables related to one particular feature on one server. For example, we can store all the user-related tables in one database and all files/chunks related tables in another database. Although this approach is straightforward to implement it has some issues:
2. Range Based Partitioning: What if we store files/chunks in separate partitions based on the first letter of the File Path? In that case, we save all the files starting with the letter ‘A' in one partition and those that start with the letter ‘B' into another partition and so on. This approach is called range-based partitioning. We can even combine certain less frequently occurring letters into one database partition. We should come up with this partitioning scheme statically so that we can always store/find a file in a predictable manner.
The main problem with this approach is that it can lead to unbalanced servers. For example, if we decide to put all files starting with the letter ‘E' into a DB partition, and later we realize that we have too many files that start with the letter ‘E', to such an extent that we cannot fit them into one DB partition.
3. Hash-Based Partitioning: In this scheme we take a hash of the object we are storing and based on this hash we figure out the DB partition to which this object should go. In our case, we can take the hash of the ‘FileID' of the File object we are storing to determine the partition the file will be stored. Our hashing function will randomly distribute objects into different partitions, e.g., our hashing function can always map any ID to a number between [1…256], and this number would be the partition we will store our object.
This approach can still lead to overloaded partitions, which can be solved by using 'Consistent Hashing'.
We can have two kinds of caches in our system. To deal with hot files/chunks we can introduce a cache for Block storage. We can use an off-the-shelf solution like Memcached that can store whole chunks with its respective IDs/Hashes and Block servers before hitting Block storage can quickly check if the cache has desired chunk. Based on clients' usage patterns we can determine how many cache servers we need. A high-end commercial server can have 144GB of memory; one such server can cache 36K chunks.
Which cache replacement policy would best fit our needs? When the cache is full, and we want to replace a chunk with a newer/hotter chunk, how would we choose? Least Recently Used (LRU) can be a reasonable policy for our system. Under this policy, we discard the least recently used chunk first. Similarly, we can have a cache for Metadata DB.
Let’s now discuss how we can handle version conflicts in a collaborative file editing scenario ensuring data integrity and making sure that users are notified when their file is being updated by their collaborators.
Client-side :
Server-Side
Conflict Resolution Strategies:
Although we have discussed a lot of things below are 3 things that we can explore to further improve our design
Network errors along the upload path might corrupt the chunks. The chunks can get corrupted even after being stored. It is important to make sure files are uploaded correctly, and stay correct.
It is also important to replicate data for robustness.
Each uploaded file should be replicated to: one copy in same data center for a quick recovery, one copy in the same region, and finally in a data center in another region in case of a disaster. All of them should be stored as chunks with version numbers, so that only the lost or corrupted chunks can be brought back to the production system.