Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

Since so much data exists, we need to split the data across multiple database instances.

We're going to partition the data by fileID. This splits the data roughly evenly across nodes. Another way is to partition by the user, but that results in hot users overload certain nodes. Partitioning by nodes can result in a hot file overloading a node.

Our file service is going to need to handle roughly 30M files per second. At 1 KB per file, this is 30GB/s of data traffic coming in just for the file. This is probably close to above our theoretical network bandwidth for a single node. And the node itself has to do some work, so we assume the file service needs to be scaled. We add a load balancer to the API gateway so that it can choose between different nodes of the file service. Since each file from the user is expected to be different we can generally assume no concurrency issue between the different file services. In normal operation each file service node will write the data into the file database, generate a blob DB link for the user to upload to, there should be no conflicts between the files.

How to make synchronization fast?

Currently each time a new client of a user logins, they send all the files they have to the file service and their last updated timestamp. The file service then needs to look up each file in the file Database and return the relevant blob links if they need to be updated. For users with 1000s of files, this can be very slow process even if there is only 1 or two files to be updated. One way to solve this is to maintain a queue that has a separate channel for each client. Now when a file is uploaded from one client, we write the blob Link of that file to a channel each client is subscribed to.

How to support updates on files that are large?

One issue with our current design is that we rely on the user to upload and update the blob DB. This is good for a new file, but for files that are updated, we don't necessarily want to upload a new blob into the DB and definitely not the full file. Instead we should have a separate API for updating a file vs uploading a new file. For updating a file, the user uploads the FILE ID and the diff. The file service will be in charge of changing the blob file in the blob DB.

Instead of having the client 2 synchronizing redownload files from the blob DB, instead the update service is writing diffs into the pending updates queue. When the client tries to sync, the update service reads the pending updates queue and returns the list of diff history. The client 2 then performs those diffs on their files. This assumes the files exist, for files that the client has never downloaded, they would have to retrieve from the Blob DB.

Submissions

Displaying Solutions with Scores Ranging from 1 to 10