Loading...
A user could get file chunks from other users.
User could get nearby peer list from other user.
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
sync_file_metadata(file_id)
get_chunk(file_id, chunk_id)
get_peer(peer_id)
file_id
file_name
location
chunks: [{
chunk_id:
chunk_path:
sum:
]}
Peer Chunk Table (HashTable):
file_id -> primary_key
chunk_id -> sort_key
peer_id: []
PeerTable
instance_id
address
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
The first step is to discover peers. Suppose each peer has a preset value of peers. It will discover the rest of the peers through get_peer call from peers. The get_peer call can be on demand.
During getting the files, it will search for known peers, to sync the metadata related to the file it is looking for. Meanwhile, it will also sync the peer table from the peers. If the file does not exist, it search another 10 unknown peers. If all the peers have been exhaustively searched, the process will go into sleep mode and will search again for every period of time.
If one peers has the file or contains the metadata of the file table, it will start downloading the chunk file. It will first get the file_metadata from that peer. Next, it will start to download existing chunks from that peer. If that machine contains chunk files other machines from the table, it use DFS to search for other peers containing different chunks. Upon network connectivity, it will try to maximize available network to maximize download speed. In case there is only one chunk available, it could not download.
Once it successfully downloads a chunk, it will update its own PeerChunkTable. For every peer it gets the metadata, it will also update their PeerChunkTable (this could be a trade off). It can broadcast the PeerChunkTable to peers upon finished downloading.
Meanwhile, if a machine is not available, it will temporarily remove that machine from its local table. The table will be broadcasted once it successfully downloads current chunk.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?