Loading...
A user could get file chunks from other users.
User could get nearby peer list from other user.
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
sync_file_metadata(file_id)
start_transmission(file_id, chunk_id)
get_peer(peer_id)
file_id
file_name
location
chunks: [{
chunk_id:
chunk_path:
sum:
]}
Peer Chunk Table (HashTable):
file_id -> primary_key
chunk_id -> sort_key
peer_id: []
PeerTable (RouteTable)
instance_id
address
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
The first step is to discover peers. Suppose each peer has a preset value of peers. It will discover the rest of the peers through get_peer call from peers. The get_peer call can be on demand.
Peer discovery could happen when a node wants to join a group of nodes. It will sends a join request to any known node. The it will iteratively contact other nodes. Other nodes will also update their peer table. (We should set an upper limit for number of close neighbor nodes, this could be a tradeoff).
Once the nodes have a neighbor table.
Now, if the machine wants to get a file, it will start connecting with neighbors to see they have this file and PeerChunkTable. If it contains a chunk, it will start to get the chunk. The peer node will initiate the data transfer on that chunk.
Meanwhile, the peer will also forward the request to other peers. If other peers has this specific chunk, they will start transmitting.
Once the transmission is complete, it will update its local ChunkPeerTable and broadcast it to its neighbors.
If the peer leaves the network, it will notify neighbors. There are circumstances where the machine disconnects randomly and did not have a chance to notify neighbors, that instance will be invalid. Therefore, when a machine is trying to request files from its neighbors, it should also see if the neighbor is reachable. If not, it will remove that instance from peers. If an instance disconnects and rejoins, it will need to start the initial setup again.
In case where a peer losts network connectivity randomly, the peer table may not immediately exclude it from peer. It may give it 3 strikes. If the peer is still not connected after 3 trails, it will be removed from peer network.
PeerChunkTable is not needed, but it is a crucial key to improve data download speed. If the traffic is routed among instances, they will be slow (this is essentially a DFS). If, on the other hand, we have this table, it will save us much time in finding peers.
Note, peer could contain contaminated chunk files. A chunk sum is necessary. The FileMetaData should be first acquired and cross-validated from peers. Only files with right check_sum should be downloaded to avoid spam data.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?