System requirements

Functional:

1. Peer Discovery: Nodes must be able to discover other peers on the network using a defined protocol (e.g., multicast or distributed hash table).
2. Data Sharing: Peers should be able to share files and data with each other directly without needing a centralized server.
3. Data Integrity: Implement checksum or hash verification mechanisms to ensure data has not been altered during transmission.
4. Dynamic Node Management: The network should handle nodes joining and leaving gracefully, redistributing responsibilities as needed.
5. Resource Availability: Nodes must allow others to request and access resources, and define policies for resource sharing.

Non-Functional:

1. Scalability: The system should efficiently handle an increasing number of nodes and data without performance degradation.
2. Security: Implement encryption for data transmitted over the network and mechanisms to protect against malicious nodes (e.g., reputation systems).
3. Redundancy: The system should ensure that data is not lost when nodes leave and that copies exist across multiple peers.
4. Fairness: The network should provide fair access to resources among peers to prevent monopolization by any single node.
5. Performance: The system should ensure low latency and high throughput during data sharing operations.

API design

Based on these functional requirements, we can categorize the APIs into the following groups:

1. Peer Management APIs

• JoinNetwork(network_id): Joins a specific P2P network.
• LeaveNetwork(): Leaves the current P2P network.
• DiscoverPeers(): Discovers other peers in the network.
• GetPeerInfo(peer_id): Retrieves information about a specific peer.
• EstablishConnection(peer_id): Establishes a direct connection with a peer.

2. Data Management APIs

• ShareFile(file_path, metadata): Shares a file with the network.
• SearchFile(keywords): Searches for files based on keywords or metadata.
• RequestFile(file_id, peer_id): Requests a file from a specific peer.
• ReceiveFile(file_id, peer_id): Receives a file from a specific peer.
• VerifyFileIntegrity(file_id): Verifies the integrity of a file.

3. Resource Management APIs

• OfferResource(resource_type, capacity): Offers a resource to the network.
• RequestResource(resource_type, required_capacity): Requests a resource from the network.
• AllocateResource(resource_id, peer_id): Allocates a resource to a peer.
• ReleaseResource(resource_id): Releases a previously allocated resource.

4. Security APIs

• GenerateKeyPair(): Generates a public/private key pair for the peer.
• EncryptData(data, recipient_public_key): Encrypts data using the recipient's public key.
• DecryptData(encrypted_data, private_key): Decrypts data using the peer's private key.
• VerifySignature(data, signature, sender_public_key): Verifies the authenticity of data.

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

High-level design

In the high-level design of a peer-to-peer (P2P) network, we'll focus on how the components will work together to form the entire system. Here’s an overview of the key elements in the design and their interactions.

High-Level Design Elements:

1. Peer Communication: Establish a direct communication channel between peers. Peers can send requests and responses using a design pattern, such as sockets or Websockets for real-time communication.
2. Data Distribution: Use techniques like sharding or partitioning where data is distributed across peers based on some criteria, improving load balancing and efficiency.
3. Discovery Protocol: Implement a discovery protocol (like Kademlia or another DHT-based method) to maintain a list of peers and enable efficient lookups for resources.
4. Security Mechanisms: Incorporate encryption (e.g., TLS or end-to-end encryption) for data transmission and a reputation system to assess peer trustworthiness and mitigate the impact of malicious nodes.
5. Fault Tolerance: Ensure redundancy in data storage by replicating data across multiple peers, allowing the network to recover if some peers go offline.
6. User Interface: Provide an interface for users to interact with the system, allowing them to upload, download, and manage resources.
7. Resource Index: Maintains metadata about shared resources (files, computing power, storage, etc.), including location, availability, and other relevant information. This enables efficient search and discovery.
8. Routing Algorithm: Determines the optimal path for data transfer between peers, considering factors like network congestion, peer load, and data proximity.

Supporting Components

• Incentive System: Motivates peers to contribute resources and participate actively in the network. This could involve reputation systems, token-based rewards, or other mechanisms.
• Monitoring and Management: Collects network metrics, detects anomalies, and provides tools for network administration and troubleshooting.
• Trust Management: Evaluates the trustworthiness of peers based on their behavior and reputation.
• Anonymity Layer: Protects user privacy by obfuscating their identity and traffic patterns.

Request flows

This sequence diagram clearly outlines the dynamic interactions in the process of searching for a file, searching for peers, downloading the file and joining the P2P network.

Description of the Sequence Diagram:

1. User sends a search query to the DiscoveryService.
2. The DiscoveryService responds with a list of peers.
3. The User then connects to Peer1 and Peer2, requesting the desired file from each.
4. Peer1 and Peer2 respond by sending the file data back to the User.
5. The User sends the received data to the DataIntegrityModule for validation.
6. The DataIntegrityModule confirms whether the data is valid or not and returns this confirmation to the User.
7. Finally, the User joins the network as a new peer.

Detailed component design

Data Sharing & Discovery Mechanics

Chunking

Chunking is a process of dividing a large file into smaller, more manageable pieces. This has several advantages in the context of P2P file sharing:

• Efficient distribution: Smaller chunks can be transferred between peers more quickly, especially over networks with limited bandwidth. This is because smaller file sizes require fewer network packets, reducing overall transfer time.
• Redundancy: By distributing chunks across multiple peers, the system can tolerate failures. If a peer holding a particular chunk becomes unavailable, the data can still be retrieved from other peers who possess that chunk. This redundancy ensures data availability and fault tolerance within the network.
• Parallel downloads: With chunking, different peers can download different chunks of the same file simultaneously. This can significantly improve download speeds compared to downloading the entire file sequentially from a single source.

Data Sharing

Once a file is divided into chunks, the P2P network facilitates sharing these chunks among participating peers. Here's how data sharing might work in this system:

1. File Registration: When a peer wants to share a file, it first registers the file with the network. This could involve generating a unique identifier for the file and its chunks, along with associated metadata (e.g., file name, size, type).
2. Chunk Indexing: The system creates an index of the file chunks, keeping track of which peers possess each chunk. This index can be distributed across the network using a DHT (Distributed Hash Table) for efficient retrieval.
3. Peer Discovery: When another peer wants to download the file, it uses the discovery mechanisms (like Kademlia DHT) to locate peers that have the desired chunks.
4. Chunk Download: The downloading peer establishes connections with the identified peers and requests the missing chunks. The data transfer protocols manage reliable and efficient chunk transmission between peers.
5. Reassembly: Once all chunks are downloaded, the receiving peer reassembles them in the correct order to reconstruct the original file.

Conflict Handling: Multiple peers sharing the same file can lead to conflicts. Strategies include:

• Versioning: Assigning unique versions to each file or chunk, allowing peers to track changes and resolve conflicts.
• Merging: Developing algorithms to merge conflicting changes, if possible.
• Replication factor: Limiting the number of replicas for a file to reduce the likelihood of conflicts.

Data Transfer Protocols: Efficient protocols like BitTorrent, which leverage peer-to-peer connections and interest-based downloads, can be adapted for file sharing.

Data Discovery with Distributed Hash Tables (DHTs)

A key challenge in P2P networks is efficiently locating resources (files or chunks) spread across numerous peers. Distributed Hash Tables (DHTs) offer a scalable and efficient solution for data discovery. Here's how DHTs work in this context:

• DHT Concept: A DHT acts as a distributed database that maps keys (identifiers) to values (locations). In a P2P file-sharing network, keys could represent unique identifiers for files or chunks, and values could be the network addresses (IP addresses and ports) of peers who possess those resources.
• Key Generation: Each file or chunk is assigned a unique key using a hashing function. This function ensures that similar files or chunks have similar keys, facilitating efficient routing. Popular hashing functions include SHA-256 and SHA-1.
• Routing: When a peer searches for a specific file or chunk (key), the DHT efficiently routes the query towards the peers responsible for that key. This routing leverages the distributed structure of the DHT, where each peer maintains information about a small subset of the keys and their corresponding values. By forwarding the query to the closest relevant peers, the DHT minimizes the number of hops required to reach the target resource.
• Overlays: DHTs create an overlay network on top of the physical network. This overlay network defines how peers connect and communicate with each other to maintain the DHT and facilitate data lookups.
• Indexing: Maintain a distributed index of available files and their locations. This can be combined with DHTs for efficient search.

Query optimization: Implement techniques like caching, query filtering, and result ranking to improve search performance.

Impact of Network Latency

• Latency reduction: Use techniques like network coding to reduce the amount of data transmitted.
• Caching: Cache frequently accessed data closer to users to minimize latency.
• Content delivery networks (CDNs): Integrate CDNs to improve content distribution and reduce latency.
• Overlay networks: Optimize the overlay network topology to minimize network hops.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?