Design a Web Crawler That Will Crawl Wikipedia

Design a web crawler that will crawl Wikipedia. We need to deploy the same software on each node. We have 10,000 nodes, the software can know about all the nodes. We have to minimize communication and make sure each node does equal amount of work.