Requirements

Functional Requirements:

Crawl 20 urls per second

Each Webpage is 200Kbytes

Each crawl checks for rate limiting and robots.txt

Non-Functional Requirements:

Crawl 20 urls per second

Each Webpage is 200Kbytes

Each crawl checks for rate limiting and robots.txt

Ability to scale up to more urls.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

High-Level Design

Use master/slave system of nodes, where URL hashes are given to each slave in the system. The hashes are generated from the name or IP of each URL. Each slave can, via consistent hashing fetch from master list. The url contents are set into mongo db. But master nodes also have access to mongodb. The db is multinode, and uses sharding.

If a certain slave is stuck/down, another one is brought up and the results from the stuck slave is no longer future valid. Can have lazy writes in the db and recrawling of every x hours. Crawling on machine should happen in parallel.

The user can access craw results through a web service .. which is load balanced with multiple mirrors.

Detailed Component Design

DB has sharding, nosql (less data), lazy readh.

Master/slave node can use something like paxos for coordination. Apache zookeeper can be an option.

Web service is load balanced.