Requirements

Functional Requirements:

Crawl 20 urls per second

Each Webpage is 200Kbytes

Each crawl checks for rate limiting and robots.txt

Non-Functional Requirements:

Crawl 20 urls per second

Each Webpage is 200Kbytes

Each crawl checks for rate limiting and robots.txt

Ability to scale up to more urls.

99% available crawling.

API Design

FetchUriContent

CrawlUri

CancelUri

DeleteUri

RefreshUri

High-Level Design

Use master/slave system of nodes, where URL hashes are given to each slave in the system. The hashes are generated from the name or IP of each URL. Each slave can, via consistent hashing fetch from master list. The url contents are set into mongo db. But master nodes also have access to mongodb. The db is multinode, and uses sharding.

If a certain slave is stuck/down, another one is brought up and the results from the stuck slave is no longer future valid. Can have lazy writes in the db and recrawling of every x hours. Crawling on machine should happen in parallel.

The user can access craw results through a web service .. which is load balanced with multiple mirrors.

Scale up happends when master brings in more urls and db adds more shards/computers. This is horizontal scalling.

Load balancers handle fault tollerance.

Detailed Component Design

DB has sharding, nosql (less data), lazy readh.

Master/slave node can use something like paxos for coordination. Apache zookeeper can be an option.

Web service is load balanced.